Harvesting Idle GPU Cycles
Preemptible batch work behind interactive LLM serving, and co-located RL rollout and training
Abstract
GPUs are paid for by the hour, but much of a paid GPU-hour produces no useful AI output: interactive LLM serving is provisioned for bursts and idles between them, and reinforcement-learning (RL) pipelines that place rollout generation and training on disjoint GPU pools leave each pool waiting on the other. We argue that the right figure of merit for such systems is useful AI output per paid GPU-hour, and we report measurements of two ways to raise it. First, we ran preemptible batch inference in the idle capacity behind interactive serving. A reactive harvester added +2.03M batch tokens per GPU-hour at a cost of +26–30% interactive time-to-first-token (TTFT) p95. With perfect traffic foresight (an oracle upper bound), GPU utilization rose from 51.5% to 93.1% at +7–9% TTFT p95. In a combined-system test on one GPU, 64 active batch items were preempted 64 times and resumed 64 times while an interactive request returned in 0.501 s, with all 64 items completed and 0 failures. Second, as a preliminary result, co-locating RL rollout and training on the same GPUs yielded 1.74–1.75 RL steps per GPU-hour versus split placement (GPU utilization 48% 96%) on eight GPUs over 3 seeds and 8 steps. We also report a negative methodological finding: a strict 0.1% loss-match correctness gate is below run-to-run noise, since two identical split runs already diverge at step 1. We describe the problem setting, evaluation methodology, and results. System internals are out of scope.
1Introduction
Accelerator time is a major cost of deploying and improving large language models (LLMs). Yet the unit that is purchased, a GPU-hour, is rarely the unit that matters. What an operator actually wants is useful AI output: interactive responses delivered within their latency targets, batch tokens that complete, and RL training steps that make correct progress. Between these two units sits idle time, and idle time is common in two settings that dominate modern LLM fleets.
Interactive serving idles between bursts.
Interactive LLM endpoints must absorb bursty arrivals with low time-to-first-token (TTFT). Provisioning for the burst means that, on average, a substantial fraction of the GPU is unused. Meanwhile, many workloads (offline evaluation, synthetic data generation, document processing, embedding and summarization jobs) are latency-tolerant and could absorb that slack if they could be pushed out of the way the moment interactive demand returns.
Split RL pipelines idle by construction.
RL post-training of LLMs [1, 2] alternates between generating rollouts and updating the policy. When the two phases are placed on disjoint GPU pools, each pool is idle while the other phase runs. Frameworks such as OpenRLHF [3], HybridFlow/veRL [4] and RLHFuse [5] have explored placement choices for these phases. The measured cost of leaving them split is the starting point for our second study.
Harvesting idle resources is an old idea in cluster management [6, 7, 8, 9, 10] and in deep-learning cluster scheduling [11, 12, 13]. What is new in the LLM setting is the combination of (i) very large, stateful GPU-resident engines whose cold start is measured in minutes rather than milliseconds, (ii) a latency-critical tenant whose SLO is defined on token-level metrics such as TTFT, and (iii) training workloads whose correctness must be checked, not just their speed.
Contributions.
This paper is a measurement study. We make the following contributions:
We frame GPU harvesting around a single metric, useful AI output per paid GPU-hour, and define what counts as useful for interactive, batch and RL work (§2).
We measure preemptible batch harvesting behind interactive serving in a reactive regime and under an oracle with perfect traffic foresight, reporting both the gain and the interactive TTFT cost (§4).
We report a combined-system test on a single GPU in which harvesting, memory isolation and idle release ran together, including 64 preemptions and 64 resumptions with 0 failures (§5).
We report preliminary measurements of co-locating RL rollout and training on the same GPUs, together with the finding that a strict loss-match correctness gate is below run-to-run noise and therefore cannot serve as a correctness criterion (§6).
Scope of disclosure.
This paper describes what the research prototype did, the problem setting, the evaluation methodology and the measured outcomes. The scheduling and preemption policies, their parameters and the system architecture are proprietary and intentionally not described. We have tried to make the evaluation protocol precise enough that the reported quantities are unambiguous, and we are explicit about where the evidence is preliminary.
2Problem Setting and Metric
Useful AI output per paid GPU-hour
Let a deployment pay for GPUs for a wall-clock interval of length , so that it pays for GPU-hours regardless of how they are used. Let denote the useful output produced in that interval. We report where is workload-specific:
Interactive serving. Responses delivered without error. The cost side of harvesting is measured as the change in interactive TTFT at the 95th percentile (p95) relative to running the same interactive traffic alone.
Batch inference. Output tokens of batch items that complete. Work on an item that is preempted is useful only if the item eventually completes. Failed items contribute nothing.
RL training. Completed RL steps (rollout followed by policy update). A step is useful only if the co-located pipeline is as correct as the split one, which raises the question of how correctness should be tested (§6.2).
Because the denominator is fixed by what is paid for, any output produced in otherwise idle capacity raises the metric directly. For scale, the on-demand price of the GPUs used elsewhere in our measurements is $3.99/GPU-h ($31.92/h per 8-GPU node). Every idle fraction of such an hour is paid for in full.
We also report GPU utilization as a secondary, mechanism-agnostic indicator of how much of the paid capacity is busy. Utilization is not itself useful output: a GPU can be busy doing work that is later discarded. We therefore always pair utilization with an output or completion measure.
Workload model
Interactive tenant.
A latency-critical LLM endpoint receives requests whose arrivals are not under the operator’s control. Its requests must never be failed in order to make room for harvested work, and its TTFT should degrade as little as possible.
Preemptible batch tenant.
A latency-tolerant tenant submits batch items. Batch items may be paused at any time and resumed later. The harvester’s obligation is that every admitted item eventually completes and that no item fails because it was preempted.
RL pipeline.
An RL job alternates between rollout generation and training. In the split placement, the two phases run on disjoint GPU subsets of the same node. In the co-located placement, both phases run on the same GPUs. Both arms use the same model, data and hyperparameters. Only placement differs.
3Evaluation Methodology
Harvesting regimes.
We evaluate preemptible batch harvesting in two regimes. In the reactive regime, the harvester observes only current and past interactive load. It has no knowledge of future arrivals. In the oracle regime, the harvester is given perfect foresight of the interactive traffic. The oracle is not deployable. It is an upper bound that indicates how much of the idle capacity is in principle recoverable and how much of the reactive TTFT cost is attributable to not knowing the future.
Harvesting metrics.
For each regime we report (i) additional batch output tokens per GPU-hour relative to interactive-only operation, (ii) GPU utilization, and (iii) the change in interactive TTFT p95 relative to interactive-only operation. Ranges are reported as measured.
Combined-system test.
To check that harvesting composes with other mechanisms required in a real deployment, we ran a single server on one GPU with memory isolation, idle release and harvesting all enabled. The server hosted a Qwen3-8B base model and one LoRA tenant. We record cold-start readiness, GPU memory before and during idle release and after reactivation, reactivation latency, output equality before and after reactivation under greedy decoding, batch completions and failures, preemption and resumption counts, the latency and HTTP status of an interactive request issued while batch work is active, and errors.
RL co-location.
We run the split and co-located arms on 8 GPUs with 3 seeds and 8 RL steps per run. We report RL steps per GPU-hour, GPU utilization and errors. For correctness, we compare per-step training loss trajectories, and, crucially, we also compare two identical runs of the same arm, which establishes the run-to-run noise floor against which any split-vs-co-located difference must be judged.
Reporting discipline.
All numbers in this paper are measured on real GPUs. We do not extrapolate.
4Results: Preemptible Batch Work Behind Interactive Serving
Table 1 summarizes the harvesting results.
| Regime | Added batch output | GPU util. | Interactive TTFT p95 |
|---|---|---|---|
| Interactive only | – | 51.5% | reference |
| Reactive harvesting | +2.03M tokens/GPU-h | – | +26–30% |
| Oracle (perfect foresight) | – | 93.1% | +7–9% |
The gain.
Reactive harvesting added +2.03M batch tokens per GPU-hour. Since the GPU-hours are paid for whether or not batch work ran, this output was obtained without any additional GPU spend. In terms of our metric, every one of these tokens increased useful output per paid GPU-hour.
The headroom.
With perfect traffic foresight, GPU utilization rose from 51.5% to 93.1%. In other words, in this setting nearly half of the paid capacity sat idle under interactive-only operation, and the oracle showed that most of it can, in principle, be filled.
The cost.
Harvesting is not free for the interactive tenant. In the reactive regime, interactive TTFT p95 increased by +26–30%. With perfect foresight, the cost fell to +7–9% TTFT p95. The gap between the two regimes is therefore largely a prediction problem: a harvester that knew when interactive requests would arrive could step aside before they arrived instead of after. We regard closing this gap with deployable forecasts as the main open problem for this part of the work.
Interpreting the trade-off.
Whether +26–30% TTFT p95 is acceptable depends on how much slack the interactive SLO has. For deployments whose TTFT target leaves headroom, the reactive regime could turn idle capacity into batch output at no extra GPU cost. For deployments operating close to their TTFT target, the oracle result suggests most of the utilization gain could be achieved at a much smaller latency cost, but only to the extent that traffic can be anticipated.
5Results: Combined-System Test
Component measurements can hide interactions. We therefore ran harvesting together with memory isolation and idle release on a single one-GPU server hosting a Qwen3-8B base model and one LoRA tenant. Table 2 reports the outcome.
| Measurement | Result |
|---|---|
| Service ready (cold) | 170.0 s |
| GPU memory, active idle | 66,527 4,529 MiB (61,998 MiB, 60.5 GiB freed) |
| GPU memory after reactivation | 65,381 MiB |
| Reactivation latency | base engine 0.528 s, LoRA engine 0.646 s |
| Reactivation vs. cold start | 260–320 faster than cold start |
| Greedy outputs after reactivation | matched pre-idle outputs exactly (2/2) |
| Harvesting (12-item run) | 12 batch items completed, 0 failed, 6,144 output tokens |
| Harvesting under interactive load | 64 batch requests active, interactive HTTP 200 in 0.501 s |
| Preemptions / resumptions | 64 / 64 |
| Batch completion | all 64 items completed, 0 failures, 16,384 output tokens |
| Errors | 0 |
Preempted items complete.
With 64 batch requests active, an interactive request arrived and returned HTTP 200 in 0.501 s. Over this phase, batch work was preempted 64 times and resumed 64 times. All 64 items completed with 0 failures and 16,384 output tokens. This is the property the useful-output metric requires: preempted items still complete, so their output tokens count toward useful output.
Harvesting composes with idle release.
In the same server, idle release reduced GPU memory from 66,527 MiB to 4,529 MiB while idle, freeing 61,998 MiB (60.5 GiB) that other work could use, and reactivation took 0.528 s for the base engine and 0.646 s for the LoRA engine, compared with 170.0 s for a cold start. Greedy outputs after reactivation matched the pre-idle outputs exactly in 2/2 checks. No errors were observed over the test.
What this test does and does not show.
The test establishes that the mechanisms operated together on one server without failures or output changes. It is a functional, single-server test: it is not a throughput benchmark, and it involves one interactive probe under load rather than a latency distribution.
6Preliminary Results: Co-locating RL Rollout and Training
Throughput and utilization
Table 3 compares split and co-located placement of RL rollout and training on 8 GPUs over 3 seeds.
| Placement | RL steps per GPU-hour | GPU utilization | Errors |
|---|---|---|---|
| Split (disjoint GPU pools) | 505 | 48% | 0 |
| Co-located (same GPUs) | 882 | 96% | 0 |
| Ratio, co-located vs. split |
Co-location gave 1.74–1.75 RL steps per GPU-hour (882 vs. 505), and GPU utilization rose from 48% to 96%, with 0 errors. The utilization figures are consistent with the motivation for the experiment: in the split arm each GPU pool is idle while the other phase runs, and co-location keeps the same GPUs busy across both phases.
Correctness: the loss-match gate is below the noise floor
A throughput gain for RL is only useful if training is unaffected. Our original correctness criterion was a strict loss-match gate: the co-located loss trajectory should match the split trajectory to within 0.1% relative loss error. The co-located arm did not match the split arm under this gate. Before attributing the difference to co-location, we measured the noise floor by comparing identical runs of each arm (Table 4).
| Comparison | Diverges at | Max rel. loss err. | Max abs. loss err. |
|---|---|---|---|
| Split vs. split (identical) | step 1 | 0.7116 | 0.0286 |
| Co-located vs. co-located (identical) | step 1 | – | 0.0693 |
Two identical split runs already diverge at step 1, with a maximum relative loss error of 0.7116 and a maximum absolute error of 0.0286 over 8 steps. Two identical co-located runs also diverge at step 1, with a maximum absolute error of 0.0693. A 0.1% loss-match gate is therefore below run-to-run noise: it would reject the split arm compared against itself. It cannot distinguish a correct co-located pipeline from an incorrect one.
We therefore make no correctness claim beyond the following: over 8 steps, split-vs-co-located differences are of the same order as run-to-run variation. This is not evidence of equivalence. Establishing it requires a gate calibrated against a split-vs-split control (for example, comparing distributions of loss or reward over many seeds and longer runs), which we have not yet run.
A methodological note.
We believe this negative finding is of independent interest. Per-step exact-match or tight-tolerance loss checks are a natural first test when changing where RL phases run, but RL loss trajectories are sensitive to sources of nondeterminism even between identical configurations. Any placement study that reports a correctness pass or fail should report the corresponding identical-run control alongside it.
7Related Work
Harvesting and co-location in datacenters.
Co-locating latency-critical and best-effort work to raise utilization has a long history. Borg [6] runs production and batch jobs in the same cells and reclaims unused reservations for lower-priority work. Heracles [7] co-locates best-effort tasks with latency-critical services while protecting their tail latency through feedback-based isolation of shared resources. PARTIES [8] extends QoS-aware partitioning to multiple interactive services on one server. Fuxi [9] manages resources and jobs at internet scale, and analyses of co-located datacenter traces study where resource efficiency is lost when online services and batch jobs share machines. Our setting differs in the resource (GPU memory and compute held by large, stateful model engines) and in the latency metric (token-level TTFT).
GPU cluster scheduling for deep learning.
Gandiva [11] introduced introspective scheduling with time-slicing, suspend-resume and migration for deep-learning jobs. AntMan [12] dynamically scales the memory and compute of co-located deep-learning jobs to use idle GPU resources while protecting resource-guaranteed jobs. Lyra [13] loans idle inference-cluster GPUs to training jobs and uses elastic scaling to reclaim them. These systems target training jobs. We study harvesting behind interactive LLM serving, where the latency-critical tenant is itself a GPU-resident engine.
LLM serving.
vLLM [14] introduced PagedAttention for memory-efficient KV-cache management. Sarathi-Serve [15] addresses the throughput–latency trade-off of LLM inference through chunked prefills and stall-free batching. Llumnix [16] reschedules requests across model instances at runtime. These works optimize serving of a given workload. Our study concerns filling the capacity that interactive serving leaves idle with preemptible batch work, and accounting for the latency cost that this imposes.
RL post-training systems.
RL post-training of LLMs [1] with policy-gradient methods such as PPO [2] interleaves generation and training. OpenRLHF [3] provides a scalable RLHF framework built on distributed serving and training components. HybridFlow (veRL) [4] supports flexible placement of RLHF models, including co-located and separate placements. RLHFuse [5] fuses RLHF stages to reduce idle time. Our preliminary RL results are a measurement of split versus co-located placement in terms of useful output per paid GPU-hour, together with the correctness-gate finding of §6.2, which we believe is relevant to evaluations of any of these systems.
8Conclusion
Measured in useful AI output per paid GPU-hour, idle GPU capacity was a large and recoverable cost. Behind interactive serving, reactive preemptible batch harvesting added +2.03M batch tokens per GPU-hour at +26–30% interactive TTFT p95, and perfect traffic foresight raised GPU utilization from 51.5% to 93.1% at +7–9% TTFT p95. A combined-system test showed 64 preemptions and 64 resumptions with all 64 items completed and 0 failures while an interactive request returned in 0.501 s. For RL, preliminary measurements of co-locating rollout and training showed 1.74–1.75 RL steps per GPU-hour and utilization rising from 48% to 96%, but correctness remains open: a strict loss-match gate is below run-to-run noise, and over 8 steps split-vs-co-located differences are only shown to be of the same order as run-to-run variation. The next steps are deployable traffic forecasts to close the gap to the oracle, longer multi-seed RL runs with noise-calibrated correctness gates, and multi-node evaluation.
References
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, others. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS). 2022.
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347. 2017.
- Jian Hu, others. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv preprint arXiv:2405.11143. 2024.
- Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, others. HybridFlow: A Flexible and Efficient RLHF Framework. Proceedings of the European Conference on Computer Systems (EuroSys). 2025.
- Yinmin Zhong, others. Optimizing RLHF Training for Large Language Models with Stage Fusion. arXiv preprint arXiv:2409.13221. 2024.
- Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, John Wilkes. Large-scale cluster management at Google with Borg. Proceedings of the European Conference on Computer Systems (EuroSys). 2015.
- David Lo, Liqun Cheng, Rama Govindaraju, Parthasarathy Ranganathan, Christos Kozyrakis. Heracles: Improving Resource Efficiency at Scale. Proceedings of the International Symposium on Computer Architecture (ISCA). 2015.
- Shuang Chen, Christina Delimitrou, Jos\'e F. Mart\'\inez. PARTIES: QoS-Aware Resource Partitioning for Multiple Interactive Services. Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 2019.
- Zhuo Zhang, Chao Li, Yangyu Tao, Renyu Yang, Hong Tang, Jie Xu. Fuxi: a Fault-Tolerant Resource Management and Job Scheduling System at Internet Scale. Proceedings of the VLDB Endowment. 2014.
- Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, others. Gandiva: Introspective Cluster Scheduling for Deep Learning. Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2018.
- Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, others. AntMan: Dynamic Scaling on GPU Clusters for Deep Learning. Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2020.
- Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, Cong Wang. Lyra: Elastic Scheduling for Deep Learning Clusters. Proceedings of the European Conference on Computer Systems (EuroSys). 2023.
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, others. Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). 2023.
- Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, others. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2024.
- Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, others. Llumnix: Dynamic Scheduling for Large Language Model Serving. Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2024.