Recovering Temporal Slack on GPUs

1Introduction

A GPU-hour is purchased whether the device is producing a user-visible result or waiting for the next phase of a pipeline. Interactive serving idles between bursts. Offline generation waits on separately reserved capacity. Reinforcement-learning pipelines alternate between rollout and training. Model-improvement loops serialize sampling, fine-tuning, deployment, and evaluation while a second GPU keeps production traffic alive. These gaps are different manifestations of temporal slack.

The conventional utilization percentage does not decide whether filling that slack is useful. A batch kernel can make the GPU look busy while pushing live requests past its latency objective. A co-located training run can finish more steps while changing the result. We therefore account for useful outcomes per paid GPU-hour and retain the guard metric for each workload: live SLO attainment and TTFT, item completion after preemption, loss or adapter equality, evaluation acceptance, and errors.

This paper combines the harvesting, live-plus-batch, rollout-plus-training, and train-serve-evaluate reports into one evaluation, and it absorbs reports 010 and 011, Train, Serve and Evaluate on One GPU and Live Traffic and Batch on One GPU, whose full results are the sections on the train-serve-evaluate loop and on live and batch generation. The experiments do not share one universal unit, so we do not average their ratios. Instead, we ask the same systems question in each campaign: how much otherwise idle paid time becomes valid work, and what measurable service or correctness cost accompanies it?

2Metric and accounting rules

For a campaign with useful output U, G occupied GPUs, and elapsed time T, useful work per paid GPU-hour is U divided by G×T. U is campaign-specific but always completion-qualified. Batch tokens count only when their item eventually completes. Live output counts only when the request meets its SLO. RL output is completed steps. A model-improvement output is a complete sample, train, deploy, and evaluate round whose adapter passes the acceptance rule.

Every occupied GPU is charged through the end-of-window drain. Work discarded by preemption, failed requests, and late live output contributes zero. We separately report device utilization because it helps explain slack, but a utilization increase is not converted into a useful-work gain.

3Reactive batch harvesting behind serving

The serving-harvest campaign replays the same interactive arrivals with and without a latency-tolerant batch backlog. The deployable harvester reacts to live demand without future knowledge. A separate oracle sees future arrivals and provides an upper bound on recoverable slack. Both regimes report additional completed batch tokens, GPU utilization, and the change in interactive TTFT p95 relative to interactive-only service.

The reactive system converts slack into 2.03 million completed batch tokens per GPU-hour without adding a GPU. That work is not free: live TTFT p95 rises by 26–30%. Whether the exchange is acceptable depends on SLO headroom. The oracle shows that most idle capacity can be recovered at a smaller 7–9% latency cost if arrivals are known, but it does not establish that a production predictor reaches that bound.

This result illustrates why occupancy cannot be the objective. The system must optimize a constrained pair: batch completions and live latency. Reporting only the recovered tokens would hide the user-facing cost; reporting only utilization would hide both.

4Preemption and recovery under a combined system

A second test on the flagship GPU enables harvesting together with memory isolation and idle release on a Qwen3-8B base model and one LoRA tenant. It records cold readiness, memory residency, reactivation latency, deterministic output equality, batch completions, failures, preemptions, resumptions, and an interactive request issued while batch work is active.

All preempted items resume and finish, so their tokens satisfy the completion-qualified numerator. The interactive request also succeeds while the backlog is active. This experiment establishes the end-to-end state transition and completion invariant when three mechanisms run together.

5Co-locating RL rollout and training

RL pipelines commonly allocate separate GPU pools to rollout generation and training. One pool waits while the other phase runs. We compare split placement with co-location on 8 previous-generation GPUs over 3 seeds and 8 RL steps per run. The same pipeline is executed in both arms and errors are counted.

Co-location yields 1.74–1.75× the RL steps per GPU-hour and approximately doubles measured utilization. The direction matches the pipeline structure: the same GPUs stay active through rollout and optimization instead of alternating between two paid pools.

Correctness noise floor

The preregistered 0.1% relative loss-match criterion was too strict for observed run-to-run variability. Two nominally identical split runs already diverge at step 1. Split-versus-co-located differences over 8 steps are of the same order as split-versus-split and co-located-versus-co-located variation. This is not an equivalence result. Correctness requires a noise-calibrated distribution over longer, multi-seed runs before the throughput gain can support a quality-preserving RL claim.

6Live and batch generation on one serving engine

The live-plus-batch campaign uses one flagship GPU and a single FP8/EAGLE3 serving engine. A bursty live trace runs beside a continuously refilled offline backlog. The strongest dedicated reference uses one live GPU and one batch GPU and may use the same fast configuration, so the comparison credits co-location rather than precision or speculation. Seeds 17, 29, and 43 are pooled after charging each occupied GPU through drain. Useful tokens are the output tokens of completed batch requests plus those of live requests that met their SLO, so live tokens that missed the SLO and failed requests earn nothing. The same live arrivals and batch prompts were replayed for every arm, backfill never looked at future live arrivals, and for each seed the dedicated reference took the best of the pre-registered stock candidates. All settings were pre-registered before any GPU time, in a preregistration with SHA-256 9a804ecee4c393a191ae3e2b3885baa0cfbba0d7b98cf59b90987b876b1b1929.

The shared engine clears the preregistered 1.5× gate by a thin margin. FCFS reaches 49.52 million useful tokens per GPU-hour, essentially matching priority at 49.54 million. The gain therefore comes from placing both queues on one engine, not from the priority policy.

Live TTFT p95 rises from 0.027 to 0.063 s, while SLO attainment remains 0.9982 with 0 errors. As in harvesting, co-location exchanges latency headroom for completed background work.

7Closing the train-serve-evaluate loop on one GPU

The improvement-loop campaign runs live serving, rejection sampling, LoRA training, adapter hot-loading, and evaluation for Qwen3-1.7B on GSM8K. All arms execute 5 rounds and evaluate on the same deterministic 500-question subset. An adapter is accepted only if it does not regress evaluation accuracy. The gate requires at least 1.5× the best SLO-qualified baseline and live SLO attainment of at least 0.95.

Nodus completes 1.65× more rounds per GPU-hour than the best qualified baseline under the preregistered field and 3.31× more on total GPU-seconds. It serves 2,171 of 2,257 requests within SLO with 0 errors. Naive sharing is disqualified: 1,595 of 3,941 requests meet SLO, attainment falls to 0.4047, and 2,346 errors occur. The guard and hot-load path are part of the result.

All 5 Nodus rounds are accepted and accuracy rises from 0.196 to 0.346; the standard loop moves from 0.196 to 0.26. These are one-seed within-run trends without confidence intervals. A product-path repeat passes all 5 gates at 48.2919 rounds per GPU-hour, 0.99683 live attainment, 0 loop errors, and 0 errors across four adapter swaps. Its baseline is reused from the harness, so it validates the product path rather than forming a new matched headline.

The subsections below carry report 010's full measurements of this loop: its setup and accounting, the per-arm throughput and live-serving tables, live latency, per-round accuracy, the ablation and the product-path validation.

Setup and accounting of the loop

Model and task. All arms run Qwen/Qwen3-1.7B [1] at a pinned revision with seed 0 for 5 rounds on GSM8K [2]. Every arm is evaluated on the same deterministic 500-question subset. A round's adapter is accepted only if it does not regress evaluation accuracy.

Serving. The serving engine is vLLM v0.30.0. Each arm carries a live request stream alongside the loop. A live request meets its SLO when it meets both its time-to-first-token (TTFT) and time-per-output-token (TPOT) targets, and SLO attainment is the share of offered live requests that do so. Errors count as misses.

Arms. Nodus runs live serving, rejection-sampling sampling, LoRA training and evaluation together on one GPU, with priority scheduling, the SLO guard and adapter hot-loading. Naive shared GPU (b1) runs the same work on one GPU without the guard. Standard loop (b2) trains, saves, restarts the serving engine and evaluates on one GPU, while live serving runs on a second GPU with first-come first-served scheduling.

Accounting. GPU-seconds count every GPU an arm occupies. The harness field for rounds per GPU-hour divides by the loop GPU's wall time only, which for b2 leaves out its live-serving GPU. We report b2 both ways, and the pre-registered gate uses the conservative field. All numbers come from one measured run whose frozen summary has SHA-256 1bd64c47287764ad831140473e38982a9a54ac1529bee6f4e7de1f01612c9228.

Loop throughput under a live SLO

Nodus completes all 5 rounds in 625.6429666420008 GPU-seconds on one GPU, or 28.770402545418175 rounds per GPU-hour. Its live stream meets the SLO on 2171 of 2257 offered requests, an attainment of 0.9618963225520603, with 0 errors. The standard loop b2 keeps perfect live attainment of 1.0 but needs 2068.418653431996 GPU-seconds on two GPUs. Against b2, Nodus delivers 1.65× more rounds per GPU-hour under the gate's conservative field and 3.31× on total GPU-seconds. The verbatim gate result names b2 as the best qualified baseline and records a pass with no failure reasons.

Sharing the GPU without the guard does not work. The naive shared arm b1 reaches 16.68 rounds per GPU-hour, but its live stream meets the SLO on only 1595 of 3941 requests, an attainment of 0.4047, with 2346 errors. It falls below the 0.95 SLO requirement and is disqualified.

Live serving latency

With the loop running on the same GPU, Nodus serves live requests at a TTFT of 0.0507 s at p50 and 0.1319 s at p95, and a TPOT of 5.91 ms at p50 and 9.12 ms at p95. The trainer paused for a total of 0.0106 s across the run to protect live traffic. For comparison, b2's dedicated live GPU reaches a TTFT p95 of 0.0432 s.

Accuracy per round

Nodus accepted all 5 rounds and its accuracy rose from 0.196 to 0.346. The standard loop moved from 0.196 to 0.26. These are within-run trends from one seed on 500 questions without confidence intervals, so we report them as measured and do not claim a statistically significant accuracy difference between arms.

Ablation

An earlier run of the same design without priority scheduling or the trainer guard reached live SLO attainment of 0.9315 with a TTFT p95 of 2.21 s, and the pre-registered gate failed on SLO. That run used a smaller evaluation set, so its accuracies are not directly comparable with the results above. The comparison shows that sharing one GPU across serving and training only meets the SLO once the guard is in place. The earlier run's frozen summary has SHA-256 e7b0807342e54187eb36515a4585c876c17c04fdab6e4cd634134046787f00fc.

Product-path validation

We repeated the loop through the Nodus product serving path rather than the experiment harness. The run used a real GPU, Qwen3-1.7B at the same pinned revision, 5 rounds of GSM8K and the same pre-registered gate. The standard-loop baseline was not re-run alongside it. The throughput gate instead uses the harness run's 8.70 rounds per GPU-hour, so the product path must reach at least 13.05 rounds per GPU-hour, which is 1.5× that baseline. The headline numbers above come from the harness run and are unchanged.

The live stream offered 1,261 requests, of which 1,257 met the SLO, an attainment of 0.99683 with 0 errors. All 4 misses were TTFT misses on requests that started during sampling. No request that started during evaluation, training or an adapter swap missed. The loop completed 48.2919 rounds per GPU-hour with 0 loop errors, and all four adapter swaps completed with 0 errors.

Accuracy rose from 0.258 to 0.298 across the run, a smaller gain than in the harness run. Like the harness run, this validation covers one seed, one model and one task. The artifact manifest has SHA-256 d7e55e979afc54f6d6ee7e4c14f9f49ed756fc0dc966204358915f5f5b41aaef.

Earlier product-path attempts

Four earlier product-path attempts did not pass. R1 failed during setup. R2 failed on a missing dependency. R3 failed when an adapter request returned 404. R4 completed but reached only 89.0% live SLO attainment because of a stale-alias bug. The run reported above followed these attempts.

8Cross-campaign synthesis

Slack exists at several time scales: milliseconds between live requests, seconds across batch preemption, minutes across rollout and training, and hundreds of seconds across model-improvement stages. The mechanism changes with the time scale, but the accounting rule does not: count completed, accepted work; charge every occupied GPU; retain the foreground guard.

Evidence boundary

The results span different hardware, models, traces, seeds, and useful-work units. They cannot be averaged into one global utilization multiplier. The serving campaigns do not prove arbitrary foreground isolation; the RL campaign is preliminary; the improvement loop does not establish general accuracy uplift. Traffic knowledge from an oracle is never treated as deployable.

9Reproducibility

Each campaign freezes arrivals, prompts, output limits, seeds, and arm definitions before the measured run. Useful-work denominators include occupied GPU count and drain time. Error counts and guard outcomes remain adjacent to throughput. The source evidence retains preregistration digests where available, frozen summaries, per-seed results, and product-path gate output.

A reproduction should repeat each matched campaign independently before adding longer RL runs with a split-versus-split noise control, practical forecast policies between reactive and oracle harvesting, and additional models and GPU types. Those extensions must preserve completion-qualified accounting rather than substituting raw utilization.

10Conclusion

Temporal slack is recoverable only when foreground service and correctness remain part of the objective. Across four real-GPU campaigns, Nodus converts idle paid time into completed batch tokens, RL steps, live-plus-batch output, and full model-improvement rounds. The gains range from 1.5066× useful live-plus-batch tokens to 3.31× improvement rounds on total GPU-seconds, with explicit latency and evidence costs. A busier GPU matters when the additional work finishes, passes its acceptance rule, and preserves the service commitment that justified sharing it.