Training That Survives Reclaim

1Introduction

More AI per GPU and per fleet depends on putting idle and reclaimable capacity to work. A GPU that serves inference at peak and waits off-peak, or one held for reserved work that has not arrived, can train in the meantime, but only if the training survives the moment the GPU is taken back. A job that loses its progress at every reclaim spends the borrowed capacity on work it then repeats. Training jobs protect themselves with file checkpoints, which save the model and optimizer state every few minutes and resume from the newest one. Our reports 033 [1] and 034 [2] showed that a containerized inference engine comes back from a GPU memory snapshot in seconds. This report asks whether the same kind of snapshot works as reclaim recovery for a training process, and whether it does better than the file checkpoint.

We trained one PyTorch [3] model on two GPUs on two hosts and protected it in three ways: a GPU memory snapshot every 60 s, a file checkpoint every 60 s, and one snapshot taken only when a reclaim notice arrives. We checked that restored training continues exactly, measured what each protection costs while the job runs, and reclaimed the job in a drill to measure the work lost and the GPU-seconds needed to finish. A trainer compiled with torch.compile [4] tested the case where starting again is expensive.

The answer is mixed, and we report it that way. Every restore was exact. As a periodic protection, snapshots lost to file checkpoints on every measure we took, because a snapshot copies PyTorch's whole GPU memory reserve. A single snapshot taken on a reclaim notice lost no work at all. Snapshots also won when restarting is expensive, as with a compiled model. The direction that follows for the product is a snapshot on notice on top of file checkpoints, not snapshots in place of them.

2Experimental setup

Platform path. Both hosts ran the Nodus node runtime's guarded runc executor on containerd 2.4.1 and runc 1.5.2, with CRIU 4.3 [5] and NVIDIA cuda-checkpoint [6], the same NVIDIA driver and GPU persistence mode on. The trainer and cuda-checkpoint used the CUDA 13.0 forward-compatible user-mode driver on top of that driver. A research test of the platform's runtime package, which is not merged, drove every run. It starts the trainer as a container through the executor, takes snapshots through the executor's checkpoint call, restores through the executor with a snapshot as the container's checkpoint, and reclaims by killing the container. The two hosts had the same CPU model, kernel and CPU flags, which the executor checks before a restore, and different GPUs.

Workload. The trainer is a character-level GPT with 12 layers, width 768 and 12 heads, 85.35 million parameters, trained with AdamW [7] and bf16 autocast on blocks of 256 characters in batches of 32, on PyTorch 2.7.1. Its 1.1 MB of text sits in GPU memory, batches come from a seeded CPU generator and dropout from the CUDA generator, and PyTorch runs in deterministic mode, so every run of the same steps produces the same bits. The job is 1800 steps of about 0.23 s each, 418.4 s uninterrupted. PyTorch's caching allocator reserved 7.14 GB of GPU memory, of which 1.46 GB is live training state, the parameters, gradients and optimizer state.

Two periodic protections. In the snapshot arm the trainer has no checkpoint code. Every 60 s after its first step the test took a snapshot that leaves the process running, and a copier sent each finished snapshot to the other host's disk. In the file arm the trainer saved its model, optimizer, step and both random generators every 60 s with torch.save to a temporary file, synced it to disk and renamed it, a 1.02 GB file, and the copier sent each one to the other host. Both copies were one stream over the hosts' private network. A protection point counts only once its copy is whole on the recovery host.

Drill and checks. The two arms ran at the same time on opposite hosts, and the test killed each container 270 s after its first step. The recovery host then started from the newest point whose copy was whole before the kill. It restored the snapshot or resumed the checkpoint, with the page cache dropped first, and trained to step 1800. The notice arm took one snapshot that stops the process 150 s after the first step, copied it to the other host and restored it there. Each run's loss bits at every step, parameter bit-sums every 10 steps and SHA-256 digests at steps 1000 and 1800 were compared with an uninterrupted reference run.

3A restored snapshot continues bit for bit

Exact. All 4 snapshot restores continued the reference trajectory bit for bit on the other host's GPU, with no checkpoint code in the trainer. A snapshot resumes the step that was running when it froze. The periodic snapshot froze during step 668, and the restored process's first line is step 668 with the reference's loss bits. The restored process prints none of the trainer's start-up lines, because it is the same process continuing. Training kept its speed after a snapshot that leaves it running, 0.2297 s per step before against 0.2289 s after. The uninterrupted runs on the two hosts were also identical to each other, so a restore on the other GPU can be checked against either.

A file checkpoint is exact only with care. The careful checkpoint, which saves both random generators, resumed exactly as well. Getting it there took one fix during the experiment, because torch.load had moved the saved generator states to the GPU and the generators refused them. The same checkpoint resumed without its random state, as resume code that saves only the model and optimizer would do, diverged at its first step and never rejoined. Every one of its 778 steps had different loss bits, and its final loss was 1.1823 against 1.2216. A snapshot needs nothing from the trainer except determinism, which a file resume needs too.

4What a snapshot costs while training runs

Why a snapshot is large. A snapshot holds the process's GPU memory, which cuda-checkpoint moves to host memory, together with the process's own memory. That is PyTorch's whole 7.14 GB reserve, not the 1.46 GB of live state. Each snapshot was 8.53 GB, within 1% of the reserve plus the process's 1.42 GB of resident host memory, and 8.3 times the size of the checkpoint file, and 8.51 GB of it is a single page image.

Why a snapshot stalls. CUDA is locked from the moment cuda-checkpoint freezes the process until its state is back on the device, so training makes no progress during a snapshot. The median stall was 12.9 s, 9.5 times the 1.36 s of a torch.save, and the snapshot call returned 0.18 to 4.77 s after training had already resumed. Over the first 270 s, four snapshots left the snapshot arm at step 945, where the reference had passed it at 217.1 s, 24.3% slower. Four checkpoints left the file arm at step 1145, where the reference had passed it at 263.6 s, 2.4% slower. The stall per snapshot is fixed, so a 30-minute cadence would cost 0.7% in stalls, but a longer cadence loses more work at each reclaim.

5Reclaim drill

Reading. The snapshot arm lost more than twice the work. Its fourth snapshot, taken at 240 s, was still being copied at the kill, with 6.82 of its 8.53 GB on the other host, so recovery used the third, taken at 180 s. The checkpoint written at 240 s had been whole on the other host 3.4 s after the save began. That is the protection lag at work: a snapshot was safe 31.9 s after it started, a checkpoint after 3.4 s. Replaying every kill instant against the measured timelines gives the same order, 42.4 s of expected lost work against 29.5 s. The restore also took longer to reach its first new step, 7.24 s against 4.77 s, because it reads the 8.53 GB snapshot back from disk, while this small trainer starts from scratch in 3.6 s. Through one reclaim, the snapshot arm needed 544.1 GPU-seconds to finish, against 459.3 s for the file arm and 422.5 s uninterrupted.

6One snapshot on a reclaim notice

Nothing lost. A snapshot that stops the process captures it where it is, so the restored process ran the very next step, and neither direction lost or repeated a step. A notice has to cover the snapshot and the copy, 36.7 and 37.0 s here, so a notice of about 37 s is enough for this job, and the restore can follow on the other host after the node is gone. The first new step came 44 s after the freeze, counting only the snapshot, the copy and the restore. The test scripts' waits between those steps made it 72.1 to 73.3 s by the wall clock, 28.0 to 29.6 s that a node starting the copy and the restore at once would not spend. Starting the copy as soon as the dump finishes, rather than when the snapshot call returns, would shorten it further, and we did not measure that. The stopping snapshot took 20.0 to 21.4 s, longer than the 12.9 s stall of a snapshot that leaves the process running, and we did not instrument why.

7When restarting is expensive: torch.compile

The restore skips the warm-up. Compiling makes the trainer faster, 0.137 s per step against 0.23 s eager, but its cold start took 29.1 s to the first step against 3.6 s. A file resume pays the compile again. It reached its first new step 28.1 s after it began, 23.0 s of that between the trainer being ready and its first step. The snapshot restore reached its first new step in 6.57 s, because the restored process already holds its compiled code. Even with each copy fetched only after the reclaim, 7.82 GB in 14.7 s against 1.02 GB in 1.88 s, the restore would reach its first step in 21.2 s against 30.0 s. The compiled snapshot was 7.81 GB and stopped training for 13.3 s, and all three recoveries were bit-identical to the uninterrupted compiled run. The snapshot and the checkpoint come from different points of the run, step 429 against step 330, so this comparison is about the time to the first step, not about lost work.

A process holding an NVML handle blocks the snapshot. The first snapshot of the compiled trainer failed in CRIU. Inductor's default compile pool had started 30 worker processes, and the first of them that CRIU reached had loaded NVML, NVIDIA's management library, and mapped the GPU device /dev/nvidia0. cuda-checkpoint did not release that mapping, so CRIU stopped with "Can't handle non-regular mapping". With TORCHINDUCTOR_COMPILE_THREADS=1, which compiles inside the training process without a pool and was the only change, the snapshot worked, and a file resume with that setting takes 4.5 s longer to compile. We expect any process in the container that holds an NVML handle, such as a GPU monitoring agent, to block a snapshot the same way, but we observed it only for the compile workers. A product has to detect this before it promises a snapshot.

8What reclaims cost at a given rate

Reading. The model uses the measured stalls, lags and times to the first new step, 12.9 s, 31.9 s and 7.24 s for snapshots and 1.36 s, 3.4 s and 4.77 s for checkpoints. File checkpoints come out ahead of periodic snapshots at every rate, 3.0% against 9.6% at one reclaim per hour. A snapshot on notice is cheapest, 1.2% at one reclaim per hour, as long as every reclaim comes with a notice of about 37 s. A job with no checkpoint code restarts from zero at each reclaim. At one reclaim per hour, a one-hour job then expects to run 72% longer and a four-hour job to take 53.6 h. For such a job a snapshot is the only protection that needs no change to the job, and even periodic snapshots bring the cost down to the 9.6% above.

9What we recommend for reclaim recovery

The measurements point to one design for the reclaim recovery of Nodus Jobs and TrainingJobs, which we recommended and which has since shipped for GPU Jobs as a Beta feature behind the per-organization flag node.gpu_snapshot_jobs, through PRs #728 and #729: a notice-triggered snapshot tier on top of file checkpoints. The cadence rule of the second follow-up below, which counts only the reclaims a notice does not cover, has since shipped too, through PR #756, behind the per-organization Beta flag sched.cadence_unannounced. File checkpoints stay the recovery state, at the job's own cadence. When a reclaim comes with a notice, either a preemption notice from the capacity's owner or a reclaim that Nodus starts itself to drain a node or make room for reserved work, the node takes one stopping snapshot through the executor and uploads it. The replacement attempt restores from that snapshot and falls back to the newest file checkpoint, with a typed reason, if the snapshot does not match the new host, arrives too slowly or fails to restore. Periodic snapshots are offered only to jobs that have no checkpoint code, at the period the cost model picks. Before promising a snapshot, the node checks that no process in the attempt maps the GPU device outside a CUDA context, as the compile workers did. The file checkpoint keeps the job's contract, and the snapshot only adds a faster path when there is time to take it.

10Scope

Each arm of the drill ran once and was killed at one point of its protection cycle, 30 s after its fourth protection point, and the replay of every kill instant rests on only 3 or 4 protection points. There was one model size, whose snapshot follows PyTorch's 7.14 GB reserve. Larger models, longer sequences, multi-GPU jobs and trimming the allocator's free reserve before a snapshot were not measured. The copies went host to host over the private network in one stream at about 0.56 GB/s, not through object storage, because the uploader for fleet snapshots did not exist yet when these drills ran. It has since shipped, through PRs #681 and #686, with restores on another node behind the per-organization Beta flag node.gpu_snapshot_fleet. The reclaim was a kill of the container on a host that stayed up, and the recovery host already had the image, so a fresh node pulling the image was not modelled. Restores onto the other GPU used the CUDA 13.0 forward-compatible user-mode driver, and a restore with the host's own driver was not tried. The trainer had no data loader worker processes, no open data files, no NCCL and one GPU. The two drill arms ran at the same time in opposite directions, so each host's training shared its network and disk with the other arm's incoming copies. The cost model is computed, not measured. All 17 runs come from one session on two hosts, and no independent verifier has rerun them.

11Related work

Periodic checkpointing and its best interval are classic. Young [8] gave the first-order optimum interval, the square root of twice the checkpoint cost times the mean time between failures, and Daly [9] refined it, which is the model used above. For deep learning, CheckFreq [10] tunes the checkpoint frequency at run time and splits each checkpoint into a fast copy of the state and a background write, which shrinks the kind of stall that made periodic snapshots expensive here. Gemini [11] keeps checkpoints of distributed training in the CPU memory of other machines, so that recovery from a failure does not wait for remote storage. These systems save the training state from inside the framework, so the job has to take part.

Transparent checkpointing saves a whole process without the program's help. DMTCP [12] does so for distributed user-space programs, and CRAC [13] extends transparent checkpoint and restart to CUDA programs. CRIU [5] checkpoints Linux processes and containers, NVIDIA's cuda-checkpoint [6] moves a process's GPU state into host memory so that CRIU can save it, and CRIUgpu [14] combines the two for GPU workloads. Singularity [15] uses transparent checkpointing to preempt and migrate deep learning jobs across a fleet. Our reports 033 and 034 measured the same stack restoring inference engines. What this report adds is a measurement of a GPU memory snapshot against the file checkpoint for one training job through the Nodus executor: its exactness, its stall, its lost work in a reclaim drill and the case of a reclaim notice.

12Follow-up 1, 8 November 2026: releasing the allocator reserve before each snapshot

Question. A snapshot is large because it holds PyTorch's whole 7.14 GB memory reserve, of which 1.46 GB is live state. In a follow-up on two new GPU hosts, the trainer released the free part of that reserve in its own process at a step boundary just before each snapshot, with torch.cuda.synchronize and torch.cuda.empty_cache, and then made no CUDA call until the snapshot had frozen and thawed it. The test asked for each snapshot through a small control file: it wrote a request, the trainer answered at its next step boundary after the release, and the snapshot was taken while the trainer waited. We also tried PyTorch's expandable segments allocator setting alone and together with the release, and reran the file-checkpoint arm in the same session. The trainer, the job, the 60 s cadence and the kill at 270 s were as above, on the same stack, with the runtime test from a later commit of the same unmerged research branch. The new hosts ran at the first hosts' speed, 0.2263 and 0.2258 s per step against 0.2260 s over the first 300 steps, and their 300-step runs matched the reference bit for bit.

Smaller and faster to restore, still behind file checkpoints. Releasing the reserve left 2.28 to 2.43 GB reserved, not the 1.46 GB of live state, because segments that still hold live tensors stay mapped, and the snapshot fell from 8.53 GB to 3.69 GB. The release itself took 74 to 88 ms. The restore reached its first new step in 3.74 s against 7.24 s, faster than the file resume's 4.90 s, and in the drill the released arm lost 68 steps against 122 for the file checkpoint. That drill was lucky: the newest snapshot's copy had become whole 2.1 s before the kill. Replaying every kill instant gives 178.1 steps of expected lost work against 134.5 for file checkpoints, because each snapshot still stopped training for 9.1 s at the median against 1.44 s for a torch.save, 6.3 times as long, and still took 22.9 s to be safe on the other host against 4.9 s. Through one reclaim the released arm needed 13.9% more GPU-seconds than the uninterrupted job, against 9.6% for the file arm rerun in the same session. Expandable segments alone changed little, 8.30 GB and a 12.4 s stall, and together with the release they gave 3.45 GB against 3.69 GB at a 9.0 s stall. All 5 snapshot restores of this session, each on the other host, and the file resume continued bit for bit, with equal SHA-256 digests at steps 1000 and 1800, and the training that continued on each original host after its snapshots matched the reference up to the kill.

Where the stall goes. In each released snapshot the trainer was held from its answer until the freeze thawed it, 8.3 s at the median, and the first step after the thaw took 0.64 to 0.76 s longer than usual. A separate run on host A timed cuda-checkpoint's two moves on the live trainer: 1.12 to 1.16 s to move the GPU state into host memory after a release, against 2.82 to 2.91 s with the full reserve, and 0.54 to 0.55 s to move it back, against 1.22 s. CRIU's own log puts the dump of a released image at 1.84 to 1.95 s, against 3.94 to 3.97 s for the 8.30 GB images. In that separate run, where the state was back on the GPU before the trainer went on, the next step took the usual 0.227 to 0.228 s, so regrowing the released reserve costs no measurable time, and the slow first step after a real snapshot matches moving the state back to the GPU after the thaw. What is left of each hold after the move out and the dump is about 5.3 s at the median, 4.2 to 8.9 s per snapshot, and about 4.3 to 4.9 s without the release. That remainder lies in the rest of the node runtime's checkpoint path, the pause, containerd's checkpoint call, the start-up of runc and CRIU and the resume, which no allocator setting touches. It is a remainder, not a measurement inside the checkpoint path. The trainer also thawed 0.8 to 2.9 s before each checkpoint call returned.

A snapshot on notice gets better. With the release, one held stopping snapshot taken 150 s in, as on a reclaim notice, again lost no steps in either direction. Its 3.92 GB took 11.2 and 15.6 s from the request to the end of the snapshot and 12.6 and 13.6 s to copy, so it needs a notice of 23.8 to 29.2 s instead of about 37 s, and the first new step came 27.6 to 33.1 s after the freeze instead of 43.7 to 44.2 s, counting only the snapshot, the copy and the restore. The test scripts' waits made it 54.8 to 56.7 s by the wall clock. Both restores were bit-identical. The cost model above, with these numbers, keeps file checkpoints ahead of periodic snapshots at every rate, 3.1% against 7.8% for released snapshots at one reclaim per hour, 1.5% against 3.7% at 0.25 and 6.8% against 17.2% at 4.

What changes in the recommendation. We recommend, and it is not built, that the snapshot on notice release the reserve first, at a step boundary, through a take point for Jobs and TrainingJobs that reuses the state-file protocol of Nodus Functions. The node writes a take request, the worker releases the cache at its next step boundary, as the Functions SDK already does before each offer, answers with its memory figures and makes no CUDA call until it is thawed or told the take is over, and a restored copy continues the same way. Expandable segments should not be set, because they change the customer's allocator for little gain. Periodic protection stays on file checkpoints, and the roughly 5 s left in the node runtime's checkpoint path is the next thing to profile, before any more allocator work.

Scope of the follow-up. There was one model and one job, whose fragmentation after the release may differ on other models, and one drill per variant at one point of its cycle, with the replay resting on 3 or 4 protection points each. The remainder in the checkpoint path is what is left after subtracting measured parts, not instrumented, and the two moves come from a separate run with the default allocator. The paired drills ran at the same time and copied in both directions at once. A step-boundary hook that needs no change to the trainer was not built or measured, because this trainer checked for requests at the end of its own loop. No independent verifier has rerun these measurements.

13Follow-up 2, 9 November 2026: the file-checkpoint cadence once every reclaim notice gets a snapshot

Question. The sections above found that one stopping snapshot taken on a reclaim notice loses no steps, and recommended it on top of file checkpoints. The snapshot on notice for GPU Jobs is now in the product, behind the per-org Beta flag node.gpu_snapshot_jobs (PRs #728 and #729, merged). A job stopped this way loses nothing to a noticed reclaim, so its file checkpoints only protect it against failures that come without notice. The product still sets a Job's file-checkpoint cadence from every reclaim, noticed or not, with the classic checkpoint-interval formula of Young and Daly [8] [9], the square root of twice the checkpoint cost times the mean time between failures. This follow-up asks how much more training one GPU gets done on reclaimable capacity when the cadence counts only the failures a notice does not cover. That cadence rule has since shipped in the Nodus platform behind the per-organization Beta flag sched.cadence_unannounced, off by default, through PR #756, merged on 9 November 2026.

Setup. One GPU trained a 302.7 million parameter GPT, the character-level trainer of the sections above scaled to 24 layers of width 1024, compiled with torch.compile and deterministic, at 0.2545 s per step. A file checkpoint of its model, optimizer, step and random generators was 3.63 GB and stalled training for 4.79 s on average, 4.54 to 5.19 s across runs. A cold start on a replacement copies the newest checkpoint file, drops the page cache, loads it and compiles again, and took 55.1 s on average from the kill to the first new step. The stop snapshot used cuda-checkpoint and CRIU 4.3 on the trainer as a bare host process, not through the product's runc executor, and was restored on the same GPU. Before each snapshot the trainer released PyTorch's free reserve at a step boundary, as in follow-up 1, and the image was 5.90 GB at the median.

Events and compressed time. Each run lasted 1500 s from its first step and replayed one of 3 seeded schedules. Noticed reclaims came every 300 s on average, each with a lead of 30 to 120 s before the machine was taken, and unannounced kills came every 1800 s on average. Time was compressed 12 times. The gaps stand for one noticed reclaim an hour and one unannounced loss every 6 hours, but every cost is the GPU's real cost, the 4.79 s save, the snapshot cycle and the 55.1 s cold start. Each cost therefore weighs 12 times more against the gaps than it would at real rates, so the absolute goodputs are those of a very hostile fleet. The differences between the arms and the cost of each event carry over, and the model below rescales them to real rates. Goodput is the steps a run had trained at the end of its window over the 5834 steps an uninterrupted run trained in the same 1500 s.

Four arms. A is today's product without the flag. It writes a file checkpoint every 60 s, and on a notice an urgent one, after which the process keeps training until the machine is taken and the replacement starts cold from the newest file. B adds the snapshot on notice: after the urgent checkpoint the process stops in one snapshot and continues from it. C is B with files every 131 s, the classic interval for the unannounced rate alone, from the 4.77 s checkpoint measured before any arm ran and the 1800 s mean gap. D takes the snapshot on notice and writes no files. After an unannounced kill, B and C restore the paired snapshot while no newer file exists and otherwise start cold from the newest file, and D restores its newest snapshot. A's 60 s is close to the 50 s the same rule gives when it counts every failure.

The snapshot on notice is the big win. B reached 0.848 of the uninterrupted run against 0.724 for A, 12.4 points more on average and +15.6, +2.4 and +19.1 by seed, or 3.296 steps per GPU-second against 2.816. The snapshot arms lost 0 steps on all 36 notices they restored, 12 per arm, and every snapshot was whole on the replacement's disk at least 15.1 s before its reclaim. From the notice to the first new step took 22.5 s at the median in B and C, with the 4.72 s urgent checkpoint, and 16.4 s in D. A lost 22 to 179 steps at each of its 9 reclaims, 101.8 on average, and then paid a cold start. The urgent checkpoint without a stop still loses work, because the process trains on after it until its machine is gone, and that training is lost. In seed 2, A's notices at 937.3 s and 954.8 s fell on one machine, so A paid one reclaim for both, which is why B gained only 2.4 points there.

Setting the cadence from unannounced failures adds a smaller gain. C reached 0.860, 1.2 points over B on average, with +3.5, +1.7 and -1.6 by seed. The longer cadence cut the periodic file-checkpoint stall from 6.56% to 2.67% of the window, 3.9 points, and gave 2.6 points back in work redone after the unannounced kills, where C lost 304 and 300 steps against 143 and 3 for B. The gain is noisy. Each arm met only 2 unannounced kills, and in seed 3 B's kill came 3 steps after a file checkpoint while C's came 300 steps after one. The model below, calibrated with the measured costs, expects +3.0 points at this failure mix, inside the per-seed range of -1.6 to +3.5.

Without file checkpoints, an unannounced kill is catastrophic. D reached 0.949 in seed 1, which has no unannounced kill, the best of any run, but 0.738 and 0.537 in the two seeds with one. Each unannounced kill sent D back to its last notice snapshot, 1377 and 2384 steps earlier, and D's mean of 0.742 fell below B's. A snapshot on notice covers only the failures that come with a notice, so file checkpoints stay, at the cadence the unannounced failures set.

Every step exact. Every step line any arm logged, 61,588 with replays included, has the uninterrupted run's loss bits, and all 569 fingerprints of the parameters, the optimizer state and the random number state match, through restores from files and from snapshots.

Break-even and product scale, from the model. With the cadence held at 131 s, C stops beating B once unannounced kills come more often than about one per 534 s of compressed time, 3.4 times the design rate of one per 1800 s. Plain Young and Daly costs put that break-even at 821 s, the product of the two cadences over twice the checkpoint cost. At real rates, with the product's own defaults of a 1 minute write, the 5 minute to 2 hour clamp and the 10% duty floor, and with one noticed reclaim an hour and one unannounced loss a day, the model puts the cadence at 644 s today and at 3220 s when it counts only unannounced losses. File checkpoints plus redone work then fall from 9.7% to 3.7% of GPU time. Both figures assume the snapshot on notice, and none of these numbers is a measurement.

What changes in the product. We recommend that a Job which will get the snapshot on notice have its cadence set from the failures a notice does not cover, the hazard of heartbeat loss plus the reclaim hazard times the share of reclaims whose notice came with enough lead and whose snapshot was restored. That share should start at 0 and grow only from measured outcomes, so nothing changes without evidence, and the unannounced hazard should stay measured, not zero, because the relaxed cadence loses once unannounced failures become frequent. The rule should follow the snapshot path, not notices in general, because an urgent checkpoint without a stop still loses the training done after it. This cadence rule is now in the product behind the per-organization Beta flag sched.cadence_unannounced through PR #756.

Scope of the follow-up. There was one workload, the 302.7 million parameter compiled trainer, on one GPU, and the gain from the cadence scales with the checkpoint cost over the cadence. Time was compressed 12 times, and only the model's real-rate figures use real rates. Each arm met only 2 unannounced kills, so the measured gain of C over B rests on where those kills landed. Copies were local. The snapshot's upload and download was a local copy and sync, 5.1 s for 5.90 GB, and a cold start copied the checkpoint file locally. At the 4.6 s up and 5.4 s down for a 2.30 GB image that a GPU test of PR #729 measured through object storage, each notice would take about 20 s more. The snapshot ran through cuda-checkpoint and CRIU on a bare host process, not the product's runc executor, and the replacement was the same GPU, at once. In seeds 2 and 3 two notices came 17.5 and 4.0 s apart. A paid one reclaim for both, while the snapshot arms took a second snapshot of the restored process, which cost B and C 20.5 to 22.4 s, 1.4 to 1.5 points, and D 16.1 to 16.3 s in each of those seeds. The first A run of seed 1 hit a bug in the test driver that let a reclaim meant for a replaced machine kill the replacement, and scored 0.690. It is kept on record, and A seed 1 was rerun after the fix. No independent verifier has rerun these measurements.

14Conclusion

A GPU memory snapshot is an exact way to move a training process off a GPU that is being reclaimed. On two GPUs it continued bit for bit in all 4 restores, with no checkpoint code in the trainer. It is not a cheap way to protect a job that already checkpoints. It holds PyTorch's whole memory reserve, 8.53 GB against a 1.02 GB file, and each periodic snapshot stopped training for 12.9 s against 1.36 s, so in a reclaim drill the job needed 28.8% more GPU-seconds to finish against 8.7% with file checkpoints. Snapshots win in two places: a single snapshot on a reclaim notice, which lost no steps and resumed training on the other host 44 s after the freeze, and jobs whose restart is expensive, where a compiled trainer reached its first step in 6.6 s against 28.1 s. For a fleet, that makes snapshots a notice-triggered tier on top of file checkpoints, so that reclaimable and idle GPUs can train with less work lost. Whether this holds for larger models, multi-GPU jobs and copies through object storage is the next measurement. Follow-up 1 found that releasing PyTorch's free reserve before each snapshot halves its size and the restore time and shortens the notice it needs to 23.8 to 29.2 s, which makes the snapshot on notice better, while periodic snapshots still lose to file checkpoints. Follow-up 2 found that once every reclaim notice gets a snapshot, file checkpoints only need to cover the failures that come without notice: on one GPU the snapshot on notice added 12.4 points of goodput, a cadence set from unannounced failures alone added a smaller 1.2 points, and a job with no file checkpoints lost 1377 and 2384 steps to its two unannounced kills.