One GPU,
More Work.

1Introduction

A single accelerator increasingly serves multiple models, teams, or tenants. NVIDIA Multi-Process Service (MPS) enables concurrent CUDA execution, but operators must still compose memory enforcement, admission, lifecycle, telemetry, and serving policy. Nodus instead treats each partition as an infrastructure object with an explicit size, tenant identity, engine, and control-plane contract.

We ask a simple experimental question: under the same GPU, model, memory budgets, engine configuration, and request trace, does this software-defined arrangement deliver more useful inference work than native MPS?

Across both tested loads, it does. The result is consistent in throughput and tail latency, while remaining bounded to one seed, GPU, model, traffic family, and two-tenant configuration.

2Experimental setup

Both arms ran on one NVIDIA A100-SXM4-40GB with Qwen/Qwen3-1.7B-FP8 and vLLM v0.30.0. Each used two independent tenant engines, 18 GiB hard caps per tenant, a 16.2 GiB vLLM target per engine, 8,192 maximum batched tokens, 64 maximum sequences, and prefix caching disabled.

The native arm ran both clients under MPS with a 100% active-thread cap per client. The Nodus arm ran two resident software partitions. Inputs ranged from 1,023 to 1,026 tokens; every request produced exactly 256 output tokens. Each offered-load window lasted 120 seconds and then drained fully.

We report completed output throughput and SLO-qualified output throughput. The latter prevents a large post-window drain from appearing healthy when request latency has collapsed.

3Results

At 12 requests per second, both systems completed all 1,440 offered requests. Nodus increased completed output throughput from 2,627.58 to 3,021.69 tokens per second, reduced p99 TTFT from 81.2 to 69 milliseconds, and moved latency-target attainment from 86.96% to 100%.

At 16 requests per second, both systems completed all 1,920 requests after drain. Nodus increased completed output throughput from 3,325.05 to 3,826.58 tokens per second and SLO-qualified throughput from 1,212.26 to 1,421.02 tokens per second.

Stress-regime caveat. At 16 requests per second, both systems missed the experiment’s 99% latency-attainment target. Nodus behaved better past the knee; neither configuration met the target.

4System interpretation

Mean SM activity rose from 78.78% to 91.0% at 12 req/s and from 82.78% to 95.2% at 16 req/s. In these runs, higher activity became useful inference work rather than overhead.

The broader product thesis follows from the abstraction boundary. VMware made physical compute programmable through a resource abstraction and a control plane for placement, isolation, lifecycle, and operations. GPU infrastructure needs the equivalent: a software-defined partition with explicit memory, compute policy, tenant identity, lifecycle, telemetry, and a portable control-plane contract.

NVIDIA MIG remains an option for hardware-enforced fault and memory-bandwidth isolation on supported GPUs. This experiment did not include a MIG arm and does not claim to beat MIG. A unified control plane should place hardware and software mechanisms behind one operational model and choose the appropriate isolation tier.

5Conclusion

Software-defined partitioning did more than reproduce native MPS behavior with a control plane. On this workload, it delivered more completed inference work and lower tail latency from the same GPU. The result motivates treating a GPU partition as a first-class infrastructure primitive.

6Reproducibility statement

All reported values come from the machine-readable result files published with this report. The comparison fixes the physical GPU, model, tenant count, memory caps, engine limits, request trace, output length, offered window, and drain policy across both arms.