LLM Serving Benchmark: NVIDIA DGX Station GB300 vs RTX PRO 6000

Most published AI hardware figures come from the people selling the hardware. These are our own measurements, taken on machines in our building with entirely public components, and the script that produced them is free to download so you can run the identical test on your own hardware and compare.

The benchmark runs NVIDIA’s own vLLM container (nvcr.io/nvidia/vllm:26.09-py3, vLLM 0.29.0) and vLLM’s built-in serving benchmark against two open-weight models from OpenAI, gpt-oss-20b and gpt-oss-120b, pinned to exact revisions. The engine, the models and every setting are fixed, so differences between machines come from the hardware rather than the software. It is published under the MIT licence with checksums.

We have measured two machines so far: the ASUS ExpertCenter Pro ET900N G3, which is the NVIDIA DGX Station GB300 architecture on a desk and the Enterprise Scale tier of the ASUS Agent Computer range (ASUS calls it a deskside agentic AI supercomputer), and a conventional high-end workstation fitted with a single NVIDIA RTX PRO 6000 Blackwell Server Edition.

What the benchmark measures

For each model it starts a vLLM server in a temporary container and drives it at 1, 8, 32 and 128 simultaneous users. Every request has a random 1,024-token prompt and generates a full 1,024-token reply, so replies are always the same length. At every concurrency level it runs a discarded warm-up, then measures three times and reports the median. The warm-up matters: the first time vLLM meets a larger batch it compiles and tunes kernels, which made early runs 3 to 6% slow at 32 users and above. The benchmark client runs inside the same container as the server, so no network hop sits in the measurement.

The two machines

Both ran the same container image digest and the same model revisions. The RTX PRO 6000 figures were repeated across three independent sessions on the 120b model and agreed within 1% on throughput, so they are stable rather than a single lucky run.

Results

All figures are the median of three runs with zero failed requests in every run.

* The three runs disagreed by more than 10% on this figure. Treat it as indicative. TPOT of 12.86 ms means each of the 128 users receives about 78 tokens per second; 44.00 ms means about 23.

Models also loaded faster from a warm cache on the GB300: 81 seconds for gpt-oss-20b and 111 seconds for gpt-oss-120b, against 123 and 154 seconds on the workstation.

The finding that matters

The GB300’s advantage widens as load increases: 2.3 times a single RTX PRO 6000 at one user, 3.4 times at 128 users on the 120b model. That is memory bandwidth asserting itself, roughly 7.1TB/s of HBM3e against roughly 1.8TB/s of GDDR7. At one user the workload does not saturate either card; at 128 it does, and bandwidth decides the outcome. For a business that is the realistic case, many simultaneous users rather than one, so this is a better guide to the difference between the machines than any single headline number.

In a separate internal run under full load the GB300 drew around 980W of its 1,300W limit at 59°C, with no thermal throttling and about 9.5 seconds of software power capping across a whole run. Those readings come from that internal run, not from the published benchmark results, which carry no power or thermal data.

Read the figures with these caveats

• The RTX PRO 6000 was measured under WSL2, not bare-metal Linux. WSL2 adds a virtualisation layer, and a native Linux install on the same card would likely be somewhat faster. The results declare this themselves (platform=WSL2).
• Time to first token on the WSL2 machine is noisy. Most of its latency figures carry the asterisk. Its throughput is solid, agreeing within 1% across three sessions, but treat its latency as indicative. The GB300’s native-Linux run only flagged TTFT at 128 users.
• Only two machines have been tested. Any GPU suggestions in the download’s documentation are based on memory capacity alone, not measurement.
• The 128-user figure is a synthetic upper-load test with fixed-length replies. It shows the ceiling, not a chat workload.
• The numbers are not comparable with Ollama, LM Studio or llama.cpp results, which use different engines and settings.
• At 128 users the first of the three GB300 runs was still 5 to 7% slower than runs two and three, with roughly double the median TTFT. The median is unaffected.

Run it on your own hardware

One download covers Linux and Windows. The Windows launcher sets up WSL2 and runs the same Linux script inside it, so the two sets of results are directly comparable. You need an NVIDIA GPU with about 24GB of memory for gpt-oss-20b or about 80GB for gpt-oss-120b, 32GB of system RAM (64GB recommended for the 120b model), around 110GB of free disk space, an internet connection and nothing else running on the machine. Allow 30 to 60 minutes for a first run of the 20b model including the downloads, and longer for 120b. The script downloads, but does not redistribute, NVIDIA’s container (NVIDIA licence terms apply) and the gpt-oss models (Apache 2.0).

Download the LLM serving benchmark v1.1.0 (zip, 27KB)
SHA-256 of the zip: 597f45558facb7e3fc439a59fb580f13df105dbbf80525290d94fddf1930c192. The zip contains SHA256SUMS for each file inside it.

On Linux

You need Docker with the NVIDIA Container Toolkit installed. Unzip the package, open a terminal in the folder and run:

chmod +x llm-serving-bench.sh
./llm-serving-bench.sh
./llm-serving-bench.sh --model 120b

The first command tests gpt-oss-20b, the second gpt-oss-120b if your GPU has the memory for it. Add --gpu-mem-util 0.85 if the GPU also drives your display, and --help lists every option.

On Windows 11

Install the current NVIDIA Windows driver first, which is the one thing the package cannot do for you. Never install an NVIDIA driver inside WSL. Unzip the package, keep all the files together, and double-click the files in this order:

• install.bat installs and configures WSL2, Ubuntu, Docker and the NVIDIA Container Toolkit. It asks for Administrator rights.
• resume.bat is only needed if Windows asked to restart. It finishes the setup.
• run.bat runs the benchmark and asks which model to test.

A technical note for Windows users: vLLM switches pinned memory off when it detects WSL, but its V2 model runner cannot start without it. The launcher sets VLLM_WSL2_ENABLE_PIN_MEMORY=1, which keeps the same V2 engine a Linux run uses, so the results stay comparable; they record wsl2_pin_memory=forced. The other published workaround runs a different engine and its figures could not be compared, so the script does not use it.

Results are written to a folder beside the script. Share summary.md, which holds the results table and the full configuration, and leave out server.log. Results record your GPU, driver, CUDA version, CPU, operating system and whether the run was under WSL; they do not record your hostname, username or IP addresses.

Send us your results

Email your summary.md to [email protected] and tell us what you ran it on. We will add further machines to this page as we measure them.

If your hardware falls short of what your models need, ask us about remote access to our GB300 evaluation unit to test your own workload before you buy. The ET900N G3 is available in 4TB and 8TB configurations. If the RTX PRO 6000 figures are closer to what you need, we build it into 2U servers with four cards, 4U servers with four or eight cards, and custom AI workstations to order. Call 0151 317 9860 to talk it through.

Questions

Can I compare these figures with my Ollama or LM Studio speeds?

No. Those tools use different engines, quantisations and settings, so their tokens-per-second figures measure something different. Run this benchmark on the same machine instead; it takes under an hour for the 20b model and gives a figure you can set directly against the tables above.

Which GPUs can run the benchmark?

Any NVIDIA GPU with enough memory: about 24GB for gpt-oss-20b, so an RTX 4090 or 5090, RTX 6000 Ada, L40S, A100 or H100, and about 80GB for gpt-oss-120b, so an H100 80GB, H200, RTX PRO 6000 Blackwell, B200 or GB300. Those examples are based on memory size alone; if a model will not fit, the script says so and skips it, which is a result in itself.

Why was the RTX PRO 6000 measured under WSL2 rather than Linux?

Because that is how most of our workstation customers would run it, and we wanted the Windows launcher proven on a real machine. A native Linux install on the same card would likely score a little higher, which is why the comparison is presented as a guide rather than a precise ratio. If you run the Linux version on an RTX PRO 6000, we would like to see the result.