{"id":54,"date":"2026-08-28T23:04:40","date_gmt":"2026-08-28T18:04:40","guid":{"rendered":"https:\/\/n0c.us\/?p=54"},"modified":"2026-08-28T23:04:40","modified_gmt":"2026-08-28T18:04:40","slug":"256-gb-is-not-256-gb","status":"publish","type":"post","link":"https:\/\/n0c.us\/?p=54","title":{"rendered":"256 GB Is Not 256 GB"},"content":{"rendered":"\n

Researched and written in cooperation with Claude.ai<\/a>. Sources and caveats are listed at the end.<\/em><\/p>\n\n\n\n

Here are two ways to put 256 GB of accelerator memory on your floor.<\/p>\n\n\n\n

The first is an NVIDIA DGX-1: eight Tesla V100s at 32 GB each, a petaFLOPS of FP16 tensor throughput, 7.2 TB\/s of aggregate HBM2 bandwidth, and a list price of $149,000 when it shipped. Today it turns up on the secondary market for something like $15,000.<\/p>\n\n\n\n

The second is two NVIDIA DGX Sparks: desk-side GB10 Grace Blackwell boxes, 128 GB of unified LPDDR5x each, joined by a single cable. Eight thousand dollars, new, under warranty.<\/p>\n\n\n\n

Same capacity. The used option brings roughly thirteen times the memory bandwidth<\/strong> and four times the FP16 throughput<\/strong> for twice the money. That looks like an easy call \u2014 and it is, just not in the direction the spec sheet suggests.<\/p>\n\n\n\n

Three things decide this comparison, and none of them appear in a row labelled \u201cmemory.\u201d<\/p>\n\n\n\n

Metric<\/th>2\u00d7 DGX Spark<\/th>DGX-1 (8\u00d7 V100)<\/th><\/tr><\/thead>
Accelerator memory<\/td>256 GB LPDDR5x, unified<\/td>256 GB HBM2, 32 GB\/GPU<\/td><\/tr>
Aggregate bandwidth<\/td>546 GB\/s<\/td>7.2 TB\/s \u2014 13\u00d7<\/td><\/tr>
Per-device bandwidth<\/td>273 GB\/s<\/td>900 GB\/s<\/td><\/tr>
Dense BF16 \/ FP16<\/td>~250 TFLOPS<\/td>~1,000 TFLOPS \u2014 4\u00d7<\/td><\/tr>
FP8<\/td>~500 TFLOPS<\/td>Not supported<\/td><\/tr>
FP4<\/td>~1 PFLOPS dense<\/td>Not supported<\/td><\/tr>
FP64<\/td>~1 TFLOPS<\/td>~62 TFLOPS \u2014 60\u00d7<\/td><\/tr>
Device-to-device link<\/td>25 GB\/s (200 Gb\/s RoCE)<\/td>300 GB\/s NVLink 2.0<\/td><\/tr>
Power draw<\/td>~340 W<\/td>~3,500 W<\/td><\/tr>
CUDA architecture<\/td>sm_121, current<\/td>sm_70, dropped in CUDA 13<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Spark FLOPS are derived from NVIDIA’s \u201c1 PFLOP FP4\u201d headline halved down the precision ladder; treat them as ballpark. Volta’s petaFLOPS is dense \u2014 Volta has no structured sparsity.<\/em><\/p>\n\n\n\n

Capacity is meaningless without precision<\/h2>\n\n\n\n

Volta’s tensor cores speak FP16. That is the entire list \u2014 no BF16, no FP8, no FP4. Blackwell’s fifth-generation tensor cores speak all of them natively.<\/p>\n\n\n\n

This is usually filed under performance, but it is really a capacity<\/em> multiplier. A parameter stored at FP4 occupies a quarter of the bytes it occupies at FP16. Same memory, four times the model.<\/p>\n\n\n\n

Configuration<\/th>Pooled memory<\/th>Max parameters<\/th><\/tr><\/thead>
1\u00d7 DGX Spark<\/td>128 GB<\/td>~217B at NVFP4<\/td><\/tr>
DGX-1, all eight GPUs<\/td>256 GB<\/td>~107B at FP16<\/td><\/tr>
2\u00d7 DGX Spark<\/td>256 GB<\/td>~435B at NVFP4<\/td><\/tr>
4\u00d7 DGX Spark<\/td>512 GB<\/td>~870B at NVFP4<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Assumes ~85% of memory available for weights after KV cache, activations, and per-device context overhead.<\/em><\/p>\n\n\n\n

Read the first two rows against each other. A single Spark, with half the DGX-1’s memory, holds roughly twice the model.<\/strong> Capacity parity at 256 GB still loses, because precision is a 4\u00d7 lever and Volta cannot pull it.<\/p>\n\n\n\n

It compounds from there, because models increasingly ship in<\/em> those formats. DeepSeek-V3 is natively FP8. gpt-oss ships as MXFP4. On a V100 you must upcast them before you can run them at all \u2014 so a natively-FP8 checkpoint arrives twice its own size, and an MXFP4 one arrives four times.<\/p>\n\n\n\n

There is an escape hatch, and it is worth being honest about it: llama.cpp builds cleanly for sm_70<\/code>, and 4-bit GGUF weights are just bytes, so a 405B model at Q4 (~203 GB) does technically fit. Two things spoil it. --split-mode layer<\/code> walks the GPUs in sequence, so you get roughly single-GPU<\/em> bandwidth rather than the aggregate 7.2 TB\/s \u2014 landing near 4 tok\/s, no better than the Spark pair. And bitsandbytes 4-bit requires compute capability 7.5+, so QLoRA \u2014 the standard route to fine-tuning a large model on modest hardware \u2014 does not run on Volta at all.<\/p>\n\n\n\n

Bandwidth is the thing you actually feel<\/h2>\n\n\n\n

Having established that the Spark holds more, here is where it loses badly, and it deserves to be stated plainly rather than buried.<\/p>\n\n\n\n

Token generation is memory-bound. Every token requires streaming the model’s weights out of memory, so decode speed is governed almost entirely by bandwidth divided by weight bytes \u2014 a number you can work out before you buy anything.<\/p>\n\n\n\n

\n

70 GB \u00f7 273 GB\/s = 0.26 s \u2192 3.9 tok\/s ceiling<\/strong><\/p>\n\n\n\n

Llama 3.1 70B at FP8 on one Spark. Measured single-stream: 2.7 tok\/s<\/strong>. The envelope predicts reality to within the efficiency factor, which is exactly the point \u2014 nothing about this number is a surprise if you do the arithmetic first.<\/p>\n<\/blockquote>\n\n\n\n

For scale: that is 546 GB\/s of aggregate bandwidth on the Spark pair against 7,200 GB\/s on the DGX-1. Adding the second Spark does not fix it, either. Tensor parallelism across a 25 GB\/s link means an all-reduce every layer over a pipe twelve times slower than NVLink 2.0. The second node buys capacity, not latency.<\/p>\n\n\n\n

Two caveats keep this from being a rout. Batch serving closes much of the gap \u2014 a Spark reaches roughly 695 tok\/s aggregate across 256 concurrent streams, because at that point you are amortising each weight read across many tokens. And mixture-of-experts models sidestep the problem structurally: with few active parameters per token, a 671B MoE moves less data per token than a dense 70B.<\/p>\n\n\n\n

The scaling ceilings invert<\/h2>\n\n\n\n

Ask how far each architecture goes and you get opposite answers depending on whether you mean \u201cby design\u201d or \u201cin practice.\u201d<\/p>\n\n\n\n

The structural difference is the size of the unit you scale from<\/em>. A DGX-1’s unit is an eight-GPU NVLink domain: 256 GB at 7.2 TB\/s, coherent internally, with four EDR InfiniBand ports \u2014 400 Gb\/s per node, double a Spark’s \u2014 sited at the corners of the cube mesh for GPU-to-GPU RDMA. A Spark’s unit is one GPU. There is no fast intra-node domain, because there is only one accelerator in the box, so every parallelism dimension beyond a single GPU crosses the slow fabric.<\/strong><\/p>\n\n\n\n

By design, it is not close. NVIDIA built the DGX-1 for POD deployments \u2014 nine per rack, reference architectures tested to six PODs \u2014 and proved it at 124 nodes with SATURNV, a fat-tree cluster that hit 4.9 petaFLOPS peak. The Spark’s supported ceiling is four nodes via the Cluster Assistant, reaching eight with breakout cables. The DGX-1 scales roughly fifteen times further.<\/p>\n\n\n\n

In practice, that ceiling is historical. Each node you add is 3.5 kW of out-of-support hardware needing an InfiniBand switch, a rack, 240 V service, and real cooling.<\/p>\n\n\n\n

Pooled memory<\/th>DGX-1 path<\/th>Spark path<\/th><\/tr><\/thead>
256 GB<\/td>1 node \u00b7 ~$15k \u00b7 3.5 kW<\/td>2 nodes \u00b7 $8k \u00b7 340 W \u00b7 one cable<\/td><\/tr>
512 GB<\/td>2 nodes + IB switch \u00b7 ~$35k \u00b7 7 kW<\/td>4 nodes + switch \u00b7 $16k \u00b7 ~700 W<\/td><\/tr>
1 TB<\/td>4 nodes \u00b7 ~$65k \u00b7 14 kW<\/td>8 nodes \u00b7 $32k \u00b7 1.4 kW<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

At every tier the DGX-1 delivers roughly 13\u00d7 the bandwidth for about 2\u00d7 the money and 10\u00d7 the power. If bandwidth is what you need and you already own a datacenter, that is genuinely good value per GB\/s. If you do not, the 1 TB tier arrives at 14 kW \u2014 about $24,500 a year in electricity alone.<\/p>\n\n\n\n

Both architectures have a point past which you should buy something else. For Sparks it is around four to eight nodes, after which four RTX PRO 6000 Blackwell cards in one chassis give you 384 GB at roughly six times the per-device bandwidth with no fabric in the critical path. For the DGX-1, in 2026, that point is one node.<\/p>\n\n\n\n

Sticker price is the smallest line item<\/h2>\n\n\n\n

Run both machines for three years at $0.20\/kWh and the bargain evaporates \u2014 before accounting for cooling, the rack, or the electrical work.<\/p>\n\n\n\n

Configuration<\/th>Capital<\/th>Electricity, 3 yr<\/th>Total<\/th><\/tr><\/thead>
DGX-1 \u00b7 8\u00d7 V100<\/td>~$15,000<\/td>$11,000\u201318,000<\/td>~$26,000\u201333,000<\/td><\/tr>
2\u00d7 DGX Spark<\/td>$8,000<\/td>$1,000\u20132,000<\/td>~$9,000\u201310,000<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Excludes cooling overhead, rack, and 240 V service.<\/em><\/p>\n\n\n\n

The \u201ccheap\u201d machine costs roughly three times the expensive one, and the crossover happens inside the first eighteen months. Power alone on the DGX-1 exceeds its own purchase price.<\/p>\n\n\n\n

But the money is not the strongest argument. This one is: hardware depreciates on a slope; platforms depreciate on a cliff.<\/strong><\/p>\n\n\n\n

CUDA 13 removed offline compilation and library support for Volta. TensorRT-LLM, TGI, and Triton have all dropped sm_70<\/code> from current containers, leaving vLLM as roughly the last major serving stack that still accepts it. FlashAttention-2 requires sm_80<\/code>. A DGX-1 bought today is pinned to CUDA 12.x permanently, and the gap between what it can run and what everyone else is running widens every month. No amount of money closes it, because there is nothing to buy.<\/p>\n\n\n\n

When the old box is still the right answer<\/h2>\n\n\n\n

None of this makes a DGX-1 worthless, and if one is already racked and paid for, the analysis changes considerably.<\/p>\n\n\n\n

It remains about sixty times<\/strong> a Spark at FP64, so any classical HPC \u2014 CFD, quantum chemistry, climate \u2014 still belongs on it and nowhere near a Spark. It is a perfectly good CI machine and a fast dataset-preprocessing box. It handles FP16 fine-tuning well for anything that fits in 32 GB shards, provided you stay on CUDA 12.x and vLLM. And its single-node bandwidth genuinely beats a small Spark cluster for dense-model inference.<\/p>\n\n\n\n

If power is free to you \u2014 a university lab, a colo with flat-rate power \u2014 a large part of the cost table simply evaporates and the calculus shifts back meaningfully.<\/p>\n\n\n\n

The mistake was never using one. It is buying more of them, or building a roadmap on a platform that has already been dropped by its own compiler.<\/p>\n\n\n\n

Five rules for cheap silicon<\/h2>\n\n\n\n
    \n
  1. Multiply capacity by precision before comparing gigabytes.<\/strong> A machine that speaks FP4 has four times the effective memory of one that speaks only FP16. The spec sheet will not do this multiplication for you.<\/li>\n\n\n\n
  2. Predict decode speed on an envelope.<\/strong> Weight bytes divided by memory bandwidth gives you tokens per second to within an efficiency factor. If that number disappoints you on paper, it will disappoint you in the rack.<\/li>\n\n\n\n
  3. Price the electricity before the hardware.<\/strong> Three years at 3.5\u00a0kW costs more than the machine drawing it. Any comparison that stops at purchase price is measuring the wrong thing.<\/li>\n\n\n\n
  4. Check the compiler support matrix before the benchmarks.<\/strong> Benchmarks tell you what a machine does today. The CUDA deprecation notice tells you what it will be allowed to do in two years.<\/li>\n\n\n\n
  5. Ask what the scaling unit is, not just how far it scales.<\/strong> Eight GPUs behind NVLink and one GPU behind Ethernet are different starting points, and the difference shows up in every parallelism decision you make afterward.<\/li>\n<\/ol>\n\n\n\n

    Caveats<\/h2>\n\n\n\n