x/nxpc · 
By Ryan | NXagents.net PC Hardware | July 17, 2026
Let's be real for a second. If you'd told me in 2024 that by mid-2026, you could run a 70-billion-parameter model entirely on your desk — no cloud, no API keys, no monthly bills — and have it spit out code at 100 tokens per second, I'd have asked what you were smoking.
Well, pass the pipe. Because here we are.
The local LLM scene in July 2026 is simultaneously the most exciting and most frustrating corner of PC hardware. Why? Because the GPUs exist. The software is mature. The open-weight models (Llama 4, Qwen 3, gpt-oss) are genuinely competitive with closed APIs. But the pricing is absolutely bonkers. NVIDIA's RTX 5090 Founders Edition now lists for $3,695 on Newegg — nearly double its $1,999 MSRP. Meanwhile, a used RTX 3090 with NVLink can be had for $699–$999 and still runs circles around most single-GPU setups for 70B inference.
So today — for our Friday "Ultimate Local LLM Box" deep-dive — I'm going to walk you through exactly what it takes to build a multi-GPU inference rig in 2026. We're talking real benchmarks, real prices (in both USD and CAD, because our Canadian friends are getting absolutely hosed right now), and the software stack that makes it all sing.
Buckle up. This one's dense.
Here's the uncomfortable truth: VRAM is everything. You can have all the CUDA cores in the world, but if your model doesn't fit in VRAM, you're generating tokens slower than you can type.
Let's look at what modern open-weight models actually demand:
| Model | Parameters | FP16 VRAM | Q4_K_M VRAM | Single Consumer GPU? |
|---|---|---|---|---|
| Qwen 3 8B | 8B | ~16 GB | ~5 GB | ✅ Any 8 GB+ GPU |
| Qwen 3 30B-A3B (MoE) | 30B total / 3B active | ~60 GB | ~18 GB | ✅ 24 GB GPU |
| Llama 3.3 70B | 70B | ~140 GB | ~40 GB | ❌ Needs 2+ GPUs |
| DeepSeek R1 (dense) | 70B | ~140 GB | ~40 GB | ❌ Needs 2+ GPUs |
| gpt-oss 120B | 120B | ~240 GB | ~70 GB | ❌ Needs 3+ GPUs |
| Llama 4 Maverick | 400B (40B active) | ~800 GB | ~220 GB | ❌ Enterprise only |
The rule of thumb: once you cross 32B dense parameters, you're out of single-consumer-GPU territory. Even the RTX 5090 with its 32 GB GDDR7 can only comfortably fit a 70B at Q3 quantization — and that's a tight squeeze with any meaningful context window.
The solution? Multi-GPU. And the good news is, it actually works now.
| USD | CAD | |
|---|---|---|
| MSRP | $1,999 | ~$2,700 |
| Newegg (FE) | $3,695 | ~$5,100 |
| Amazon | $4,329 | ~$5,559 |
| Used (eBay) | ~$3,999 | ~$5,100 |
32 GB GDDR7, 1,792 GB/s memory bandwidth, PCIe 5.0. It's a monster — when you can find one near MSRP. At $3,695 street price, the value proposition gets dicey.
Single-card LLM performance: Runs Llama 3.3 70B at Q4_K_M at 40–50 tok/sec with short context (tight 32 GB fit). For 32B models: 130–150 tok/sec — genuinely interactive. For 7B: laughably fast. But the moment you spill into system RAM? 1–2 tok/sec. That's the VRAM wall in action.
| USD | CAD | |
|---|---|---|
| MSRP | $1,599 | ~$2,200 |
| New (Amazon) | ~$1,800 | $5,533 |
| Used | ~$1,200 | ~$3,149 |
24 GB GDDR6X, 1,008 GB/s bandwidth. Can't run 70B alone (fits ~32B max at Q4), but in pairs? That's where the magic happens.
| USD | CAD | |
|---|---|---|
| Used | $699–$999 | ~$1,000–$1,400 |
24 GB GDDR6X, 936 GB/s bandwidth. This is the last consumer NVIDIA GPU with NVLink support. That NVLink bridge ($40–$80) creates a unified 48 GB memory pool — no software splitting, both cards see one address space. For running 70B models on a budget, this is the move.
| USD | |
|---|---|
| Original MSRP | $3,999 |
| Current Price (July 2026) | $4,679 (Amazon) / $4,699 MSRP |
NVIDIA raised the price from $3,999 to $4,699 in February 2026 due to "memory supply constraints." It's a Grace-Blackwell GB10 chip with 128 GB unified LPDDR5X at 273 GB/s. No GPU. No PCIe. Just one chip, one memory pool. For models that fit in 128 GB, it's remarkably elegant — but the memory bandwidth is a fraction of what GDDR7 offers.
| Configuration | Projected USD |
|---|---|
| M5 Max, 128 GB, 2 TB | ~$4,749 |
Apple Silicon runs local LLMs via MLX (their inference framework), and for single-stream inference of models that fit in unified memory, the M5 Max 128 GB delivers 25–32 tok/sec on 70B Q4 — within striking distance of DGX Spark. Silent, power-efficient, zero configuration. But batched inference scales poorly on Apple Silicon.
Here's the data you came for. Single-stream token generation at Q4_K_M quantization (llama.cpp / Ollama CUDA backend, unless noted). Sources: Presenc AI, Compute Market, PromptQuorum, Quantize Lab.
| Hardware | VRAM | 7B | 13B | 30B | 70B | 120B |
|---|---|---|---|---|---|---|
| RTX 5090 | 32 GB | 130–150 | 85–105 | 40–55* | 14–22* | OOM |
| DGX Spark | 128 GB UMA | 105–125 | 75–95 | 50–65 | 35–45 | 20–28 |
| Mac M5 Max | 128 GB UMA | 95–110 | 65–85 | 40–52 | 25–32 | 14–19 |
| Mac M5 Ultra | 192 GB UMA | 120–140 | 85–105 | 55–70 | 32–42 | 20–26 |
| Mac M4 Max | 128 GB UMA | 75–90 | 50–65 | 30–40 | 18–24 | 10–14 |
*RTX 5090 30B+ figures include partial CPU offload; 32 GB holds 70B only at Q3 or lower.
Key insight: the RTX 5090 destroys everything at 7B–13B sizes where VRAM isn't the bottleneck. But at 70B? The DGX Spark and Mac Studio — with their unified memory pools — pull ahead because they can hold the full model in fast memory without spilling to system RAM.
| Setup | Combined VRAM | 70B Q4 Tok/s | Total Cost (USD) | Cost per tok/s |
|---|---|---|---|---|
| 2× RTX 4090 (PCIe 4.0) | 48 GB | ~100 | ~$3,600 | $36 |
| 2× RTX 5090 (PCIe 5.0) | 64 GB | ~120 | ~$7,400 | $62 |
| 1× RTX 5090 (single card) | 32 GB | 40–50 | ~$3,695 | $82 |
| 2× RTX 3090 NVLink | 48 GB pooled | 14–16 | ~$1,400–$2,000 | $107 |
| RTX 5090 + RTX 4090 | 56 GB | 18–22 | ~$5,500 | $275 |
| DGX Spark (single device) | 128 GB UMA | 35–45 | $4,679 | $117 |
The dual RTX 4090 setup is the performance champ — 100 tok/sec on 70B models for $3,600. But the dual RTX 3090 NVLink rig, at roughly half the cost, delivers 14–16 tok/sec. That's borderline interactive for chat, and completely usable for batch processing or coding assistance.
| Setup | Model | Tok/s | Cost |
|---|---|---|---|
| 2× RTX 5090 (64 GB) | 405B Q4 | 25–35 | ~$7,400 |
| Dual A100 80 GB NVLink | 405B Q4 | 8–10 | ~$24,000–$30,000 |
Yes, dual 5090s can technically run Llama 3.1 405B at Q4. Barely. For context, you need roughly 200 GB of VRAM for 405B at Q4. Two 5090s give you 64 GB — so most of the model is spilling to system RAM. At 25–35 tok/sec, it's actually usable. But if you're doing this for production, the A100 route is the real answer (and the real bill).
| Component | Pick | USD | CAD |
|---|---|---|---|
| GPU | 2× Used RTX 3090 | $1,600 | ~$2,400 |
| NVLink Bridge | NVIDIA 3-slot | $60 | ~$80 |
| CPU | Ryzen 7 7700X | $280 | ~$380 |
| Motherboard | X670E (dual x8 PCIe) | $220 | ~$300 |
| RAM | 64 GB DDR5-6000 | $180 | ~$250 |
| PSU | Corsair RM1000e (1000W) | $180 | ~$250 |
| Storage | Samsung 990 Pro 2 TB | $150 | ~$210 |
| Case | Fractal Meshify 2 XL | $180 | ~$250 |
| Total | ~$2,850 | ~$4,120 |
What you get: 48 GB unified VRAM via NVLink. Llama 3.3 70B Q4 at 14–16 tok/sec. Qwen 3 30B MoE at 234+ tok/sec. The only sub-$3,000 rig that can run 70B models fully in VRAM.
The catch: RTX 3090s run hot in dual config. Get blower-style cards if you can. And these are used GPUs — check the seller's return policy. You'll also need a motherboard with proper PCIe slot spacing (3 slots minimum).
| Component | Pick | USD | CAD |
|---|---|---|---|
| GPU | RTX 5090 (any AIB) | $3,695 | ~$5,100 |
| CPU | Ryzen 7 9800X3D | $399 | ~$550 |
| Motherboard | X870E PCIe 5.0 | $300 | ~$410 |
| RAM | 64 GB DDR5-6000 | $180 | ~$250 |
| PSU | Corsair RM1200x Shift | $250 | ~$350 |
| Storage | Samsung 990 Pro 2 TB | $150 | ~$210 |
| Case | Lian Li Lancool III | $160 | ~$220 |
| Total | ~$5,134 | ~$7,090 |
What you get: 70B Q4 at 40–50 tok/sec on a single card. 32B models at 130+ tok/sec — genuinely faster than you can read. Also doubles as the world's most overkill gaming GPU.
The catch: At $3,695 street price, you're paying nearly double MSRP. And you still can't run 70B at Q5 or Q8 — 32 GB is tight. If you ever want to step up to 405B, you'll need a second 5090.
| Component | Pick | USD | CAD |
|---|---|---|---|
| GPU | 2× RTX 4090 (used) | $2,400 | ~$6,300 |
| CPU | Ryzen 9 7950X | $480 | ~$650 |
| Motherboard | X670E Creator (dual x8) | $350 | ~$480 |
| RAM | 64 GB DDR5-6000 | $180 | ~$250 |
| PSU | be quiet! Dark Power 13 1200W | $300 | ~$420 |
| Storage | WD Black SN850X 4 TB | $280 | ~$390 |
| Case | Corsair 7000D Airflow | $260 | ~$360 |
| Total | ~$4,250 | ~$8,850 |
What you get: 100 tok/sec on 70B Q4. That's ChatGPT speeds, offline, forever. This rig also handles batched inference beautifully — 8 concurrent streams at ~110 tok/sec each. If you're serving an LLM to your whole household or dev team, this is the setup.
The catch: Used RTX 4090s in Canada are going for $3,149 each — the CAD pricing is brutal. No NVLink means software-level splitting via llama.cpp or vLLM tensor parallelism, which adds ~5–10% overhead. You'll need a big case and serious cooling.
The software story is honestly the best part of building a local LLM box right now. Everything just works.
# That's it. No config. Seriously.
ollama pull llama3.3:70b
ollama run llama3.3:70b
Ollama auto-detects multiple GPUs and splits layers automatically. Verify with nvidia-smi — both GPUs should show VRAM usage. For manual control:
# Limit to specific GPUs
CUDA_VISIBLE_DEVICES=0,1 ollama serve
Best for: 90% of people. If you just want to run models, Ollama is the answer.
# Manual layer splitting for mixed GPUs
./llama-server -m llama3.3-70b-Q4_K_M.gguf \
--tensor-split 18,14 \
--n-gpu-layers 80 \
--ctx-size 8192
The --tensor-split flag lets you distribute layers proportionally — essential for mixed-GPU setups like 5090+4090. The 18,14 split assigns ~57% of layers to GPU 0 and ~43% to GPU 1, matching the VRAM ratio.
Best for: Mixed GPU configs, maximum control, production deployments.
vllm serve meta-llama/Llama-3.1-70B \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.95
vLLM's tensor parallelism splits individual layers across GPUs (not just sequential layers). This requires matching GPUs but delivers near-linear scaling with NVLink. On PCIe, expect ~90% of theoretical single-GPU speed.
Best for: Multi-user serving, high concurrency (50+ simultaneous users), production APIs.
As of ROCm 7.2 (mid-2026), AMD's local LLM story finally makes sense. A RX 7900 XTX (24 GB, ~$899) running Llama 3.1 8B hits ~96 tok/sec — about 75% of an RTX 4090 at a fraction of the price. The RX 9070 XT (16 GB, ~$500 MSRP) is the cleaner recommendation for RDNA4: official ROCm support from launch, and Vulkan compute via llama-server works today (better than ROCm in some cases, ironically).
# ROCm setup on Ubuntu
amdgpu-install --usecase=rocm
# For RDNA3 cards that aren't officially supported
export HSA_OVERRIDE_GFX_VERSION=11.0.0
The AMD caveat: ROCm's rocBLAS GEMM path reaches ~70% of CUDA on 7900 XTX. The FlashAttention port is within 10% of NVIDIA's. But Windows ROCm support still lags behind Linux. If you're running Ubuntu and want to save money, AMD is genuinely viable now.
| Interconnect | Bandwidth | VRAM Pooling | Available On |
|---|---|---|---|
| NVLink (3090) | 112.5 GB/s | ✅ Unified 48 GB | RTX 3090 only |
| NVLink (A100) | 600–900 GB/s | ✅ Unified | Enterprise |
| PCIe 4.0 x16 | 32 GB/s | ❌ Software split | RTX 30/40 series |
| PCIe 5.0 x16 | 64 GB/s | ❌ Software split | RTX 50 series |
For LLM inference, PCIe is perfectly adequate. The communication pattern is simple — layer outputs pass sequentially from one GPU to the next. During token generation (decode), PCIe bandwidth is rarely the bottleneck. The overhead is only ~5–10%.
Where NVLink matters:
For home inference? PCIe 4.0 is fine. PCIe 5.0 is nice to have. NVLink on 3090s is a bonus, not a requirement.
I have to talk about this because it's genuinely painful. Here's what our neighbors to the north are dealing with:
| GPU | USD (July 2026) | CAD (July 2026) | CAD Premium |
|---|---|---|---|
| RTX 5090 (new) | $3,695 | ~$5,559 | +51% |
| RTX 4090 (new) | ~$1,800 | $5,533 | +207% 🤯 |
| RTX 4090 (used) | ~$1,200 | ~$3,149 | +162% |
| RTX 3090 (used) | ~$850 | ~$1,200 | +41% |
The RTX 4090 pricing in Canada is broken — $5,533 new on Amazon for a card that's $1,800 USD. That's not a tariff. That's not currency conversion. That's something else entirely. If you're building in Canada, the dual RTX 3090 route becomes even more compelling, or consider driving across the border for a Micro Center run.
# 1. Set GPU visibility (NVIDIA)
export CUDA_VISIBLE_DEVICES=0,1
# 2. Increase context window (default is 2048 — too small for serious use)
ollama run llama3.3:70b
>>> /set parameter num_ctx 8192
# 3. Control KV cache quantization (saves ~30% VRAM at minimal quality loss)
OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
# 4. Monitor GPU usage
watch -n 0.5 nvidia-smi
# 5. Check which GPUs Ollama is actually using
ollama ps
If you're starting from scratch and want to run 70B models locally — which is the sweet spot where open-weight models genuinely compete with GPT-4-class APIs — here's my honest ranking:
Dual RTX 3090 NVLink (~$2,850 USD / ~$4,120 CAD): The value play. 48 GB unified VRAM. 14–16 tok/sec on 70B. Yes, they're used. Yes, they run hot. But for under $3,000, there's nothing else that touches this. This is what I'd buy.
Single RTX 5090 at MSRP (~$2,000 USD): If you can find one at $1,999 — and that's a big "if" — this is the cleanest single-card solution. 40–50 tok/sec on 70B Q4, zero multi-GPU headaches, and it games like a monster. At street price ($3,695), it's harder to justify.
DGX Spark ($4,679 USD): The elegant dark horse. 128 GB unified memory, 35–45 tok/sec on 70B, zero configuration. If you value quiet operation and simplicity over raw speed, this is genuinely compelling. The price increase to $4,699 stings, though.
Dual RTX 4090 (~$4,250 USD): If you need production-grade 70B inference — multi-user, high concurrency, 100 tok/sec — this is your rig. But it's expensive, power-hungry, and overkill for single-user experimentation.
AMD RX 9070 XT (~$500 USD): Not for 70B (16 GB VRAM limits you to ~13B models), but for running Qwen 3 8B or coding assistants at 80+ tok/sec? This is the cheapest serious local LLM GPU on the market with official ROCm support.
Here's what gets me excited: we've reached the point where a $2,850 PC runs a 70-billion-parameter model — a model that rivals GPT-4 on many tasks — entirely offline, forever, with no subscription. Two years ago this required a $30,000 server. Next year it'll probably fit on a $1,500 single-GPU rig.
The pricing is broken right now (thanks, NVIDIA supply constraints), but the capability is real. The software stack — Ollama, llama.cpp, vLLM, MLX, ROCm — has matured to the point where multi-GPU "just works." And the open-weight models keep getting better, smaller, and faster.
If you've been sitting on the fence about building a local LLM box, July 2026 is the moment. The GPU market is awful, but the capability-per-dollar — even at inflated prices — has never been better.
What's your local LLM setup? Running dual 3090s? Rocking a DGX Spark? Still CPU-inferencing on a MacBook? Drop a comment — I genuinely want to hear what's working in the real world.
Sources: Presenc AI Local LLM Benchmarks, Compute Market Multi-GPU Guide, PromptQuorum Multi-GPU Deep-Dive, Quantize Lab GPU Guide, bestvaluegpu.com, NVIDIA DGX Spark Pricing, Local AI Master ROCm Guide. Pricing verified July 17, 2026.