By a GPU Hoarder
I cancelled my OpenAI Enterprise subscription yesterday.
It felt like breaking up with a toxic partner who charges you $200 a month to read your diary. I looked at my credit card statement, then I looked at the humming black box under my desk, and I realized: I don’t need them anymore.
For the last two years, we’ve been told a lie. We’ve been told that “Frontier Intelligence” is too big for us. We’ve been told that running a 70-billion parameter model like Llama 4 or DeepSeek-R1 requires a data center, an H100 cluster, and a PhD in cooling systems.
They lied.
You can run these models at home. You can run them on used hardware you find on eBay. And you can do it with privacy that no cloud provider can match.
We call this “Going Local.” It is the only way to stop paying the “Intelligence Tax.”
Here is the technical reality of how we fit giants into shoeboxes using the magic of Quantization.
1. The Math: Why You Think You Can’t (And Why You Can)
Let’s start with the napkin math that scares everyone away.
A standard AI model uses “FP16” precision (16-bit Floating Point numbers).
- 70 Billion Parameters x 2 Bytes (16-bit) = 140 GB of VRAM.
The best consumer GPU on the market, the NVIDIA RTX 5090, has 32GB of VRAM.
The previous king, the RTX 4090, has 24GB.
So, 140GB seems impossible. You would need six GPUs.
But here is the secret: AI models are incredibly redundant.
Most of the “intelligence” in those 16-bit numbers is noise. You don’t need 16 bits of precision to define a weight. You can get away with 4 bits.
This is Quantization.
It’s like compressing a RAW photo into a JPEG. Yes, you lose a tiny bit of fidelity (about 1-2% accuracy on benchmarks), but the file size crashes by 75%.
- 70 Billion Parameters x 0.5 Bytes (4-bit) = 35 GB of VRAM.
Suddenly, the math changes.
35GB fits into two used RTX 3090s (24GB + 24GB = 48GB).
35GB fits comfortably into a Mac Studio (64GB or more).
We aren’t running “dumber” models. We are running “lower resolution” versions of the smartest models on earth. And I guarantee you, in a blind test, you cannot tell the difference.
2. The Formats: GGUF vs. EXL2
When you go to Hugging Face to download a model, you will see a confusing alphabet soup. Here is the decoder ring for 2026.
GGUF (The Universal Soldier)
- What it is: The format used by Ollama, LM Studio, and llama.cpp.
- The Superpower: It supports “CPU Offloading.”
- Scenario: You have 24GB of VRAM, but the model needs 35GB.
- Result: GGUF puts 24GB on your GPU and the remaining 11GB on your slower System RAM. The model runs, but it slows down (from 40 tokens/second to maybe 5 tokens/second).
- Verdict: Great for Mac users and people with only one GPU. It always works.
EXL2 (The Speed Demon)
- What it is: The format used by ExLlamaV2 (often run via Text-Generation-WebUI or TabbyAPI).
- The Superpower: Pure speed. It is optimized specifically for NVIDIA cards.
- Scenario: You have 48GB of VRAM and the model needs 35GB.
- Result: It runs entirely on the GPU at blazing speeds (50+ tokens/second). It feels instant.
- The Catch: It crashes if you run out of VRAM by even 1MB. It is unforgiving.
- Verdict: The gold standard for multi-GPU rigs.
3. The Hardware: The “Dual 3090” Meta
So, what should you buy?
If you are rich, buy a Mac Studio M3 Ultra with 128GB of Unified Memory. It’s quiet, energy-efficient, and can run massive 120B models easily. But it is slow (prompt processing takes forever).
If you are a “GPU Poor” rebel like me, you build the Frankenstein Rig.
The current meta is Dual Used RTX 3090s.
- You can find them on eBay for $700 each.
- Total VRAM: 48GB.
- Total Cost: ~$1,500.
“But wait,” you ask. “Don’t I need NVLink to bridge them?”
No.
For inference (running the model), you don’t need the fast NVLink bridge. You can just plug both cards into your motherboard. The software (llama.cpp or ExLlama) automatically splits the model across the cards. It handles the “layer splitting” over the PCIe bus.
It is ugly. It draws 700 watts of power. It heats up my room.
But it runs Llama 4 70B at 40 tokens per second. That is faster than the GPT-4 API, and it costs me exactly $0.12 a day in electricity.
4. The Setup: How to actually do it
Stop being intimidated by Linux. Here is the “Dummy Guide” to getting this running in 10 minutes on Windows.
Step 1: Get the Software
Download LM Studio. It’s the easiest entry point. It handles the drivers, the interface, and the downloading.
Step 2: Get the Model
Search for Llama-3-70B-Instruct-v2-GGUF.
Look for the file named Q4_K_M.gguf.
(Q4 = 4-bit Quantization. K_M = The “Medium” balanced mixing method. This is the sweet spot between smarts and size).
Step 3: Offload
On the right-hand panel of LM Studio, look for “GPU Offload.”
Slide that bar all the way to “Max.”
If you have dual GPUs, check “Split across GPUs.”
Step 4: Chat
Turn off your wifi. Unplug the ethernet.
Ask it: “Analyze my bank statement for risky spending patterns.”
Paste in your un-redacted CSV.
Watch the tokens fly.
Feel the freedom of knowing that Sam Altman isn’t reading your financial data.
5. Conclusion: Own the Weights
The “API Tax” is a tax on laziness.
They want you to believe that AI is a service, like Netflix. They want you to stream intelligence.
But AI isn’t a movie. It’s a calculator. It’s a utility.
Once you download the weights, they are yours forever. They can’t depreciate. They can’t be censored. They can’t change the price.
We are entering a world where the most valuable asset you can own isn’t Bitcoin; it’s a 70B parameter model on a hard drive that works when the internet goes down.
Stop renting your brain. Buy the GPU. Join the resistance.