I hooked up a 3090 24GB and a 3080 10GB for a total of 34GB VRAM, and this setup will run the 4-bit quantization with 8-bit KV cache and the full 262K context at a satisfyingly good speed, and it’s smart enough to be useful.
Wouldn’t Q4 with FP8 KV cache and full context fit on the 24GB card alone? I’m fairly sure 64K fits, if not, and I’m also really curious about loss rates for some of the newer experimental INT4 KV caches.
So I heard, but I couldn’t get it to fit. I’m really fumbling my way around this stuff so it’s possible I’m just not doing it right.
What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.
It’s llama.cpp. If you know where I could find those recipes I’d appreciate a link.
They’re posted on GitHub pretty regularly, here’s one for the 3090/A5000 specifically.
https://github.com/mikecovlee/qwen3.8-27b-24gb-recipe
In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):
-c 65535 –cache‑type‑k q8_0 --cache‑type‑v q8_0
Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.
