I hooked up a 3090 24GB and a 3080 10GB for a total of 34GB VRAM, and this setup will run the 4-bit quantization with 8-bit KV cache and the full 262K context at a satisfyingly good speed, and it's smart enough to be useful.
I don't know if I'm doing this right, but this is what llama-bench just gave me:
>llama-bench -m 'C:\...\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q4_K_XL.gguf' -p 2048 -n 128
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 34815 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
Device 1: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes, VRAM: 10239 MiB
load_backend: loaded CUDA backend from F:\llama\llama-b10764-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from F:\llama\llama-b10764-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from F:\llama\llama-b10764-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium | 16.34 GiB | 27.32 B | CUDA | -1 | pp2048 | 1599.03 ± 3.30 |
| qwen35 27B Q4_K - Medium | 16.34 GiB | 27.32 B | CUDA | -1 | tg128 | 37.11 ± 0.76 |
build: 3d3d7c818 (10764)
The motherboard is pretty basic (Asus Prime X570-P) so the 3090 is connected to PCIe at 16x speed and the 3080 at 4x speed, which will slow things down a bit. The OS is Windows. I have Linux on the machine too but I haven't tried it under Linux yet.
Wouldn’t Q4 with FP8 KV cache and full context fit on the 24GB card alone? I’m fairly sure 64K fits, if not, and I’m also really curious about loss rates for some of the newer experimental INT4 KV caches.
What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.
In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):
-c 65535
--cache‑type‑k q8_0 --cache‑type‑v q8_0
Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.
Thanks. I did get it working on the 3090 alone with reduced context and KV cache quantized to 4 bits, but it was getting itself muddled up and going round in circles more. Once I got it up to the full 262K context and 8-bit KV cache, it seemed noticeably less confused and more useful, but it needed the extra 10GB for that.
9 Comments
floofloof@lemmy.ca · 4 pts · 1d
I hooked up a 3090 24GB and a 3080 10GB for a total of 34GB VRAM, and this setup will run the 4-bit quantization with 8-bit KV cache and the full 262K context at a satisfyingly good speed, and it's smart enough to be useful.
inari@piefed.zip · 4 pts · 1d
T/s?
floofloof@lemmy.ca · 2 pts · 1d
I don't know if I'm doing this right, but this is what llama-bench just gave me:
The motherboard is pretty basic (Asus Prime X570-P) so the 3090 is connected to PCIe at 16x speed and the 3080 at 4x speed, which will slow things down a bit. The OS is Windows. I have Linux on the machine too but I haven't tried it under Linux yet.
CyberSeeker@discuss.tchncs.de · 2 pts · 1d
Wouldn’t Q4 with FP8 KV cache and full context fit on the 24GB card alone? I’m fairly sure 64K fits, if not, and I’m also really curious about loss rates for some of the newer experimental INT4 KV caches.
floofloof@lemmy.ca · 2 pts · 1d
So I heard, but I couldn't get it to fit. I'm really fumbling my way around this stuff so it's possible I'm just not doing it right.
CyberSeeker@discuss.tchncs.de · 2 pts · 1d
What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.
floofloof@lemmy.ca · 2 pts · 1d
It's llama.cpp. If you know where I could find those recipes I'd appreciate a link.
CyberSeeker@discuss.tchncs.de · 2 pts · 1d
They’re posted on GitHub pretty regularly, here’s one for the 3090/A5000 specifically.
https://github.com/mikecovlee/qwen3.8-27b-24gb-recipe
In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):
-c 65535 --cache‑type‑k q8_0 --cache‑type‑v q8_0
Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.
floofloof@lemmy.ca · 1 pts · 1d
Thanks. I did get it working on the 3090 alone with reduced context and KV cache quantized to 4 bits, but it was getting itself muddled up and going round in circles more. Once I got it up to the full 262K context and 8-bit KV cache, it seemed noticeably less confused and more useful, but it needed the extra 10GB for that.