In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):
-c 65535
--cache‑type‑k q8_0 --cache‑type‑v q8_0
Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.
What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.
Wouldn’t Q4 with FP8 KV cache and full context fit on the 24GB card alone? I’m fairly sure 64K fits, if not, and I’m also really curious about loss rates for some of the newer experimental INT4 KV caches.
Why would you be concerned about this? If you are running a server, it makes perfect sense to expose API endpoints to allow other tools an interface to those services.
The only issue of note is the binary blob, and NVIDIA’s refusal to digitally sign the driver. Thus if you have Secure Boot enabled, you need to take the added steps to resign your drivers every system update, which is a pain, or disable secure boot, which may lock you out of some future stream games.
Hence the Tehuantepec Interoceanic Corridor in Mexico, which is an overland rail project, but as with any mega project, is rife with its own problems with deforestation and displacement of indigenous peoples.
Curious if this would have any impact on zram? lz4 and lzo are more recent defaults, I think, but zstd offers better compression. If this improves reads by 70%, it may make zstd the go-to choice.
From this test, lzo-rle is a bit more than twice as fast for IO swap throughput (a best case multi page, LZO-RLE: 8,248 MiB/s vs Zstd: 3,005 MiB/s). But with a 70% increase in speed, that gap may be significantly lessened, to the point where there may be a noticeable improvement due to the significantly better compression ratio (2.68:1 vs 3.96:1).
So if we assume they manage to build a fully rapidly reusable rocket and launch infrastructure (the latter of which is incredibly questionable), the only marginal cost increase is just labor to conduct the launches and consumables.
And the physics says more like 17 launches, but at the end of the day, that’s just how much it costs to send 200 tons to the moon, AND bring the rocket back. They’re not jettisoning stages or leaving the rocket behind in the moon like Apollo.
The numbers are mind boggling, and short of nuclear rockets and space elevators, there is literally no way to do this short of many, many, many tiny launches, and it’s way less risky to have those tiny launches refill a tanker in LEO than to send 20 launches to the moon independently. Plus there would be no way to send large pre assembled components like nuclear reactors and construction/manufacturing equipment. Everything would have to be manufactured and assembled on the moon.
This is pretty interesting as a half measure towards unified memory. Obviously the PCIe bus is the bottleneck here, but using the 16x slot should be slightly faster than using your NVMe.
Even O365 regularly botches line, paragraph, and table formatting compared to the desktop versions. If you really need your formatting to be page-perfect, you should probably be using Scribus, Microsoft Designer, or Adobe InDesign anyways.
They’re posted on GitHub pretty regularly, here’s one for the 3090/A5000 specifically.
https://github.com/mikecovlee/qwen3.8-27b-24gb-recipe
In particular, I think you want at least the following flags to limit the context length and quantize the Key and Value caches (full context is 16GB alone at full precision KV):
-c 65535 --cache‑type‑k q8_0 --cache‑type‑v q8_0
Using q8_0 for Key and Value cuts that in half. So native full context of 262,144 tokens costs 8 GiB instead of 16, but to fit on one card, something like 65535 would only add 2 GB of VRAM at Q8 KV, respectively. I have also heard good things about using q4_0 for KV, which would cut the context budget by half again, with less than 2% loss.
What software are you using for your inference server (llama.cpp, ollama, vllm, sglang, etc)? There are quite a few published recipes that should get you where you want to go.
Wouldn’t Q4 with FP8 KV cache and full context fit on the 24GB card alone? I’m fairly sure 64K fits, if not, and I’m also really curious about loss rates for some of the newer experimental INT4 KV caches.
Why would you be concerned about this? If you are running a server, it makes perfect sense to expose API endpoints to allow other tools an interface to those services.
The only issue of note is the binary blob, and NVIDIA’s refusal to digitally sign the driver. Thus if you have Secure Boot enabled, you need to take the added steps to resign your drivers every system update, which is a pain, or disable secure boot, which may lock you out of some future stream games.
Hence the Tehuantepec Interoceanic Corridor in Mexico, which is an overland rail project, but as with any mega project, is rife with its own problems with deforestation and displacement of indigenous peoples.
Good job repeating the title
Curious if this would have any impact on zram? lz4 and lzo are more recent defaults, I think, but zstd offers better compression. If this improves reads by 70%, it may make zstd the go-to choice.
From this test, lzo-rle is a bit more than twice as fast for IO swap throughput (a best case multi page, LZO-RLE: 8,248 MiB/s vs Zstd: 3,005 MiB/s). But with a 70% increase in speed, that gap may be significantly lessened, to the point where there may be a noticeable improvement due to the significantly better compression ratio (2.68:1 vs 3.96:1).
A ship ship can ship as much ship as a ship ship can ship since a ship ship can ship ship.
Linux users switching distros after 10 minutes of using Linux:
They’re mostly short novellas.
What is a “huge amount”? There are tons of options at the $129-200 price point. What options are you looking for, size, layout, RGB, media keys?
So if we assume they manage to build a fully rapidly reusable rocket and launch infrastructure (the latter of which is incredibly questionable), the only marginal cost increase is just labor to conduct the launches and consumables.
And the physics says more like 17 launches, but at the end of the day, that’s just how much it costs to send 200 tons to the moon, AND bring the rocket back. They’re not jettisoning stages or leaving the rocket behind in the moon like Apollo.
The numbers are mind boggling, and short of nuclear rockets and space elevators, there is literally no way to do this short of many, many, many tiny launches, and it’s way less risky to have those tiny launches refill a tanker in LEO than to send 20 launches to the moon independently. Plus there would be no way to send large pre assembled components like nuclear reactors and construction/manufacturing equipment. Everything would have to be manufactured and assembled on the moon.
At a really high level, SATA III had a 6Gbps data rate (8 bits per Byte, so 750MB/s) so most SATA SSDs were giving you 550MB/s at the most.
Conversely, NVMe is usually a PCIe x4 slot, which tops out at 4GB/s. Most M.2 NVMe drives will give you 3200MB/s, which is easily six times faster.
Edit: for a fun point of reference, old HDDs were about 60MB/s, with high latency on top of that due to seek time.
This is pretty interesting as a half measure towards unified memory. Obviously the PCIe bus is the bottleneck here, but using the 16x slot should be slightly faster than using your NVMe.
Even O365 regularly botches line, paragraph, and table formatting compared to the desktop versions. If you really need your formatting to be page-perfect, you should probably be using Scribus, Microsoft Designer, or Adobe InDesign anyways.
This is patently false; what if you are provided for by someone else? What if you are not eligible to collect unemployment benefits?
I can almost understand 2019, which was End of Support in 2023. Blame Apple’s certificate requirement, I guess?
But 2021 is still literally under support until later this year, which is mind blowing.
The so-called architect put those hairs there, you moron.
“It's One Banana, Michael. What Could It Cost, $10?”