I run 177B model on SSD (It's Fast!) | Qwen 3.8 Flash Next Test transcript

This is my local AI setup. 24 gigs of VRAM from two budget GPUs paired with 32 gigs of system RAM, a total of around 50 gigs usable memory. But in this video, we're going to run an 85 gigs model on it, a model nearly double our total memory capacity. And to my surprise, it actually achieves a very acceptable speed of 22 tokens per second while still delivers great output results. Check it out in this video. Qwen 3.8 Flash Next was launched 2 weeks ago and it's already the new open weights flagship.

Its intelligence index is right up there with frontier models like Opus 4.8 and GPT Terror. However, with a total size of 177B parameters, even the smallest 1-bit quant takes up around 72.5 gigs, which might seem out of reach for anyone without expensive hardware. Fortunately, Qwen 3.8 Flash Next features a brand new architecture, an embedded N-gram lookup table. The 177B parameter total is actually a combination of 125B core model weights and a 51B N-gram table.

While the core model layers need the fastest memory speed possible, the N-gram component is simply a lookup table. And in theory, streaming it directly from an SSD shouldn't slow us down much, hopefully. For this test, I'll be using Atomic Chat's IQ4XS quant. On paper, the actual memory requirement is 45.8 gigs for the model weights versus 39.1 gigs for the N-gram table. Technically, we should be able to fit the core model into our VRAM and RAM while still leaving enough room for a decent context window.

I'll start the llama.cpp server with the following flags: a 4-bit KB cache set to a 100,000 context length, offloading 32 layers to system RAM, and enabling lazy mode along with load mode none to force llama.cpp to read the N-gram table from the SSD instead of putting it in swap file or virtual memory. And since we're pushing our memory resource to the limit, Linux sometimes auto kill the server for using too much RAM.

I have to prevent this by using tune command and set the Llama server OOM score to minus 1,000. After the server is up and running several test prompts, here are the results. To my surprise, despite offloading more than half of the model weights to RAM and the N-gram table to the SSD, it still runs at 22 tokens per second. Even with a long context of 76,000 tokens, the average speed stays at a very acceptable 14 tokens per second.

Unfortunately, if you run this on Windows, the generation speed drops by about 20% bringing it down to roughly 16 tokens per second and around 10 tokens per second at long context lengths. And since we're using the 4-bit IQ quant, the model retains most of its original accuracy while delivering fantastic outputs. Before we end this episode, I'm still surprised that I can run a model with frontier level intelligence at a 100,000 context 4-bit quant on my poor hardware. >> [music] >> The big takeaway here is that offloading the N-gram table to an SSD didn't slow us down nearly as much as expected.

So, don't be discouraged by total parameter numbers when you see this type of model architecture. Always look into the actual core model size first. I think the embedded N-gram could probably become the new standard architecture for open-source AI in the future. [music] Let me know your thoughts in the comments and don't forget to subscribe for more local AI updates. Thanks for watching and see you in the next episode.