I ran Qwen3.8 27B on single 8GB Card! (Better than expected) transcript

These are the results from Qwen 3.8-27B running on a single RTX 4060 8 gigs. All of them were done in one shot, no agent looping. It's no secret that it was slow and there were some difficulties to get this setup run properly without getting timeout or failed to calling. But still, to imagine that these are all done on a single 8 gigs GPU is very amazing. In this video, I'm going to show you how the test was set up, how to config the harness and parameters, and then compare the final result with Claude Opus.

Check it out in this video. Qwen 3.8 was released about 2 weeks ago and it's been all the rage among the community. A mere 27 billions model, yet scored better than Opus 4.6 in several coding and agentic benchmark. Now, lots of people recommend running Qwen 3.8 with 24 gigs card. However, if you look at the quantized version performance chart, especially the IQ4XS, is only about 6% from the base model, yet cut the size down to about 14 gigs.

This means it almost fit a 16 gigs mid-range card. But what about 8? Is it going to be too slow? Let's test it. So, I download the Unsloth IQ4XS. For 8 gigs, we need to be frugal. So, 70K tokens should be plenty for a small project. We also need to quantize the KV cache to 4-bit to save some context VRAM. Then, we'll offload about 26 layers of model to GPU. This should make the total VRAM usage to be around 7 gigs, leaving some room for OS usage as well.

Next, make sure to set the CPU thread pool size to match your CPU core count for better performance. For me, my CPU has six physical cores, so it's six. Also, make sure that the speculative decoding is set to MTP with two max draft token. After the model is loaded, turn on the enable thinking in inference setting and make sure to remove the reasoning budget. This is very important as Quant 3.8 thinks a lot. For example, the model used about 20,000 tokens on one of my moderate size prompt to just generate a website.

So, to prevent the model from aborting during the thought process, remove it. Next, I'll use the recommended sampling setting from Onslaught. So, temperature at 1, 20 top K, one repeat penalty, and zero min P. Next is the harness, which is also important. An 8 gig set up need a really lightweight harness with least embedded system prompt as much as possible. So, I'm going to use Pi. So, let's start the first test and put in the prompt to generate the Minecraft 3D landing page.

After an hour passed, I found that the model was stuck in a loop when trying to write the output. Somehow, Pi terminated the write tool call after writing for a while. With some research, I found that I need to disable the HTTP timeout in the setting since our token generation speed is too slow. I also need to increase the output max token size as well, or the process will get terminated again when writing large file.

I reran the prompt again, and after 2 hours and a half, here is the result. The average token speed is around 4.2 token per second. It was around six at first, but gradually slower with longer context. Here is also the result from same prompt by Opus 4.6. Next is Cyberpunk style neon city 3D scene using 3.js. This scene was done in about 1 and 1/2 hour with average token speed around five token per second. Next is a highly detailed prompt to create a personal website, around 2 hours with 4.6 token per second.

And the final test, a landing page for interior design company. Around 1 and 1/2 hour at five token per second. So, quant 3.8 is really a game-changer for local AI model. While the token speed is still too slow for a practical use on eight gigs, the result is solid at a model this size. I think it even did better on some test than Opus 4.6. And if you have a 16 gigs GPU, you'll get like 12 to 14 tokens per sec with around 70,000 context, which is in practically usable range.

Also, you might wonder why I did not use three-bit version with eight gigs setup. Actually, during the test, the three-bit is slower than four-bits. When you heavily offload it to CPU, any quant indivisible by two will suffer a speed penalty. So, what's your thought about quant 3.8 27B? Let me know in the comment. Stay tuned for the next episode for interesting local model test. Thanks for watching and see you in the next one.