Qwen 3.8 27B GSQ RCO tested - 16GB Local LLM setup transcript

Hey, welcome to Linus Tech Tips Lab. It's great to have you here. So, in today's video, we're going to be looking at a fine-tune on quant 3.8 27B, which is GSQ RCO. So, a few of you have been requesting this one, and that's because it's another one that can fit entirely in a 16-gig GPU. So, if we come down here and look at the quantizations available, you can see the largest size they have is 12.1 GB, and that's because it includes the MTP header.

So, in this video, this is the one we are going to go with, the largest, because this will still fit comfortably in a 16-gig GPU. So, what is this? This is a non-uniform GGUF quantizations produced with GSQ and RCO. So, if we come down here, there's some results, some charts on the differences between the quantizations they offer, and then down here, it will explain. So, GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type.

RCO assigns the per-tensor quantization types under a size budget. Together, they yield a non-uniform GGUF at the requested size. So, we're expecting decent performance at low quantization, essentially. I'll have this page linked down below, as always, if you want to read more about the way this model was quantized. So, a quick look at the tests we'll be running in this video. So, we've got a performance where we test the prefill and decode on our 16-gig GPU, memory benchmark where we test the model's ability to find data within a large context, reasoning, which has now replaced the agency benchmark, which will have the model work through increasingly difficult reasoning challenges.

We'll have the OpenAI human eval, so 164 Python challenges, then move on to Kanban, which is our front-end web development test, then Sam Physics and Dungeon Crawler, which are algorithmic and mathematical capability testing, and then finally at the end, we'll move into MCP with Blender and Godot. This will test tool calling capabilities, and this will test the model's ability of running difficult, long-running tasks in complex software environments.

So, a quick look at the system specs. I've made a bit of changes recently here. So, we've still got our RTX 2000 with 16 gigs of VRAM, and then our 32 gigs of DDR4 RAM to overflow into. But yes, there's now a 5090 sitting in here, too. However, I only use my 5090 to make test run faster, and all of my configuration and targeting remains for 16 gig GPUs. All right, then. So, let's start running through our benchmarks first.

So, we've got the IQ3S quantization, as I mentioned earlier, and we're using 130K context, and this is running on my 16 GB GPU. So, we're going to start with the prefill, the speed at which the model will ingest tokens. So, on a short prefill, I'm getting 159 non-cached and cached tokens per second. Coming up to a large prefill, I'm getting 200 tokens a second uncached and just below 23,000 tokens a second uncached. And then, if we come down to our decode, so that's our token generation speed.

So, I've got MTP configured on this, and I'm getting 4.6 tokens a second on that RTX 2000. Now, the RTX 2000 is a 70-W workstation GPU. If you've got a gaming-focused 16-gig GPU, you're likely going to see double this. So, you'll be closer to 10 tokens a second, I would estimate. Moving on to our memory benchmark now, so this is where we fill up the context of the model, insert pieces of data at 0, 25, 50, 75, and 100% of depth, and we ask the model to find that data, and we do it three times over.

So, 15 runs in total, and this model has passed every time, and it's found the data at every depth, is what this is telling us. So, memory recall ability on this fine-tune is very good. We can look at some of the outputs. So, here's an example of a very clean output. Here's an example of one that's a bit more messy, but the data was still found within the output. What we're seeing here is actually at the start, everything's clean, and at the end, everything is clean, too.

But, at 25% depth, they're always not so clean. So, while the model is performing well, it does look like at around 25% depth, it's not as clear as it is at the rest of the context depth. So, something to note. So, moving on to our brand new reasoning benchmark now. So, there are 48 questions, all progressing in difficulty. And what this is showing is our model has passed on the easy, medium, and hard questions, and only got 42 on our expert questions.

Now, this is a brand new benchmark, so we obviously don't have anything to compare this to yet, and I'm still working on this one, tweaking it around the edges. But, this will be our new benchmark going forward that will replace agency. And then, finally, onto OpenAI Human Evals. This is 164 handwritten Python challenges from the people over OpenAI. This test will also be updated soon because it's obviously going to be in training data, having been out for so long, as you can see, 5 years ago.

But, I'll have this link down below if you would like to try this out and read about it yourself. So, going into the results, the model's actually done quite well here. So, 87 pass rate overall, with a 98% answer rate. So, the pass rate and the pass rate of the questions it's answered is almost identical because it's almost answered everything. So, this fine-tune has done very well on this test. So, let's move on to our coding challenges now.

So, we're going to start with Kanban, which is our front-end web development test. You can see at the end here, we've got to 82.3% of our context used at 130K. So, jumping over now, this is what the model has delivered. I have fully tested this, it does work in its entirety, and it only took one extra prompt to get to this end state. So, the model initially came back at 81.4% context used, and whenever I opened a card or wanted to create a new card, whenever I pressed any key on the keyboard, it would close this window.

So, I prompted it to fix this, and then it only used another 1% of the context to get to this end state. So, for the Kanban, this fine-tune has done a very good job in building this with a reasonable amount of context used. This is quite a complicated task for front-end web development. So, moving on to our sand physics now, this is a test that will typically not take too much tokens to generate. So, maybe 50% of our context is quite standard, but with this fine-tune, we compacted twice, and then sit at 30% context used to get to an end state, which I would say is acceptable.

So, you can see we can place the sand. We can place the water and the acid and everything. But, the thing is the water is still not perfect. So, if I put the sand here, you can see the water is teleporting up to the top. So, it's still not perfect, this result. So, let me run you through how we got here. So, I gave the model the prompt, it worked away, and it got to 47% context used and delivered. This is normally about right for a model, but then when I checked, the water wasn't correct.

The model then came back at 83%, and I said the water is still not working. So, the water was behaving almost like sand. It would like stack on top, but it wouldn't flow down like sand either. It just wasn't behaving correct at all. We have seen water behave like this with models and fine-tunes in the past on this test. So, after I told it it's not working still, it then went ahead and kept going, compacted, kept going right through 100% of that, compacted again, and then came back to where you see it at now, where we've compacted twice and we're at 30%, and the water's okay, but it's still not perfect.

But at this point, I would call this test a failure because it's taken way more tokens than other fine-tunes or the base model would on this test. So, it's definitely a failure for this. There are better models for this task, basically. And then we move on to our dungeon crawler next. So, a good test of algorithmic capabilities of the model, and with this one, it was more pleasantly surprising. So, this one came back 64.4% of our context used, and that was it.

When it delivered, it delivered this, and it looks great. So, the ray casting appears to be working nicely. We've got the fog of war. The god mode is working. We can regenerate the dungeons. So, yeah, it's a one-shot, and it's done a really nice job on this dungeon one. So, the model definitely did not struggle as much on this test as it did on sand physics. So, moving into our MCP challenges now, so this is where the difficulty will ramp up, and we're into Blender, where we asked the model to create a starfall lantern and then deliver screenshots from different angles.

So, you can see here, we've ended up after one compaction and 42.4% context used, quite common for this task. So, there's nothing out of the ordinary here. So, the model actually came back at around 40%, and all that was missing was the screenshots. So, I prompted it for the screenshots, it used up a couple more percent, and then it delivered. So, this is our scene in Blender. So, you can see there's a mistake here with the ring on the top.

This is we've seen this with other fine-tunes and quantizations before, where it does this. So, this is the asset it's generated, not too bad, I would say. And then if we look at the screenshots, so we can see what it looks like when it's rendered. And here is our rendered lantern. So, here is the main shot. We've then got a close-up here, then a shot from the opposite side to make sure it's not taking any shortcuts hiding stuff, and then just another shot of it here.

So, I would say it's it looks quite good. You can see the geometry on the base is quite simple, no texture or anything. But, what I do really like is this light inside where it has a glow like this. This is not normally something we see, so I like that. And everything else looks good. So, the the glass is behaving as it should, the light is pleasing on the scene, I would say. So, yeah, I'll let you guys be the judge of how you think it's done here.

So, moving on to our final and most difficult challenge in here. So, this is the MCP connection to Godot, the game engine. And we asked the model to create a simple 3D platformer where the user must collect some orbs or keys, and then that unlocks a finish tile where they can reach to complete the game, and there also must be moving platforms as well. So, on this one, I gave the prompt over to the model, and you can see here session compacted twice, operation aborted.

So, this means I essentially interrupted the model while it was working. So, I did really just prompt the model here and then just leave it to do its thing, and then I came back quite a while later, and I decided to just test the game of what it had done, and the game was already a decent state. So, that's why I interrupted the model because it's good enough state, I think, to show you. So, just now we've got to this double compaction and then 75% of our context used but without any human intervention at all.

So, this is our project in Godot with our scene. So, here we are. This is our scene. You can see these are the keys that you're supposed to collect. It's got like a nice floating animation. So, we can grab that, and then we have key one of three. So, we have our moving platforms. Now, the only problem here is with this key, it cannot be obtained without dying as far as I'm aware. So, there we go. We've got that one. And then moving on to the end.

So these red platforms, they fall after you jump on them. So you have to move quick over those. We've got this key here. If you touch this, you will die. And then we're able to reach our goal. And then that's it. We have our end state goal reached. So it's done a very good job, actually. So that brings us to the end of this video on the GSQRC quantization. So I will leave some numbers on the screen now for our benchmarks and our testing, so you can have a quick look at the summary of how it did.

And then for my opinion on this, I would actually say it's quite good. So it definitely did have a hiccup with sand physics and it performed worse than other things. But on other tasks, I think it did quite well. So on Kanban and Dungeon Crawler, it did well. And then once we move into the more complicated things, such as Blender and Godot, typically in our previous testing, the base model 3-bit quantization did struggle with some of these more difficult tests, where this fine-tune, I would say actually did quite a good job, especially in Godot.

I think it done very well in Godot. Whereas if you go back to my quantization comparison video for the base model, you could see these lower quantizations really struggled in something like Godot. So there is definitely a positive effect this quantization method is having on the model here. So I would say if you're doing some complicated work and you are limited by your VRAM, this fine-tune is probably worth trying. So I can definitely recommend it for more complicated tasks.

But if the task is not so complicated, I would say you can still stick with even a Q2 on the base model. So what do you guys think about this fine-tune? Have you tried it yourself? I did test this based on a lot of feedback of people asking me. So, if you do have any other requests fine chains you'd like to see, just let me know. So, thank you for watching to the end of the video. I'd really appreciate it. Huge thanks to my members here on YouTube and on Patreon.

You guys are amazing. Give the video a thumbs up if you enjoyed it and subscribe if you want to see more content like this. Bye.