Bonsai 2 27B tested - 16GB Local LLM setup transcript

Hey, welcome to Luke's Dev Lab. It's great to have you here. So, in this video, we're going to be looking at Turnary Bonsai by Prism ML, and this is their version two. So, this is the model in question, Turner Bonsai 227B by Prism ML. So, I did actually look at the version one of the Bonsai models and wasn't too impressed with them back when I looked at them, but we'll see if version two is any better. So you can see this one is popping off on hugging face nearly 2 million downloads in the last month but really it's just in the last few days.

So very popular these ones. So this is based on the Quen 27B model. So in this video we are going to look at both the one bit and the two bit. So we'll test both. And as usual they do make some big claims here. 9.3 times smaller than FP16, but 98.2% of intelligence retained. So, when I test this and some people say like, "Oh, well, it's in a size class of, you know, Q2 or whatever, and you should compare it to Q2." But no, because they always claim that it's like as good as the FP16.

So, at the very least, it should be able to stand up to my previous testing of like Q4 of the 27B model. So, that's exactly what we're going to find out in this video. So, I will have this page linked down below if you want to come and have a read on more of the details on how this Turner stuff works. But for now, let's just get into it and test it. So, a quick look at the test suite for anyone that's new around here.

So, first we look at the performance of the model with the prefill and the decode speed. We then run a memory test where we fill up the context and see if it can find data throughout. We then run it through some reasoning challenges. Run it through human eval remix which is 100 Python challenges. Then canban for our front-end web development test followed by SAM physics and dungeon crawler for mathematical and algorithmic capabilities.

And then finally we ramp up the difficulty with Blender and GDAU for MCP connections where we have the model create an asset in Blender and a 3D game in God. A quick look at my system specs. So iuntu server is my operating system. I'm serving the models with Llama CPP and you can find the configs for this at the website linked in the video description. So my primary GPU for testing models, I have a RTX 2000 AD, which is a workstation GPU with 16 gigs of VRAM.

This is 70 W of power and about 225 to 250 GB a second of memory bandwidth. So it's about half the performance of a gaming GPU. and then 32 gigs of DDR4 RAM to overflow into, but we won't need to for these models because they easily fit. I do have a 50/90 in here, which I only use for speeding up some of my benchmarks and tests, but my target is always 16 gigs. And that does go for the configs that you'll find down below.

Everything is configured there for a 16 gig GPU. All right, so let's jump into our benchmarks. Now, I'm going to start with our performance. Now, for all of these, I'm going to run side by side with the Q1 on the left and then the Q2 on the right. So we can just have a quick look at both of them together. So with our performance test coming down to our prefill. So this is the speed at which the model will ingest tokens.

So on Q1 we're starting off around 246 tokens a second on a uncashed prefill about 420 on the Q2 ending at about the same on the Q1 and then about the same as well on the Q2. So performance is pretty steady for uncashed from a very short to a very long prefill. Come down to cached. So, we're starting off just under 2,000 on the Q1, 2.5,000 on the Q2, coming up to 116,000 on the Q1, nearly 150,000 on the Q2 on a long prefill uncashed.

And then we come down to our decode, which is our token generation speed. Now, this is all running on my 16 gig GPU, the RTX 2000 I just mentioned in the system section. So, we're getting about 24 tokens a second on the Q1 and we're getting about 25 tokens a second on the Q2. And that's because this is not just strictly Q1 versus Q2 in the traditional sense of quantization. They are both quantized in different ways which go to the model profile if you want to read more into that.

So onto the memory test where we fill up the context of the model to 256k and we ask it to find pieces of data at either 0 25 50 75 or 100% depth. Do that three times over each depth for a total of 15 runs. So they're both looking a bit shaky here. So we can see the Q1 60% pass rate and the Q2 67 and they're both showing a very similar pattern in the in the middle of the context. So 128k out of our 256, they have failed to find the data every time.

But then at 75% depth, they've both picked it right back up and found the data every time. And then right at the end of the context, the Q1 has fallen down three out of three times. And the Q2 has fallen down two out of three times. So it's only found it once there. So the results are not looking great for our memory recall ability of the models here. And we can see the outputs. They're not the cleanest outputs. There's a lot of padding, a lot of thinking padding around them.

So, yeah, memory test not the best on this. Moving on over to reasoning. So, we've got 40 out of 48 on Q1 and then 39 out of 48 on Q2. So, this is where we run the models through 48 reasoning questions that ramp up in difficulty as they go. So, we can see Q1 and the Q2 have both got 100% in the hard category, which is actually really good. So, it's not often you'll see 100% in the hard category, but then in the expert category, they've fallen down quite a lot, which is to be expected.

Even Frontier models don't get like great scores in expert. That's the whole point. But what my takeaway from this is actually it's quite decent on reasoning, which could be a use case for this model then if it's strong at reasoning at least. And then finally onto the 100 Python challenges. So, they've both got a 100% answer rate, which is really good to see. It typically means they don't overthink or get lost in thought.

And they've both actually got the same score here where they've got 73 correct out of the 100. So, not too bad. And then speed basically identical because we saw the token generation was almost the same. So, yeah, so far these models do seem very close in their benchmark results at least. Okay, so let's jump into our coding challenges now. We're going to start with Canban, our front-end web development test. So, as with before, I'm going to have the Q1 results on the left and then the Q2 results on the right.

So, for Q1, it is functional. It's not a great user experience as you can see, but it is quite functional. So, I can actually move the columns. I can move the cards. We can do the light and dark. The filters are working. So it is actually functional. This it has delivered a functional app from Q1 here. And then coming over to Q2, we have not got a functional app running over here. So let's jump into the code and see what it took to get to this.

So on Q1, we had to compact three times and reach 72.6% 6% of our context use to get to that beautiful result you just saw. And then with Q2, we compacted four times and it was still not able to get the project running after running a test suite multiple times for quite a while working away at this. So I decided to call it at that point because after four compactions the memory is going to degrade. We saw the memory performance is not great on the model as it is.

So, it just didn't do a great job. So, let's move on to sand physics next. So, we've got the Q1 here. And then Q2, we don't have it. And we'll see why in a bit when we get to the code. But how is Q1? Well, it's like that. So, there's our sand water, the walls. So, you can see there is kind of something working there. like the the materials properties do seem to behave as we would want them to but you know this is not ideal.

So if we jump into the code, I was unable to get the model to resolve this issue after multiple attempts in the code here. And we did reach compactions in this one, but I can't seem to get the information on how many compactions were reached in here. So here we've got some stats on this actually. So I messaged twice. The assistant made 102 tool calls, a total of 8 million tokens used. So yeah, it took a lot of tokens to get to the result you saw.

And then if we come over to the Q2 and I expand out well let me first explain. Every time it went to write the HTML code, it would always get truncated and reach its token output limit. So it would be like okay I'm going to write the file and then it would it would be bigger than the 130k context that's available which seems strange because it should not require this much context to write this HTML file. So then one of the times it was writing and I was at my computer.

So I decided to expand out what was going on in the code cuz I was curious. And then I discovered that it was actually just looping at the end every time. It was unable to write the HTML file. So this is why it was just wasting all the context. It was just doing this. So yeah, really great job there. So you can see it kind of writes everything. Maybe that's everything that's needed. I'm not sure. And then when it wants to close the body in the HTML, it just does this forever.

And then it decides it can do a bit of code again. Looks like it's starting the file again or something. I just saw a head tag in there in HTML. Oh no, it's another attempt at trying to write the HTML file. Okay, so yeah, Q2 could not deliver because it was unable to close an HTML file. So on to the dungeon crawler now and a good test of the model's mathematical and algorithmic capabilities. So Q1 on the left side here.

So we can see it's not done too bad a job, but the ray casting is a bit off. you can see there. But other than that, it's actually not too bad. So, we can do god mode and we can regenerate and it's okay. And then coming over to the right side here with Q2, it's basically the same story. So, you can see the ray casting is not perfect here, but other than that, again, it's not too bad. And we can do god mode and we can regenerate.

You can see it's gone for a very different style of dungeon here. So, two very distinct styles and they've both got to a very similar state where the generation is decent, god mode works, player movement, ray casting is just a bit off. So, jumping into the code. So, we reached two compactions and then 62.7% with Q1, which is a lot of tokens for this task. And then the Q2 just reached one compaction and then 58.4% which is a bit less.

So, not too bad, but it's still as this test goes, it's quite a lot of tokens used. But this is probably one of the stronger results for the models on this one. So, we're going to move into more difficult challenges now, the MCP portion of the video, where we start with Blender and ask the model to create a 3D lantern asset. So, we're starting off with the code first and then we'll go into Blender. So we've got the Q1 at three compactions and then half the context used.

And then the Q2 at three compactions and almost half the context used. So definitely more tokens than you'd see from something like the base 3.827B model to reach a good result. But let's see the results. So if I go to my folder for the Q1 model, there is nothing. The model did not save a project. It didn't do anything. So when I looked in Blender, it was still on the default cube and it hadn't even saved a project file.

So yeah, it just spent, you know, the best part of over a quarter million tokens not doing too much. Didn't really get a good result out of that. And then if we go to the Q2, we've got some tests shade hunt where it took some screenshots. It never saved it again, so I don't have the project file, but it took some screenshots of what it was working on. And this was the asset. So that's all the evidence we have of what it was working on.

So I think the clearest image might be this one. So but yeah, there was no project to look at and it certainly wasn't able to deliver even after three and a half times the context usage. So didn't do too well on these ones. It's a bit of a fail for these. So we'll move into the final and most difficult challenge now where we get our God game engine to have the model create a simple 3D game. So with the Q1 over here, we can see we're at five times compaction and 90% of our context used.

And that's because the model was unable to compact further on here. And then on our Q2, six compactions and 80% of our context used. So we're also starting to hit a limit of compaction and memory degradation on this one too. So let's have a look at their results. So this is Q1. If we have a look at our 3D scene, there is something there. It's been working. And then if we play this, so I can't click. I can't engage mouse and look around.

And if I press W or D or A or S, that's all that happens. I can jump and then any of the keys, they just yeet us off the map basically like super speed. So can't really progress too much with this. And like I say, the memory is degraded too much from all that compaction now to really proceed with the development of this from the model. So I guess probably what you would want to do at this point is start a completely fresh context, give the project to the model and ask it to continue.

You can see it's it's actually got the platforms moving. There are keys visible. It looks like a lot of the elements are in place, but it took over half a million tokens to get here. And then if we load up our Q2 project here, so it doesn't look like there's anything going on here. And then if we press play, we do actually have something. So I can look around. the left and right is inverted and it's once my mouse comes out of the window I stop being able to turn.

So this is sort of my limitation. I can move around on this one. And if I try and move forwards, uh I'm actually blocked here by some invisible wall. But I found if I move to the side, there is a way through and then we can proceed. But we can't make this jump. The platform's not moving. It doesn't look like there's an end state. There's no keys either. So, and again, this is more than half a million tokens burned to get to this result.

So, yeah, it's not great either on this one. All right, I think that will do it for Turner Bonsai by Prism ML. So, let's first have a quick overview at the results. So, if you head over to Luke's Dev Lab, I have made some improvements. So you can now type in here to find your models. And then I've also added a compare feature as well. So we can have some side byside action. So now we've got the Q1 on the left as always and the Q2 on the right.

So for performance, it's pretty similar, especially on decode. But then the prefill is definitely better on the Q2 here. Coming down to our memory test. So they've both not done great where we saw them completely fall down in the middle of the context and then at the end they've done pretty poorly as well. Coming down to reasoning, they actually did quite well. To get 12 out of 12 on hard is actually pretty good. So reasoning could be a strong thing for these models.

Coming down to the Python challenges, 100 Python challenges, they both scored 73 out of 73, which is not too bad. I'm still kind of getting a feel for this one cuz this is a new benchmark. And then also just note that you might see future videos where I'm back to OpenAI human eval because I've recorded these videos not in the order that they are being posted on YouTube because I'm going away for a few weeks soon. So I'm kind of trying to get a backlog of videos to post.

But anyway, moving on to our agent test. So our coding challenges. So Canban was a pass on the Q1 and it was a fail on the Q2 where it just didn't work. But even on the Q1 model, the result was pretty poor. before the UI was quite terrible. So, wasn't a great result, but functionally it did work. So, I can't put a fail on it. Then for Sam physics, they both failed at this. So, Q2 just did nothing. And then Q1 had a pretty terrible result as we saw.

Dungeon crawler was actually one of the things the models were able to do today. So, I say today, I spent all weekend running these models through these tests. So, Dungeon Crawler was okay on both, but the ray casting was a bit off, but other than that, it was quite decent with the result. And then once we ramp up the difficulty with MCP, yeah, Blender and Godo was obviously too much for this 98% intelligence FP16 model to handle.

So, they both did very poor in these challenges, but it wasn't too bad on the Q1. Like, we could see it had moving platforms and a lot of the elements seem to be in place, but it wasn't playable still. and we're reaching our compaction limit for both the models here with more than half a million tokens used. So in summary, I would say honestly it's another letdown on this model just as it was with the first version. So I just I wish they wouldn't make these claims that it's almost as intelligent as FP16.

I don't know what test they're running to make these claims, but it's clearly not true. And then I would say like should you use this model? No, obviously not. But the good thing is it's really small, so it can run on smaller systems or it can comfortably fit into a 16 gig GPU. But the thing is, if you can run the base model, which this is based on, even if you run the base model at like five tokens a second versus this at 25 tokens a second, you're going to get a better result on the base model.

And it's probably going to take about the same amount of time anyway. Because where this will burn half a million tokens to give you a mediocre result, the base model will burn like 100,000 or even 150,000 tokens. So, it's going to be like the same time at the wall, but it's going to give you a way better result. So, if you can fit the base model, just use it over this. Like, don't use this. That's kind of all I have to say on this model really.

It's I wouldn't really recommend it. So, what do you guys think of this model? Have you tried it? Has it been good for you in some aspects? I would be interested to hear if it has been good at something. I mean, reasoning seemed like it wasn't a complete fail on this model, so maybe it's good for something in that area. So, thank you for watching the video. I do really appreciate it. Huge thank you to my supporters here on YouTube and Patreon.

You guys are amazing. Give the video a thumbs up if you enjoyed it and subscribe if you want to see more content like this. Bye.