Which one of them made these overlapping bar labels?
Hey, a lot of people sucked making matplotlib plots before LLMs existed.
It’s a bit misleading.
Qwen 27B has way less “world knowledge” than GPT-5. Ask it random trivia without internet search access, and GPT would know waaay more.
This is generally true of small vs large models.
…But honestly, Qwen 27B is better at tool use or agentic stuff. It’s hyper optimized for just that and coding assistance, basically.
This is often true of old vs new. Most newer models have hyper focused on agents/coding, often to the detriment of other use cases.
Quantization for practically running Qwen 27B also has an impact. A off-the-shelf Q4_K_M is not the same as the unquantized weights in real-world use, or even an “optimized” quantization like a custom exl3.
I mean baking knowledge into a model isn’t really all that useful to begin with. Just download wikipedia locally and have it access it through tool use, it’s way more efficient and more accurate. And yeah, I find Q6 tends to be the sweet spot where it’s close enough to full 16 bit in performance, but doesn’t chew up too much memory.
I sort of get it because of you know you’re only using the model when you have Internet and want it to search then just let it.
What version do you use and how do you run Qwen3.6? I’ve played around a bit with the Q4 version in LM studio +Zed, but I was not happy with the results. It looses track very often and often enters infinite loops or just stops…
I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.
This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.
Ok, thanks! Thaf sounds quite advanced, but I’ll have a read afterwards :)
Facts! Thank you!
This seems like a great time to mention what I built and hosted months ago. https://masland.tech/ai-efficiency-index/
The AI Efficiency Index tracks the cost/intelligence mix. Low intelligence is useless. Capability scales exponentially. Expensive ≠ better value. MiMo-V2.5 is the leader.
If you have any questions feel free to ask. This uses the Artificial Analysis leaderboard. They have “cost per task” which I think is weird.
Those models are for general use. If you have business use case and data related to it you can finetune model for specific use that will outperform all of frontier models and run at fraction of cost.
that’s the other huge advantage of open models you can run locally
How do you get into this? Any articles you can share? OP says you can run on your desktop… How? Doesn’t this stuff require huge data centers?
Running model that is good at everything require huge amount of energy and huge data center. Those models are mixture of experts. Latest Kimi K3 have 896 experts. Imagine you have company with 896 employees. Each question involves 16 employees to figure out what to do in what area of your business. Like a brainstorm to solve problem. Now if you know exactly what you want and in which area you actually need only 1-5 people. Like an agile team instead of all those people that you have. So you can hire just couple Kimi K3 experts. 16 experts are 100B parameters so roughly 1 expert in frontier open source model is 6B parameters. 5 experts is 30B parameters. You can run 27B Qwen 3.6 quantized into int4 on your computer like other people are doing right now.
I posted link below to example where they fine tuned model ( take it like a employee training ) for specific task.
what kinda hardware do you need to “run this on your desktop”?
You need a GPU with around 16gb vram at a minimum to run qunatized version.
I run Qwen 3.6 27B quantized down to Q4_K_M or even a variant Q5_K_S on my 8gb VRAM entry level AMD GPU, with support of my CPU and 32gb system RAM. Yes, I also limit the Context Length heavily to something like 18k. It’s slow. But the point is, you don’t need necessarily 16gb VRAM.
But it’s better to use faster models for this type of hardware anyway. The MoE type of models (such as 26B A4B, or 35B A3B) are vastly, vastly faster for normal usage, but they are a bit worse in some cases. Also Googles QAT trained model versions also have less RAM requirements without losing much quality.
I’m just saying that, so others are not discouraged too much. It’s not the same experience as the full version off course, but you can use them with some tricks on weak hardware too.
I’m fairly optimistic that people will figure out how to optimize the models a lot further going forward. One obvious path is to try and separate the reasoning network from the trivia that gets baked into the model, and some work is being done in this area. If you could have a context free reasoning engine and then feed the facts it needs to know on the fly based on the context you’re running it in, then you could likely have a much smaller model that’s very capable.
according to performance on standard benchmark. somewhat covered by the controversy surrounding the term: benchmaxing.
if you see all benefit as a linear one dimensional height on a bar graph…
its almost like you assume that the previous model gave the same exact answer(same style) and the new mode gave the same exact answer PLUS additional useful information.
it might be convenient if measuring progress was so simple. but unfortunately/fortunately , it is not so simple . the most important benchmark are the comparison of outcomes on the problems that YOU have & prompts that YOU can(will) write. nothing else matters for YOU.- i admit benchmarks are well designed to objectively measure competence on challenging problems that require skill and really need only ONE correct answer.
Sure, a benchmark doesn’t capture all the subtleties and different use cases, but it does give a general idea of the capabilities of a model. Obviously, you have to run the model and see if it does what you need. But the chart isn’t really about the nuance, it’s showing how drastically the efficiency of the models has improved in just a year. The fact that we can even reasonably compare a model you can run on a desktop to one that needed a data center just a year ago is phenomenal.
i see what your saying. i didnt mean to discredit standard benchmarks entirely.
i guess its obvious that it measures capability regardless of imprecision.
2 major proposed changes:
**first, i dont really know. aside from saying “benchmark your own prompt+usecase”
a proposed plan:- approach one: pay attention and credit new or improved architecture designs and research.
- approach two: spend more attention on benchmarks. especially specific benchmarks ( that are not focused with industrial domain tasks.) **domain task pursuit, is useful!.. but it depends on if your interest align to popular domains.
- approach three: if willing to utilize remotely hosted models. rating should also take in consideration… tools and everything else: websearch performance, RAG performance, smooth interface, pref/balance between speed vs comprehensiveness, cost (if relevant), etc… .
Honestly, I think the most reasonable approach is just to see what other people’s experience is like and which models are well regarded, then try them out and see which one is the best fit for what you’re doing. You might not even need the top performing one necessarily, and speed or lower resource usage might be a bigger factor.
🙄
Is it still telling people to put glue on their pizza and convincing teenagers to kill themselves?
So which one tried to cheat its way into the test?
Llama is one for sure
Kimi is better. Waiting for it to appear on Ollama Cloud.
The difference is that you can run Qwen completely local though.
