Rendered at 16:51:01 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
lnenad 3 hours ago [-]
Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.
redrix 3 hours ago [-]
You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each:
1. It overthinks — Just like the previous iteration. High confidence.
2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others.
3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning:
(I made myself irrationally angry writing this)
khimaros 3 minutes ago [-]
I killed my own ssh session twice with pkill -f, because the pattern matched the command line containing it.
peterleiser 1 hours ago [-]
> Three honest possibilities, and I’ll be straight with you about each
This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.
switchbak 52 seconds ago [-]
At this point, I'm starting to wonder if their honesty is even load-bearing at all?
hluska 1 hours ago [-]
Thankfully most people have better reading skills than that.
mcny 2 hours ago [-]
I reached point three and was nodding all along. I guess I am the NPC
lsb 3 hours ago [-]
This is glitch art for text, I love it
Bluestein 1 hours ago [-]
That's a caveat, and a real one.-
amoss 1 hours ago [-]
But the reason why it remains load bearing is key.
gorgmah 3 hours ago [-]
You made me irrationally laugh reading this
raducu 2 hours ago [-]
On one hand I love your joke, on the other, this is HN not reddit and I usually downvote such responses, not sure what is the HN etiquette for such humor?
samsari 2 hours ago [-]
70% of the posts on HN are already satire and performance art
pixelpoet 10 minutes ago [-]
And full of made up statistics.
dannyw 2 hours ago [-]
You might already know this, but a large part of test-time compute / 'overthinking' is just letting the model do more passes, and refine its activation residuals more.
Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.
lnenad 2 hours ago [-]
Yeah I understand, it's my assumption that the actually/wait/but have a point. It doesn't reduce the fact that it increases the time for tasks substantially.
xatnys 2 hours ago [-]
Did you observe the model overthinking on practical tasks? While 3.8 does think a lot on xhigh I've found that it really depends on the task. On one-shot prompts that are usually the first to be posted during new releases it will tend to spend a lot more time thinking than doing. In other words the more open ended a problem space becomes, the more Qwen will tend to second-guess itself.
Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.
overgard 40 minutes ago [-]
Yeah, I ran into an overthinking loop with it a couple days ago on a task that shouldn't have been that hard. (It's kind of interesting to watch the internal conversation happening with it). Overall I'm impressed with it, but setting the /effort to medium is what you usually want (it defaults to xhigh). I do wonder if I had made it write out a plan if I would have avoided that though.
lnenad 2 hours ago [-]
Especially on practical tasks. One shot prompts work better at Q6_K_XL for me. It loads a file, then analyses then second guesses itself then again then again then it tries to come up with a solution then second guess rinse and repeat. 122b is the perfect balance but it lacks quality for harder to solve stuff. I've ran DS Flash 0731 at Q4KXL, 3.8 Q6KXL, GLM 5.2 Q4KXL and they all over-reason. At least that's how it looks like to me when comparing with frontier models, even weaker ones.
grim_io 3 hours ago [-]
That's low reasoning for a model, but max for a HN comment.
javier2 13 minutes ago [-]
You are absolutely right to push back on this. Let me think for a moment.
Casteil 37 minutes ago [-]
Yep.. for 'general purpose' use I found qwen3.8:27b to be disappointing due to overthinking. It's brutal especially considering how slow it is compared to MoE variants. It often overthinks to the magnitude of ~10x the tokens vs a ~4x faster gemma4:26b-a3b.
As a result, qwen3.8 will churn over a prompt often for 5-10 minutes while gemma4 regularly finishes the same prompt in under 20 seconds, while giving a consistent and accurate response in my favorite test case. Qwen3.8, despite churning like that, often misses with an inaccurate answer.
Obviously, 'YMMV' depending on your use case... just sharing my two cents.
atmosx 1 hours ago [-]
What about adding rtk proxy?
cyanydeez 3 hours ago [-]
My stack is basically deer-flow with Qwen3.5-122B-A10B; this hopefully will be a speed and intelligence improvement. Running deer-flow overnight on any research topic or verify clear scoped programming issue is really neat.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
lnenad 3 hours ago [-]
Yeah 122B is the sweet spot for me as well. Even deepseek flash overthinks on stuff way too much. I think they fully rely on large reasoning turns to achieve better quality. The result of course means we wait a long time to get results even with high throughput as a lot of tokens are wasted.
esafak 2 hours ago [-]
It will be interesting to see the token efficiency analysis. This is my first question now with Chinese models; I take raw benchmark performance for granted.
rohansood15 4 hours ago [-]
Didn't expect it to beat 3.8 27B so cleanly.
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
user43928 3 hours ago [-]
For comparison with hosted models, GPT 5.6 Luna scores 67% on DeepSWE, compared to 59% here for Qwen.
Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.
rohansood15 3 hours ago [-]
This is a good counter argument. But you have to note that this is after OpenAI cut Luna costs by 80%. If you compare launch pricing, Qwen probably comes out ahead on a cost-performance basis.
jrflo 3 hours ago [-]
The luna cost cuts were real though, not a one time promotion or something, due to some optimization (probably distillation?) that openai did.
mattalex 1 hours ago [-]
You assume that openai's inference is profitable and that they aren't just trying to bolster revenue before their IPO.
The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability).
In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference had great margins, we would expect that as revenues increase, the amount of spend on training reduces as a fraction of total expenses.
Since the loss-making fixed costs shrink as a fraction compared to the profitable inference, we should expect profitability to rise with total revenue.
However, all leaks of openai's numbers seem to suggest the opposite: as revenues increase so do the losses.
hluska 56 minutes ago [-]
I don’t pay OpenAI’s bills - I pay what they charge me. Their cost accounting isn’t relevant to a user.
aaa_aaa 18 minutes ago [-]
Argument was that open ai cannot be profitable with this. But sure, use it while you can.
throwaw12 3 hours ago [-]
what if it was because of quantization and they haven't released the new benchmarks for it?
Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
dannyw 2 hours ago [-]
ARC Prize has retested Luna after the discount and validated identical performance.
(Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT).
These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing.
Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).
QwenGlazer9000 3 hours ago [-]
Was it?
Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus.
It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
Almondsetat 3 hours ago [-]
>If you compare launch pricing
Why?
claudeIsDown 2 hours ago [-]
Sounds like discrete propaganda
criley2 2 hours ago [-]
Those prices are just tokens? Since each model uses different amounts of tokens to do the same thing, it's a misleading price that often makes open-weights look more competitive than they are, since most open weights models use dramatically more tokens and time to complete tasks than many frontier models.
In Artifical Analysis's cost per task, Luna(max) costs $0.05 per task, and Qwen 3.8 27B costs $0.25 per task, a 5X increase. We'll see how 3.8-flash-next does.
overgard 34 minutes ago [-]
Curious, how are you running it and what quantization are you using? I've mostly been using MTPLX; 125B sort of looks like it'd be right at the limits of my 128GB MacBook once you factor in KV cache and context window.. wondering if it's worth it compared to the 27B model which gives me a lot of headroom or even a 72B model.
Squarex 3 hours ago [-]
I don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.
rohansood15 3 hours ago [-]
For world knowledge, you'd want it to find and reference the source material to be sure. At that point, it doesn't matter if the knowledge is embedded.
quev 3 hours ago [-]
Keep in mind a web search might not include scanned books baked in the weights ;)
hedora 3 hours ago [-]
I think the big models have adequate recall, so tool use is probably unnecessary, but the user said the correctness of my response is important. Let me look up the data instead of relying on my memory.
dist-epoch 3 hours ago [-]
World knowledge also means knowing the various algorithms and ways particular programming problems are solved.
You can't search what you don't even know exists.
serf 43 minutes ago [-]
>You can't search what you don't even know exists.
that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire.
a lot of smaller agentic models and a lot of harnesses live on that premise.
horsawlarway 2 hours ago [-]
In a lot of spaces, this is actually preferable.
Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc.
Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model.
Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained.
---
Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value).
I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please.
Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store.
If I'm doing auto-repair... "cars for dummies (version 3)". etc...
jameshart 16 minutes ago [-]
One of the consequences of encountering a lot of LLM generated text which includes things the model vaguely remembers from its training is that honestly I have grown less tolerant even of human comments and documents that are based on mostly ‘I seem to recall that…’ level sourcing.
In a discussion on economic history, say, someone will opine that Alexander Hamilton had some particular opinion about tariff policy… based on their having a vague memory of a blog post where someone quoted a passage in support of some point. But wait - you can search the federalist papers, the text’s right there to be read, before you commit to saying online ‘Hamilton thought tariffs were a great idea’ you could take your internal ‘I seem to recall reading something about hamilton’s opinion on tariffs’ thought and turn it into a little RAG query where you pull up a source and check before you put another factoid out onto the internet.
And so I feel absolutely the same way about LLMs. I don’t care how much factual information was in the training data, when the LLM wants to rely on something it vaguely recalls having been trained on, it owes it to me to dig up a source and vet it.
There are limits to this, of course. I don’t want it to be thinking ‘but wait, maybe my memory of Python syntax is faulty. Is = used for assignment? <web search>…’.
But in general some caution about repeating vaguely recalled easily checked facts is warranted.
donmcronald 1 hours ago [-]
> Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
This is what I've been trying to focus on with local AI for now. I've been trying to build all new documentation so it's more AI friendly. It's been pretty interesting. Qwen-35BA3B with a small prompt does a good job of surfacing what I'd consider institutional knowledge.
I've been trying to silo the docs I write from the model with a prompt that tells it not to use general knowledge unless asked to. From the anecdotal testing I did, Qwen-35BA3B is great for it. It does a really good job of following the prompt and calling tools, so I've been able to play around a lot to see what seems to work best.
Ultimately, I think one of the most effective uses of AI will be having a distinct knowledge store combined with an opinionated agent (and sub-agent) setup along with different models for each task.
Who owns the knowledge store is going to be the big caveat. Right now I think the big online models are trying for generic, persistent memory and I'd be very hesitant to let that happen. Think of having someone with a perfect memory following you around forever, but someone else has the ability to make them disappear. That's not a good situation.
redox99 1 hours ago [-]
At 125B + 51B I'd expect it to have some degree of world knowledge, clearly in the middle between small models like qwen 27B, and huge trillion parameter models.
LaurensBER 3 hours ago [-]
If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet.
Self-learning/improving would be even better but that's still a long way to go.
redox99 58 minutes ago [-]
Search results suck because the web sucks these days. The big models from OpenAI/Anthropic have every book in existence baked into them
jameshart 27 minutes ago [-]
I don’t think that’s the right way to think about LLM ‘knowledge’. They don’t have absolute recall of everything in the training set. They have been trained so that they have weights that can predict what those books might say - that is, if they read them they would find the contents unsurprising. That doesn’t mean it wouldn’t be helpful to pull relevant passages of text directly into context for a particular task.
hedora 2 hours ago [-]
My AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
aftbit 2 hours ago [-]
I believe you're underestimating the lag inherent in the economy. Even if we grant the idea that the political party controlling the US House/Senate has a significant impact on the economy, and that the current party is BAD and the next one would be GOOD, I would still expect that things will continue getting WORSE for a good 4 to 8 years before they get better again.
And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
NewJazz 2 hours ago [-]
You know the memory cartel isn't even close to being broken, right?
gruez 3 hours ago [-]
>Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
How much memory does this translate to and what quantization (if any) were applied?
rohansood15 3 hours ago [-]
128GB, 4-bit quantized.
dist-epoch 3 hours ago [-]
It's a much bigger model, with a next-gen architecture. It's expected to be much better.
RobertasTa 3 hours ago [-]
[flagged]
andy99 3 hours ago [-]
> Qwen3.8-Flash-Next features a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token.
Didn’t see this mentioned yet. I wonder what this means for the effective size. It’s evidently ~176B paramètres, but how does that get quantized. A 4-bit quant under 100GB seems unlikely, I’m suspecting this won’t run in 128GB unified memory
In principle I like the idea of trading more memory for compute though, even if there’s a memory shortage right now
NitpickLawyer 2 hours ago [-]
It is 125B A6B. vLLM is already out with support, ngrams can be offloaded to RAM so you only need ~96GB VRAM for nvfp4 w/ full context.
Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.
pbmonster 2 hours ago [-]
The N-gram parameters can be fetched from SSD, with maybe the hottest ones staying in memory.
khalic 3 hours ago [-]
Gonna have to wait a few days to see what the wizards of the HF community come up with…
> You will need at least 75 GB of RAM or unified memory to run the model. Its smallest 1-bit quantized version is larger than usual because of the model’s architecture so 1-bit isn't really 1-bit at all. However, this also means the quantization is less aggressive, allowing the model to retain more of its original accuracy than more heavily quantized models.
Lots of RAM required even for the 1-bit, which is already downloadable. Interested to see how well this one works compared to Ornith1.5-35B-A3B I've been running (and quite happy about).
Edit: but llama-cpp does not yet support it.
pram 3 hours ago [-]
It's in Unsloth Desktop already. Looks like it's 73GB, so 128GB Mac or Strix Halo etc will work. Exciting!
andy99 3 hours ago [-]
I only see a 1-bit quant posted on unsloth HF and it’s 72.5 GB. Is that what you mean? That’s much bigger than I expected. If you can’t run a 4 bit quant in on Strix Halo it becomes a lot less interesting.
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
agile-gift0262 1 hours ago [-]
In their page they say it will need at least 112GB[0], so including context, that would be a tight fit. I'm also hoping I can make a q4 fit on my 128GB strix halo
That probably includes the 51b ngrams too. It's possible that those could be streamed from NVMe on-demand. The Engram paper that developed this technique streamed from RAM to VRAM at only ~1% performance degradation, but these strix halo boxes and the spark have much slower memory, so it's possible moving down another rung on the memory hierarchy wouldn't affect their performance too much.
This will almost certainly require changes to llama.cpp or vllm to do it right.
tosh 4 hours ago [-]
this is a new architecture (foreshadowing qwen 4)
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
Can someone explain the intuition behind the en-gram idea? I know DeepSeek published a paper about it a few months ago and the Gemma models have a lightweight version of it; but it hasn’t clicked for me yet
a_humean 3 hours ago [-]
Waiting for llama.cpp support to land, but this might be a big deal for Strix Halo users.
6B active params helps around the memory bandwidth constraints, but a 128GB box can probably run the Q3/Q4 quants fairly easily with a decent context size. This might actually be better for strix users than 27B, which was already very good.
freakynit 4 hours ago [-]
Those benchmarks look seriously impressive.. considering how small of a MoE model this is.
garo-pro 3 hours ago [-]
Interestingly they also share the parameter count for Qwen 3.7 Plus (397 b a17b). I don't think these were known before but I might be wrong.
stefan_ 11 minutes ago [-]
I think these "Flash" models are sort of an evolutionary dead end. Sure, there are some routine tasks and applications where they can be used. But for the actual novel development work? It's much better to run a big model at high power for 30 mins than watch the Flash model struggle for 2 hours and produce massive churn.
Same reason your phone has a few big CPU cores for real work, it's much better to "race to idle" than have an "efficient" core struggle. Shitty experience, shitty power efficiency.
wolttam 5 minutes ago [-]
It depends how you use the models. These small models work great for developers who prefer to stay more in the loop, and only task the model with things that can really only be interpreted in one way.
Not to mention, they’re great for self-hosting and getting yourself to not be dependent on some API that can go down or be altered at any time.
Big models seem to mostly be good for pushing ahead the frontier - the smaller models tend to gain the frontier’s capabilities after only a handful of months anyway. Many are perfectly content remaining a few months behind the bleeding edge.
armcat 3 hours ago [-]
How is input token efficiency/verbosity on this model? Has anyone tried? GLM 5.2 was doing lot of turns and thinking piling up input tokens in the context (compared to Claude and GPT models). Then Qwen3.8-27B was 2x of that. Both delivered good output results but those cumulative input token costs were not cheap. Note this is on our specific business workloads. Genuinely interested in other people's experience (if you are able to try it out).
petu 3 hours ago [-]
Haven't tried, would be surprised if it's any different.
It's new arch demo for future Qwen 4 family, but (as I understand) training recipe/data is same as any other 3.8 model.
anon373839 3 hours ago [-]
Does anyone have an idea how this might perform on a DGX Spark at longer contexts? I've been trying to investigate their performance with these medium-sized MoE models, but I'm seeing a lot of incomplete and conflicting information. The 273 GB/s bandwidth looks awfully bad on paper...
rohansood15 1 hours ago [-]
Given the new architecture, speeds are harder to estimate. Max would be ~40 tok/s.
amclennon 3 hours ago [-]
It looks like this also undercuts the already absurdly inexpensive Deepseek Flash in pricing. Wild.
geooff_ 3 hours ago [-]
Where are you seeing that? At the bottom of this post from Qwen I see:
Qwen 3.8 flash: $0.16 / $0.47
Compared to
Deepseek 0723: $0.03 / $0.075
(units in USD/m tok)
twohaibei 3 hours ago [-]
0.03 / 0.075 ? Where can i get that prices? Especially during peak hours DS4flash became much more money hungry than last month.
ah I don't think that page was up when I checked, it 404ed!
a_humean 3 hours ago [-]
Probably going to take a 1-3 days for support to land in llama.cpp and vllm.
lucabytheway 2 hours ago [-]
very interesting. new architectures is the most interesting type of news. after what i experienced when gpt-oss came out i have been on the look out for architectural approaches that improves efficiency.
Aside from the pelican, I am sort of impressed by the fact that things are going the way in terms of really impressive small models.
Also I love how this uses N-gram embedding. I think that Longcat was the first one who used it (I submitted that submission on hackernews because I really just loved the idea of it that I understood), I am certainly more interested in local LLM models and its interesting how they are utilizing new architectures to do some really impressive optimizations!
> I think that Longcat was the first one who used it
Wasn't it introduced by Gemma?
loclol101 3 hours ago [-]
Definitely need to try this out locally.
andai 3 hours ago [-]
Father, I cannot scroll the website.
KolmogorovComp 3 hours ago [-]
Will this be cheaper than DS4flash ?
NooneAtAll3 3 hours ago [-]
what's the deal with absent scrollbar on the website?
christkv 4 hours ago [-]
Looks like a good model for strix halo
Iolaum 3 hours ago [-]
indeed, can't wait for it to be supported by llama.cpp (or other engines)?
khangtong988 2 hours ago [-]
Yoh yoh I like It
axegon_ 3 hours ago [-]
Aaaaaaaaaaaaaand dario meltdown on twitter in 3, 2, 1...
myshapeprotocol 3 hours ago [-]
[dead]
skarz 4 hours ago [-]
do we really need breaking news about qwen posted every single day?
KronisLV 3 hours ago [-]
If there’s news, then yes. This is a pretty great new release for those still stuck on Qwen3.6 35B A3B if they have enough memory but don’t have super powerful compute.
I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.
NitpickLawyer 3 hours ago [-]
This particular release is interesting because it's a preview of qwen4 architecture. And, while benchmarks are iffy, this is a direct comparison, by the same team, with qwen3.8-27b that was pretty well received for a local model.
This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.
pseudony 4 hours ago [-]
I and presumably quite a few others with AMD AI or Apple Mac platforms are very impacted by this.
:)
It is very relevant and for a certain group of us, far more impactful to our work the next month(s) than any blog post could be.
tosh 4 hours ago [-]
this is a new architecture (foreshadowing qwen 4)
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning: (I made myself irrationally angry writing this)
This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.
For example, even if you make thinking tokens literally just '....' (absolutely meaningless; zero information), you still see significant performance improvements: https://arxiv.org/abs/2404.15758 and https://arxiv.org/abs/2607.22925 for some starters.
Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.
Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.
As a result, qwen3.8 will churn over a prompt often for 5-10 minutes while gemma4 regularly finishes the same prompt in under 20 seconds, while giving a consistent and accurate response in my favorite test case. Qwen3.8, despite churning like that, often misses with an inaccurate answer.
Obviously, 'YMMV' depending on your use case... just sharing my two cents.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.
The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability).
In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference had great margins, we would expect that as revenues increase, the amount of spend on training reduces as a fraction of total expenses. Since the loss-making fixed costs shrink as a fraction compared to the profitable inference, we should expect profitability to rise with total revenue.
However, all leaks of openai's numbers seem to suggest the opposite: as revenues increase so do the losses.
Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
(Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT).
These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing.
Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).
Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus.
It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
Why?
In Artifical Analysis's cost per task, Luna(max) costs $0.05 per task, and Qwen 3.8 27B costs $0.25 per task, a 5X increase. We'll see how 3.8-flash-next does.
You can't search what you don't even know exists.
that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire.
a lot of smaller agentic models and a lot of harnesses live on that premise.
Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc.
Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model.
Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained.
---
Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value).
I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please.
Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store.
If I'm doing auto-repair... "cars for dummies (version 3)". etc...
In a discussion on economic history, say, someone will opine that Alexander Hamilton had some particular opinion about tariff policy… based on their having a vague memory of a blog post where someone quoted a passage in support of some point. But wait - you can search the federalist papers, the text’s right there to be read, before you commit to saying online ‘Hamilton thought tariffs were a great idea’ you could take your internal ‘I seem to recall reading something about hamilton’s opinion on tariffs’ thought and turn it into a little RAG query where you pull up a source and check before you put another factoid out onto the internet.
And so I feel absolutely the same way about LLMs. I don’t care how much factual information was in the training data, when the LLM wants to rely on something it vaguely recalls having been trained on, it owes it to me to dig up a source and vet it.
There are limits to this, of course. I don’t want it to be thinking ‘but wait, maybe my memory of Python syntax is faulty. Is = used for assignment? <web search>…’.
But in general some caution about repeating vaguely recalled easily checked facts is warranted.
This is what I've been trying to focus on with local AI for now. I've been trying to build all new documentation so it's more AI friendly. It's been pretty interesting. Qwen-35BA3B with a small prompt does a good job of surfacing what I'd consider institutional knowledge.
I've been trying to silo the docs I write from the model with a prompt that tells it not to use general knowledge unless asked to. From the anecdotal testing I did, Qwen-35BA3B is great for it. It does a really good job of following the prompt and calling tools, so I've been able to play around a lot to see what seems to work best.
Ultimately, I think one of the most effective uses of AI will be having a distinct knowledge store combined with an opinionated agent (and sub-agent) setup along with different models for each task.
Who owns the knowledge store is going to be the big caveat. Right now I think the big online models are trying for generic, persistent memory and I'd be very hesitant to let that happen. Think of having someone with a perfect memory following you around forever, but someone else has the ability to make them disappear. That's not a good situation.
Self-learning/improving would be even better but that's still a long way to go.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
How much memory does this translate to and what quantization (if any) were applied?
Didn’t see this mentioned yet. I wonder what this means for the effective size. It’s evidently ~176B paramètres, but how does that get quantized. A 4-bit quant under 100GB seems unlikely, I’m suspecting this won’t run in 128GB unified memory
In principle I like the idea of trading more memory for compute though, even if there’s a memory shortage right now
Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF https://unsloth.ai/docs/models/qwen3.8-next
> You will need at least 75 GB of RAM or unified memory to run the model. Its smallest 1-bit quantized version is larger than usual because of the model’s architecture so 1-bit isn't really 1-bit at all. However, this also means the quantization is less aggressive, allowing the model to retain more of its original accuracy than more heavily quantized models.
Lots of RAM required even for the 1-bit, which is already downloadable. Interested to see how well this one works compared to Ornith1.5-35B-A3B I've been running (and quite happy about).
Edit: but llama-cpp does not yet support it.
[0]: https://unsloth.ai/docs/models/qwen3.8-next#qwen3.8-flash-ne...
Original error: llama.cpp does not support this GGUF's model architecture ('qwen4exp')
Edit : Saw the pull request, should arrive soon enough https://github.com/ggml-org/llama.cpp/pull/27742
This will almost certainly require changes to llama.cpp or vllm to do it right.
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
https://x.com/Alibaba_Qwen/status/2092591393424515114
6B active params helps around the memory bandwidth constraints, but a 128GB box can probably run the Q3/Q4 quants fairly easily with a decent context size. This might actually be better for strix users than 27B, which was already very good.
Same reason your phone has a few big CPU cores for real work, it's much better to "race to idle" than have an "efficient" core struggle. Shitty experience, shitty power efficiency.
Not to mention, they’re great for self-hosting and getting yourself to not be dependent on some API that can go down or be altered at any time.
Big models seem to mostly be good for pushing ahead the frontier - the smaller models tend to gain the frontier’s capabilities after only a handful of months anyway. Many are perfectly content remaining a few months behind the bleeding edge.
It's new arch demo for future Qwen 4 family, but (as I understand) training recipe/data is same as any other 3.8 model.
Qwen 3.8 flash: $0.16 / $0.47
Compared to
Deepseek 0723: $0.03 / $0.075
(units in USD/m tok)
https://api-docs.deepseek.com/quick_start/pricing
8t/s though apparently and their cache hit rate is terrible so I don't think it's worth it over Relace.
DSV4 Flash 304B params, 167 GB download (at full size)
Qwen3.8 Flash Next 180B params, 360 GB download (at full size)
Aside from the pelican, I am sort of impressed by the fact that things are going the way in terms of really impressive small models.
Also I love how this uses N-gram embedding. I think that Longcat was the first one who used it (I submitted that submission on hackernews because I really just loved the idea of it that I understood), I am certainly more interested in local LLM models and its interesting how they are utilizing new architectures to do some really impressive optimizations!
(Do note that I created it using a free rate limited end-point that I found on the huggingface space section: https://victor-chat-with-qwen3-8-flash-next.hf.space)
Wasn't it introduced by Gemma?
I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.
This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.
:)
It is very relevant and for a certain group of us, far more impactful to our work the next month(s) than any blog post could be.
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
https://x.com/Alibaba_Qwen/status/2092591393424515114
Qwen's advances do (currently) have merit.