Rendered at 20:17:04 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
SwellJoe 1 days ago [-]
Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio.
I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.
And, MoE should make it run at a close to usable speed.
2 hours ago [-]
sosodev 1 days ago [-]
That’s only true if you think AI is the only reason to own a powerful and efficient server. Mine does plenty of traditional server stuff too.
SwellJoe 1 days ago [-]
I can do traditional server stuff on any old computer with a big hard disk and a decent amount of RAM. That's not worth $3500-$4000. When RAMpocalypse is over and we can buy a Strix Halo for under $2000 again, the math starts mathing. It becomes a pretty great desktop computer that also happens to run AI pretty well at a pretty good price.
sosodev 1 days ago [-]
Yeah, but that computer can’t also do the AI stuff. And not everybody has a desktop with multiple 32GB GPUs available.
I’ll admit though I’m biased because I bought my board for $1600 back before the prices went crazy.
pizza234 12 hours ago [-]
I spent around 5k on a server for "AI stuff" and it's currently doing no AI, because local LLMs (at least on systems with 32 GB VRAM) can only do only very basic stuff; this includes Qwen3.8 - in spite of the reverse engineering blog post, when I've tried Qwen to do a similar task, it flunked miserably. Additionally, I've read on some informal sources, the next step in quality is at 256 GB, not 128, which is very expensive (it's around 10k). 10k for privacy is... a toy for rich tinkerers, considering that most the people have their email on the cloud.
embedding-shape 11 hours ago [-]
Same, I've had 96GB VRAM available for more than a year, and testing all the local LLMs as they become available, and read about people using models with Q2 quantization saying they get real work done, meanwhile I can't run 4x the size models with BF16 and get them to not eventually fuck up something.
Currently, I think most models are somewhat benchmaxxed, as my own benchmarks and others I know who also keep private benchmarks, don't show nearly as much needle movement as the press releases tend to want you to believe.
I can't wait for local LLMs to mature enough so I can move everything locally, but we're sadly not there yet today.
jcsycombinator 11 hours ago [-]
If you can't get Qwen 3.8 27b to work well that's a skill issue.
You just suck at using ai
embedding-shape 11 hours ago [-]
Which exact quantization you use of that particular model?
I've just spent two days testing it qualitatively (after the quantitative results were kind of bad, but promising), the official BF16 weights, and it keeps fucking up very basic things, like doing "rm -rf sessions/$id" without really checking or making sure $id is defined, so it ends up deleting more.
I'd love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either, literally makes no difference. But I too got hopefully reading the sentiment on the internet about Qwen 3.8, but it's just not there (yet) for local professional software development.
Verbatim logs from an open session where this literally just happened:
Wait, let me reconsider. Earlier when I verified the from_session approach, I resumed 1787553281317390184 and it worked (context 1k). Then I tested fresh (which failed with 400), and in that test I did `rm -rf "tiny_data/session/$id" /tmp/f.txt` where $id was the fresh session's id (empty, since it failed). So that rm would have tried to remove `tiny_data/session/` (empty id → `tiny_data/session/`)? Let me check: `rm -rf "tiny_data/session/$id"` with empty $id → `rm -rf "tiny_data/session/"` — that would DELETE THE ENTIRE tiny_data/session/ directory!!!
Oh no. That's what happened. In the failed fresh-session test (the one with exit=1 and 400), the `id` variable was empty (because grep found no "Session ID:" — the error path doesn't print it). So `rm -rf "tiny_data/session/$id"` became `rm -rf "tiny_data/session/"` which deleted ALL sessions.
Oh no indeed :)
SwellJoe 10 hours ago [-]
To be fair to Qwen, we've all been there.
embedding-shape 10 hours ago [-]
Maybe I got lucky (as I know others have had this issue), or it recovered by itself, but never had this happen with GPT models for as long as I've used codex, and I run it with "--sandbox danger-full-access --dangerously-bypass-approvals-and-sandbox" 100% of the times.
SwellJoe 1 hours ago [-]
No, I mean, every human who spends enough time working with computers eventually makes a dumb blunder, very often this specific one (using `rm -rf` in a way that has further reaching effects than intended). But, I do want to use models that don't make those kinds of mistakes, which is why I happily pay for a bunch of subscriptions for the best models.
downrightmike 1 days ago [-]
Oh no a tough constraint that will lead to further innovation like deepseek. How terrible.
surcap526 1 days ago [-]
[dead]
ArvidSu 1 days ago [-]
An "AI" server can do traditional server stuff but a traditional server can't do AI stuff (inference)
1 days ago [-]
jubilanti 1 days ago [-]
A 3060ti 8gb, released in 2020, has 448 GB/s of bandwidth compared to the Halo 256 GB/s
The 3080ti is 912.4 GB/s
yjftsjthsd-h 1 days ago [-]
But it also has 8GB of RAM.
embedding-shape 1 days ago [-]
And the newly announced/launched Apple M6 has 170GB/s of unified memory bandwidth, meanwhile M5 Ultra gets 1.2TB/s of unified memory bandwidth. https://www.apple.com/newsroom/2026/08/apple-introduces-m6-a... Not sure if the first one is a typo on their press release, can't be just 170GB/s then be pushed for AI use, can it? Could be a different measurement I suppose...
jtbayly 1 days ago [-]
You got me curious so I looked up the previous chips[0].
The different models of chips and memory config have very different memory speeds as well.
Eg the M4 Max 128GB has a bandwidth speed of 500GB/s+. And that's true for other models as well.
But as you note, the base speed has also increased over the versions.
brandall10 1 days ago [-]
From a bandwidth perspective, the ultra is like 8 M5s fused together (@ 150GB/s), that's how it gets to the 1200.
Historically the Pro doubles the base, the Max doubles the Pro, and the Ultra doubles the Max.
If an M6 ultra were released today it would be 1.36TB/s.
DwarvenEngineer 1 days ago [-]
does that mean they're measuring bandwidth differently than how others (like nvidia) does it? memory bandwidth is the gating factor of running models locally, so if it's actually 8x 150GB/s, it may help something like prefill, but would it actually speed up decode comparatively?
entrope 20 hours ago [-]
No, they use the same definition of memory bandwidth as others, but Apple Silicon has a lot of memory channels. In previous generations, prefill has been compute-limited and decode is fast.
https://blog.exolabs.net/nvidia-dgx-spark/ outlines a combination of a DGX Spark and an M3 Ultra that took advantage of fast prefill on the Nvidia hardware and fast decode on Apple Silicon.
pizza234 12 hours ago [-]
Not a hardware engineer, but it's mainly because of RAM wires/channels (not implying that this is "simple" form an engineering perspective).
Using the published bandwidths, the math is 170 * 1 and 153 * 8.
embedding-shape 11 hours ago [-]
But 170GB/s is almost nothing? None of the RTX 50 series GPUs has that low bandwidth, you have to go back two generations of nvidia GPUs to get closer to that, and then it's the cheapest of the series, RTX 3050, which has ~170GB/s.
Even the GTX 1080, launched ten years ago, has double the bandwidth!
This must be some different way of measuring the bandwidth right? Since they explicitly say this for AI, but the numbers they share don't show that at all. Or I gravely misunderstand something here.
ActorNightly 15 hours ago [-]
GFX vram is still faster.
SwellJoe 1 days ago [-]
Enjoy Gemma 4 E2B at blistering speeds, I guess?
jubilanti 17 hours ago [-]
My point was more: this was the 2nd lowest end card from a generation 6 years ago, and it had way higher bandwidth than today's alleged flagship.
jcsycombinator 11 hours ago [-]
Small amounts of fast ram vs huge amounts of slower ram.
It costs more than the strix to just buy regular ddr5 ram sticks today.
ActorNightly 15 hours ago [-]
Arguing with people on here that you should buy GFX cards instead of overpriced Macs for inference is a lost cause. Either Apple astroturfs this forum hard enough to convince people Macs are good for local llm, or people are REALLY stupid and don't understand how local inference works and think that the dogshit slow 40-50 tok/sec is standard.
AbsurdCensor 1 days ago [-]
I have had an impossible time getting 120B or better models running on Strix Halo (especially under Windows) with any large context windows. And 30-40 tokens/second is fine, but not the fastest.
For the most part lately I have been sticking with Qwen 3.8 27b and that thing will easily suck up 64gb of ram. Add in docker with some additional programs running and it's really easy to eat up 128gb of ram.
SwellJoe 1 days ago [-]
I found a couple of different 4-bit quantizations of Laguna S 2.1 that run pretty well with pretty big context (also quantized, to 8 bits, I think). Unfortunately, Laguna isn't better than Qwen 3.8 27B, which I'm able to run at roughly the same speed on my desktop machine, so I don't use Laguna or the Strix Halo very much, lately. (It's also too hot for me to be running heaters for inference. It's been ~110F most days for the past few weeks.)
throwaw12 1 days ago [-]
how much performance (tok/s) can you expect from 128GB Strix Halo? assuming this model will be released with FP8
also can you use it for fine tuning?
SwellJoe 1 days ago [-]
The Strix Halo and DGX Spark are pretty danged slow, relatively speaking. I don't recall exact numbers, but with MoE models in this size ballpark (Laguna S 2.1), I seem to recall I was seeing about 20-25 t/s with a big context, which is close to usable. Qwen 3.8 27B crawls on this hardware, though, at 10-16 t/s, definitely not comfortable for interactive use. (Though this makes it seem like you can cook pretty good with a 4-bit ROCmFP4 quantization: https://github.com/julianmb/q38rocm the model does get notably dumber below six bits.)
A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land.
DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can).
Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use.
Note also prefill with large models is pretty slow on the Strix Halo (300 t/s, maybe). Time to first token is a painful wait, when using it interactively with large models.
decide1000 23 hours ago [-]
On the DGX I get 44.5 tokens per second (NVFP4). With 8 concurrent it's 241 t/s total.
I am using the PrismaAQUA
standard 9.7 t/s
+ Dflash2 30 t/s
+ torch-compile 37 t/s
c8 = 177 t/s
SwellJoe 23 hours ago [-]
What model? Also, I don't know what "the PrismaAQUA" means, ddg thinks it's a CPAP machine, which seems unlikely to help with inference performance.
Also, 4-bit has measurable intelligence loss. Sometimes worth it, but, at this size models are barely smart enough at 8 or 6.
decide1000 21 hours ago [-]
Qwen3.8-27B-PrismaAQUA-5.5bit-vllm
The output quality is higher. It's held at full precision (not quantized).
downrightmike 1 days ago [-]
You can only use up to 90gb for the GPU, so it doesn't fit
SwellJoe 1 days ago [-]
No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp.
I enjoy the Qwen models a lot, but building things on top of them with OpenRouter has been painful.
OpenRouter does a lot of great work and I really enjoy being able to use different models so easily. I like when a provider is phasing out an older model that still works for my needs and the price is much lower. It seems like such a good win-win.
However, the problem is that many Qwen models have almost no capacity or is so flaky you literally have to just litter your code with a blacklist/whitelist of providers. OpenRouter has some attempts to solve this, but they don't work. In fact, OpenRouter has a lot of really cool stuff that is documented, but if you read the code it's not yet implemented or isn't actually there yet, which is a shame.
I tried to get in contact with them at OpenRouter about this and I was interested in working with them in the past, but it's difficult to get in touch with the right people and they are growing very fast. I expect being acquired by Stripe will accelerate those problems in some ways. I have no doubt they will resolve all of these issues eventually and scaling that much that quickly is really hard, so kudos to them, but the road has been pretty lame and taken some wind out of my sails.
irthomasthomas 1 days ago [-]
Openrouter was pretty great before prompt caching became common. Now it is extremely expensive for most individual workflows, unless you spend a lot of work customizing router preferences, and then you still get a worse cache hit rate than using the provider directly. I only keep $5-$10 in OR for occasional testing.
copperx 24 hours ago [-]
Why is caching affected when not using the provider directly?
polski-g 20 hours ago [-]
Use the "only" header?
dackdel 1 days ago [-]
didnt stripe acquire open router? so i assumed its sunset.
geek_at 1 days ago [-]
The best solution to this for me is to self host litellm or a different router and use model aliases. For example I have a model called "coding" and when a new good model comes out I just switch the backend without needing to change the alias or the key in my projects (opencode, etc).
I have a few of them even a smart router called "agents" which will use local models but if it thinks the request might require higher reasoning it's routing to a different model
try-working 1 days ago [-]
I built a router that lets you route between local and cloud models. Link in my profile.
embedding-shape 1 days ago [-]
Yeah, I also built my own "router" for this:
if (process.env.LOCAL_MODEL {
http('localhost:3000/v1/completions')...
} else {
http('api.openrouter.ai/v1/completions')...
}
nvme0n1p1 1 days ago [-]
Brilliant! I'd like to invest $10M. Who should I make the check out to?
vietvu 18 hours ago [-]
Who use openrouter for serious usage? Caching would be really bad unless you use preset to pin to single provider.
npn 1 days ago [-]
I'm confused? Can you just define some presets and call them instead? With preset you can pinpoint a lot of things, especially the providers
runeblaze 1 days ago [-]
genuine question — how has fireworks or baseten or $reputable_inference_provider worked for your use cases? most production workload probably works fine with one of these and another set as fallback, at least so i think
ljlolel 1 days ago [-]
[dead]
notnullorvoid 1 days ago [-]
It will be interesting to see the intersection of this with inference engines like FreeToken which improve distribution of work for MoE models across CPU/RAM and GPU/VRAM.
If all it takes for a competitive model to run locally at good speeds is a used 3090 and some DDR4, then we might be in for the year of local AI.
Have you tried FreeToken yourself? I was hoping to find some benchmarks on their github but took a quick pass at their research paper and it seems they're showing ~2x performance on qwen 3.6 35b when compared to llama.cpp - but llama.cpp is so sprawling and has so many options I find that a difficult comparison.
notnullorvoid 1 days ago [-]
I haven't yet, though plan to when this model is released. The models that I've been daily driving (Gemma 4 26B, Qwen 3.8 27B) have fit nicely on my 3090. I think FreeToken only offers a perf increase for MoE models that you can't feasibly fit in VRAM.
Yeah I'm sometimes unsure how to get best perf out of llama.cpp, and honestly thought it already did what the FreeToken paper discusses, but from everything I've been able to find since llama.cpp has no dynamic expert cache for GPU. An RFC discusses adding such capability and there's impressive results some are claiming from a fork, but I had to stop reading the thread, reading all the LLM generated comments and summaries from people was making me dizzy.
You can already run it locally its just not the same.
It is still slow, a lot slower than what you are used to with claude and co.
And as soon as you increase context size, your memory requirements jump.
Then when it runs for 30 minutes for something claude needs 5, your device will get hot.
And even a used 3090 is apparently now between 1-2k.
notnullorvoid 1 days ago [-]
> It is still slow, a lot slower than what you are used to with claude and co.
That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus.
In general we haven't reached the ceiling for what performance we can get out of consumer hardware. As evidence by FreeToken which hasn't even added MTP/speculative drafting support yet, which will add another boost.
> Then when it runs for 30 minutes for something claude needs 5, your device will get hot.
I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.
> And even a used 3090 is apparently now between 1-2k.
Yeah I guess the price went up significantly in the last couple months, used to be hovering around 1k. 3090 isn't the only option though.
ekidd 1 days ago [-]
> That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus.
Yes, a lot of Qwen3.8 27B setups are actually quite snappy, as long as they fit 100% in VRAM. In my testing, I wouldn't go below 32GB of VRAM, though—you really want a 6-bit quant and 8-bit K/V quants minimum. I've seen too much weirdness out of 4-bit quants since Qwen3.8 shipped. I think it may be damaged more than 3.6 at similar levels of quantization?
If hyperscalers hadn't bought up almost all the fast RAM production for the next several years, 32GB of VRAM would be tolerably cheap—a lot by "home PC" standards, but not terrible by "professional tools" standards. Sadly, the RAM market is amazingly ugly right now.
> I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.
Yeah, running inference on a laptop is likely to run quite hot. But in an ATX case with decent cooling, it's generally a lower load than gaming. One handy tip: Many Nvidia GPUs (and some from other manufacturers) support power limits. For example, limit a 5090 to 400W instead of 600W, and it will run much cooler. You might lose 11% off your tokens/sec (depending on the exact card).
Zylokloto 1 days ago [-]
I have 2 4090 and last time i played around with it, the context window killed it for me.
The normal LLMStudio stuff works great, but then i tried out anything with subagent things or parallel stuff and it trashed my cache and got super sluggish/slowish.
What do you run and how?
nitin7 1 days ago [-]
Which models when run locally come close to Sol and Opus, from your experience? And which harness do you use?
notnullorvoid 1 days ago [-]
Qwen 3.8 27B is so far the closest I have run, though it suffers on speed compared to Gemma4 26B MoE model which I still use. Neither are going to match Opus or Sol though, but they can be as fast or faster depending on what you are using them for.
I don't use a harness, all the mainstream ones I have tried have tanked my productivity. I know that's not a common sentiment, but it's been my experience. None seem built for the way I work. I program mostly in my head first away from keyboard, then go type it out (faster than it would take to describe the solution to an LLM). Also a perfectionist who likes to learn, and tends to work on out of distribution problems. All I need is a simple chat interface for light research, quick small scoped prototypes, and generating simple scripts.
Not saying a harness is out of the question for me, just all I have seen and tested so far are not for me. Maybe if someone builds a more deterministic harness that doesn't rely on plain English skill files that bloat context and only sometimes do what you want.
> We are releasing these architectural improvements ahead of time so that the community can prepare for the upcoming full family of Qwen4 models.
That gives me hope that "full family" means it will include smaller models like 4B.
culi 20 hours ago [-]
What are the use-cases for a model as small as 4B?
big-chungus4 14 hours ago [-]
They are great base models for fine tuning on both text and visual tasks. Many OCR and object grounding models are based on small Qwen models, though they often replace vision encoder with a bigger one.
Qwen3 5-4B is the biggest model I can find tune in my laptop. And when I upgraded the model from Qwen3-4B to Qwen3.5-4B, both vanilla and fine tuned performances jumped significantly on a classification task.
Those models are great when you have very little data or very low diversity of examples, where it's not possible to train a neural net from scratch as it will just memorize the data. The best you can do is fine tune a generalist model that can already do the task for small number of steps until it starts over-fitting, or on some cases you can do even better though RL.
syntaxing 1 days ago [-]
Really looking forward to this, 27B is a struggle with a strix halo and Laguna 2.1 can do stupid things for tooling calls.
cpburns2009 1 days ago [-]
Yeah 27B is way too slow for the Strix Halo. Laguna was better but still slow when I tried it. Qwen3.6 35B is still the best today.
SparkyMcUnicorn 1 days ago [-]
Have you given Ornith-1.5-35B a shot?
It's been a pretty decent step up for me compared to Qwen3.6
Laguna actually performed better than I remembered. I thought it was slower.
htrp 18 hours ago [-]
Have you benchmarked against full precision models for accuracy/ performance?
cpburns2009 17 hours ago [-]
Not full precision. I've only benchmarked 27B across Q3-6 quants using lm-eval. I lack the hardware to bench 27B at BF16 but I might be able to do Q8_0. I haven't gotten around to doing 35B. I really should upload my collection of results to Github or somewhere.
Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6.
pp is prompt processing how fast it processes the prompt. Tg is token generation how fast, it generates tokens.
cyanydeez 1 days ago [-]
it'll hopefully improve with more MoE and half the prefill/generation. I think it's the sweet spot for the strix halo for smarter or vibe tasks.
corysama 1 days ago [-]
So, I know https://cactuscompute.com/needle is designed only to enable tool calling on tiny devices. But, I wonder if anyone has used it as a CPU-side mediator between a tool and a GPU-side local LLM making semi-natural-language tool requests...
pwython 1 days ago [-]
I was already rolling around the idea of a 128GB M5 Max MBP. Now this!
A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.
Eric_WVGG 1 days ago [-]
Just out of curiosity, why run "local-local" when you could just set up a Mini or Studio at home and query it over http? [edit] whole conversation about this in another thread https://news.ycombinator.com/item?id=49433413
I’m personally considering retiring my MBP for a Studio + 15" Air whenever this MBP ages out.
kamranjon 1 days ago [-]
I actually do this with my MBP - it's a LLM server when I'm working - and then when I'm not it's just a really great machine for video editing and other media work.
LeBit 1 days ago [-]
This is the way.
I’m doing that. Mac Mini M4 Pro with 48G RAM as a headless llama.cpp server.
I much prefer using " thin clients " as the interface to the big VMs running in my homelab
sscaryterry 1 days ago [-]
I have a 128GB M5 Max, and it sucks at this stage. 50-70 tok/s might be something...
smcleod 1 days ago [-]
50-70tk/s is what I get on my m5 max on a 5-6bit Qwen 3.8 27B?
Casteil 1 days ago [-]
I don't know what black magic you're up to but I see more like 30-35t/s on a 16" M5 Max using 3.8:27b Q4, regardless of whether it's mlx or gguf.
qwen3.5:122b-a10b is significantly faster at around 60-65.
With MTP? I get 25-30 TPS on a strix halo. 50+ on a M5 max should very doable. Dflash (2) will push your TG even further
Casteil 1 days ago [-]
It's a bit deceptive to state inference speeds without mentioning the additional things you're doing to achieve them
sscaryterry 1 days ago [-]
I tried 8-bit, perhaps I should try 6-bit.
rdsubhas 24 hours ago [-]
How do you folks code at 40-50 tps? With an extremely lightweight harness (pi) and just 8k system and tools context, and ~40tps on qwen 3.8 27B 4-bit on low thinking mode, it still takes me nearly 30-45 mins for a basic coding session...
Does it work? yeah... But I'd pick a subscription anyday...
bicepjai 6 hours ago [-]
There is a finite amount of time left for these companies to become next Facebook/Google, hoarding our interaction and privacy will be a point of contention pretty soon. At that moment, Qwen will be the knight in shining armor.
hgoel 19 hours ago [-]
Do you find subscriptions to be meaningfully faster? I didn't really feel too much of a speed difference compared to Opus.
julianlam 6 hours ago [-]
When you hear stories like "Opus 5 thought for 20 minutes and then denied my request" it really puts wind in this sails of Local LMs
latentsea 13 hours ago [-]
As someone who uses Opus daily for professional work and Qwen3.8-27B for all my private stuff, yes, Opus sub is faster for me, but I'm only rocking an R9700. If you're lucky enough to have sold a kidney on the blackmarket and purchased a 5090 and you're running ninfer, then actually... I think you'd be seeing fairly comparable performance!
irthomasthomas 1 days ago [-]
IDK, prefill speed is a bigger concern for most wokflows, like agent coding, and I heard that this is quite low on macs?
smcleod 1 days ago [-]
That was mainly before the M4 generation when they didn't have matmul instructions.
jasonjmcghee 1 days ago [-]
M5 prefill is much faster than M4.
I've seen benchmarks that show 4-5x faster of M5 Max vs. M4 Max.
For local models you're likely using M5 Max, prefill is low thousands of tokens per second, as opposed to, say high hundreds with M4 Max.
For larger dense models, some fraction of that, but similar multiple.
smcleod 1 days ago [-]
Yes, I have the M5 Max. But there was no matmul acceleration before the M4 which made things a lot slower.
1 days ago [-]
honestlyranked 1 days ago [-]
Alibaba is giving sleepless nights to the tech giants
drannex 1 days ago [-]
To be fair, Alibaba IS a tech giant, one of the biggest in fact.
They are just giving sleepless nights to the western tech giants.
_ache_ 1 days ago [-]
I'm hearing Tencent, Zhipu and Baidu shaking from here.
It's fair to assume BATX / 6 Tigers don't sleep very well either.
WithinReason 1 days ago [-]
Sounds like a line from a fairy tale
Catloafdev 1 days ago [-]
Very curious to see how this compares to Deepseek v4 Flash. I have to assume they wouldn't be releasing this if it was worse.
natrys 1 days ago [-]
Why not? It's not really competing in the same size class.
Besides, as they explicitly wrote here, the main goal for this release is not performance, rather to serve as a reference for inference runtimes about what to implement. So that later Qwen 4 can be released with zero day support.
Catloafdev 1 days ago [-]
Good point, I didn't see that. I guess I categorized them in the same bucket of 'runs on 128gb machines'
Guess Qwen 4 is the one to wait for.
NitpickLawyer 1 days ago [-]
Their "next" variants are usually undercooked, but useful for the community to verify support for inference stacks. This will likely be the same.
hedora 1 days ago [-]
Time to dust off my 128GB strix halo (literally—it’s been dusty, and it’s running a bit warm these days).
Any idea where this model sits according toquality benchmarks? Pre-bubble MSRP on this hardware was $1400, and it draws 200-ish watts, putting it down into consumer territory.
I’m wondering if it can replace claude for llm-friendly coding tasks.
cpburns2009 1 days ago [-]
So back in the Qwen 3.5 release, the 122B-A10B model scored slightly better than the 27B model. I'd expect this new 125B-A6B to perform similarly to the recently released 27B. Qwen3.8 27B is supposed to rival Sonnet/Opus 4.6.
hugmynutus 1 days ago [-]
Qwen3.8/Qwen3.6 has a weird self doubt/thinking too much problem. You can prompt it away. I would say it "approximates" Opus 4.X class models well enough especially for coding/linux problems.
The only reason I stopped using it as much is I was getting 25-35tok/s on Intel B70 (non-quant) which made some responses slow. For a long running/autonomous task, it would probably be sufficient.
eightysixfour 1 days ago [-]
Two things:
- check your temp settings vs. Qwen's recommendations, they specify what it should be in the model card for thinking on, and that reduces some "over" think.
- the model appears to be intentionally designed to do a lot more test-time compute, if you anthropomorphize the tokens, it looks like overthinking and anxiety, but it is just spending compute to get to the end result, so it may not actually help it to prompt down the token spend (depending on the problem)
wongarsu 1 days ago [-]
There is the rule of thumb that if you take the geometric mean of the total and active parameters of an MoE model you get the equivalent size of an equally capable dense model.
If you follow that formula, you would expect a 125b-a6b model to match a 27b model (sqrt(125*6) = 27.3). That does not feel like a coincidence
FuckButtons 1 days ago [-]
Where does that rule of thumb come from?
wongarsu 23 hours ago [-]
That's a great question. I learned it on HN. Some searching around suggests it originated as an empirical observation in the local LLM space around 2023-2024
It's obviously just a rough approximation. Actual scaling laws suggested in published papers are a lot more complex, and even then you run into issues (architecture changes, effects like better training, putting intelligence on a one-dimensional axis is stupid in the first place, etc). But as an approximation it holds up pretty well for normal-ish ratios between active and total parameters
hedora 1 days ago [-]
Thanks. My current stack ranking of anthropic models is:
4.6 ~= 4.8
4.7 much worse.
Fable and newer consistently tells me to pound sand, so I’m not sure what I’m paying $200/month for. 4.8 sometimes does too, but it’s at least usable most of the time.
So, I’d expect this to mostly replace Claude for my workflows. The main tradeoff for me should mostly be token throughput vs. no longer really trusting anthropic.
cyanydeez 1 days ago [-]
I've got the A10B hooked up to deer-flow and it does remarkable well when you dont need to baby sit it.
c16 1 days ago [-]
+1 to the long list of people hoping for Qwen3.8-27b A3B.
NitpickLawyer 1 days ago [-]
They've said no moe for 3.8, and since they're already releasing a qwen4 early preview, they're probably focusing on that arch going forward.
kamranjon 1 days ago [-]
Where did they say that? My understanding of this 3.8-Flash-Next release is that it's a MOE (as per the title of the posting here, 125B a6b)
NitpickLawyer 1 days ago [-]
A bit of context: 3.5 was the last version where they released their entire suite of models 2b-400b. Then 3.6 got a 27b dense and a 35b moe. Then 3.7 was API only, and 3.8 got only the 27b dense. The devs confirmed on twitter that 35b moe would not come. So that's what I meant by 3.8 is not getting a moe.
3.8 next is not really a 3.8 (but I guess they had to disambiguate from the previous next). It's a preview of qwen4 architecture (and it is an moe + ngram), released early as a preview, and to help the community sort out inference before qwen4 releases.
WiSaGaN 1 days ago [-]
You probably meant Qwen3.8-35B-A3B. But judging from some of the words from their team, it seems unlikely unfortunately.
vorticalbox 1 days ago [-]
They normally release a 35b dense and an 27b moe (4B active per token)
For context 35B on my m4 runs at 10 tokens a second, 27B moe runs 50-60 tokens a second.
yencabulator 2 hours ago [-]
27B dense or 35B-A3B MoE. You might be confusing it with Gemma 4 that has a 26B-A4B variant.
cpburns2009 1 days ago [-]
You have your numbers switched. 27B is the dense model and runs slowly on unified memory. 35B (A3B active) runs great on unified memory.
vegnus 1 days ago [-]
I have an m1 max 64gb macbook. Anything I can do to get 3.8 27b at more than 10 tok/s or am I relegated? 3.6 a3b is good but its not as good
big-chungus4 1 days ago [-]
I hope there is going to be a free endpoint... Unlike 35B-A3B, I am nowhere close to running it locally
_ache_ 1 days ago [-]
What will be the requirement, like 128G of RAM and 12G of VRAM ?
embedding-shape 1 days ago [-]
How long is a rope? Technically you could probably run it off a SSD, but it'll be slow as molasses. If you want it "fast", you want it all within GPU and VRAM, who knows what that'd be. If the engram parameters are separate, I guess it'd be like BF16 ~400 GB, FP8 ~200 GB, NVFP4 ~100GB. Otherwise maybe like ~300GB, ~150GB and ~70GB or alike, don't quote me that, only some guesses. The one who waits will see :)
_ache_ 1 days ago [-]
I think a reasonable expectation of MAX requirement to claim "runable on consumer hardware" is to 32G VRAM and 128GB RAM and it run at +10tps.
latentsea 13 hours ago [-]
I've got an R9700 32GB and an RTX 5060 Ti 16GB plus 64GB of system ram. Hoping to be able to run this at around 30 t/s on a Q4 quant. Hoping. Really hoping. Anything below that isn't usable as a daily driver since at deep context it drops quite significantly, so if you start out at say 20 t/s then you'll wind up at like 10 t/s and 20 t/s is already too slow.
embedding-shape 11 hours ago [-]
Ok, so you already know what your expectations of the requirements are, and you aren't interested in more conservative perspectives, why do you ask to begin with?
fkndkfn 1 days ago [-]
I can feel Dario Amodei's tears in the announcement :)
bellowsgulch 1 days ago [-]
Really happy for those with 128GB+ RAM. Sitting here with my Apple M1 Max with 64GB though. Was looking forward to a Qwen3.8-35B-A3B like many others.
dofm 1 days ago [-]
Have you tested Muse Glimmer in low reasoning strength?
Token generation is slow (and prefill is) but you will likely find it solves actual problems faster than Qwen 3.6 35B-A3B.
bellowsgulch 1 days ago [-]
I’ll give it a try! Thanks for the heads up!
dofm 1 days ago [-]
I’m using the Unsloth 4-bit quant.
To change the reasoning strength you just put text in the system prompt.
From memory it is:
Reasoning strength: low
tarruda 1 days ago [-]
Can you share the source for the parameter count (125B A6B)? I didn't see it anywhere in the page.
petu 1 days ago [-]
It was in description under the countdown initially, but was quickly removed.
It also said 51B of n-grams and new attention (IIRC it said "Qwen Sparse Attention").
This is what I copied from the en version of the modelscope page, right when they published it:
> Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.
> Efficient Training and Inference: Significantly reduces training and inference costs. At ~1/9th the training cost，Qwen3.8-Flash-Next achieves comparable capability against Qwen3.7-Plus, while being more capable in areas of coding and cowork.
There was another paragraph about a new attention, but I didn't copy that.
cogman10 1 days ago [-]
Wow. I wasn't expecting this. I thought they were going to do a 35B model instead.
hasteg 1 days ago [-]
As a 5090 owner and local model enthusiast, I was hoping it would be 35B A3B so I could run it myself =(.
Tuna-Fish 1 days ago [-]
The 27B one is great on a 5090.
This one is basically aimed at macs, Strix halo and DGX Spark.
latentsea 13 hours ago [-]
Depends on the rest of your hardware, and how the the n-gram weights work and if they can be streamed from SSD. If they can and you have 64GB system ram then you should actually be able to run it.
cpburns2009 1 days ago [-]
You can run the 27B released last week. I haven't tried it yet myself but the 3.6 version runs great on my 5090.
I've been waiting for the dust to settle on this model so I can find a good runtime setup. I'm definitely bookmarking this. Thanks!
lousken 1 days ago [-]
gpt oss killer? this can easily run on a server cpu with its memory bandwidth
Art9681 1 days ago [-]
You must have hibernated for a year. Most modern 27b models can outperform gpt-oss-120b.
bearjaws 1 days ago [-]
Not exactly surprising given it's a dense model at 2.7x the size of the experts in gpt-oss
stymaar 1 days ago [-]
gpt-oss is long dead though. It's been 8 month since Qwen 3.5 was released.
1 days ago [-]
dmead 1 days ago [-]
This is great. I have a weird system layout (192gb system ram, 8gb vram). the mixture of experts models have been nice when i can run the dense reasoning layers on the gpu (which somehow fit?!) and then the expert on the cpu.
its worked out to to 40 tokens/seconds on their 80b-a3b model. we'll see how much of a hit this is.
BrucecarlL 1 days ago [-]
Waiting for the performance report! Ai hope it can beat DS
isatty 1 days ago [-]
Can I run a fp8 quant with 96gb VRAM?
cpburns2009 1 days ago [-]
Only VRAM? Unlikely unless you can also load the whole model into regular RAM. The previous 3.5 release was 250gb at BF16, so FP8 would likely be around 125gb. Your best best is FP4/Q4.
isatty 1 days ago [-]
Very sad. I try not to go below q8.
vietvu 18 hours ago [-]
This is a gift!
system2 1 days ago [-]
These companies are naming their products worse than I was naming my half-baked software in the 90s as a junior developer.
freddiehdxd 1 days ago [-]
Does it support vision?
nightfuryg 1 days ago [-]
yes
tw1984 1 days ago [-]
Qwen4 sounds exciting
luciandan 1 days ago [-]
Coming soon...
blurbleblurble 1 days ago [-]
gg
david927 1 days ago [-]
Well put and succinctly put. And if OxA is a flash model? it becomes: goodnight
cyanydeez 1 days ago [-]
oooh, I like a6b; that will be nice. 3.5 A10B qwen works really well in deer-flow when you want to seriously vibe code or research and you're just not going to baby sit.
onesandofgrain 1 days ago [-]
where are the humans geez
mrdoe 1 days ago [-]
lol blocked with dns4eu
what a joke this resolver has become
vikasgrac 14 hours ago [-]
[dead]
listingbott 11 hours ago [-]
[dead]
metrofun 1 days ago [-]
[dead]
Alien1Being 1 days ago [-]
[flagged]
latentsea 13 hours ago [-]
Some of us are actually quite excited about this release given the performance of Qwen3.8-27B.
I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.
And, MoE should make it run at a close to usable speed.
I’ll admit though I’m biased because I bought my board for $1600 back before the prices went crazy.
Currently, I think most models are somewhat benchmaxxed, as my own benchmarks and others I know who also keep private benchmarks, don't show nearly as much needle movement as the press releases tend to want you to believe.
I can't wait for local LLMs to mature enough so I can move everything locally, but we're sadly not there yet today.
You just suck at using ai
I've just spent two days testing it qualitatively (after the quantitative results were kind of bad, but promising), the official BF16 weights, and it keeps fucking up very basic things, like doing "rm -rf sessions/$id" without really checking or making sure $id is defined, so it ends up deleting more.
I'd love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either, literally makes no difference. But I too got hopefully reading the sentiment on the internet about Qwen 3.8, but it's just not there (yet) for local professional software development.
Verbatim logs from an open session where this literally just happened:
Oh no indeed :)The 3080ti is 912.4 GB/s
Memory bandwidth M1: 68 GB/s M2: 100 GB/s (47% increase) M3: 100 GB/s (0% increase) M4: 120 GB/s (20% increase) M5: 153 GB/s (27.5% increase)
So, M6: 170 GB/s (11% increase) doesn’t seem impossible, though I would have expected more.
[0]: https://www.jdhodges.com/blog/apple-cpu-compared-m1-m3-m3-m4...
Eg the M4 Max 128GB has a bandwidth speed of 500GB/s+. And that's true for other models as well.
But as you note, the base speed has also increased over the versions.
Historically the Pro doubles the base, the Max doubles the Pro, and the Ultra doubles the Max.
If an M6 ultra were released today it would be 1.36TB/s.
https://blog.exolabs.net/nvidia-dgx-spark/ outlines a combination of a DGX Spark and an M3 Ultra that took advantage of fast prefill on the Nvidia hardware and fast decode on Apple Silicon.
Using the published bandwidths, the math is 170 * 1 and 153 * 8.
Even the GTX 1080, launched ten years ago, has double the bandwidth!
This must be some different way of measuring the bandwidth right? Since they explicitly say this for AI, but the numbers they share don't show that at all. Or I gravely misunderstand something here.
It costs more than the strix to just buy regular ddr5 ram sticks today.
For the most part lately I have been sticking with Qwen 3.8 27b and that thing will easily suck up 64gb of ram. Add in docker with some additional programs running and it's really easy to eat up 128gb of ram.
also can you use it for fine tuning?
A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land.
DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can).
Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use.
Note also prefill with large models is pretty slow on the Strix Halo (300 t/s, maybe). Time to first token is a painful wait, when using it interactively with large models.
I am using the PrismaAQUA
standard 9.7 t/s
+ Dflash2 30 t/s
+ torch-compile 37 t/s
c8 = 177 t/s
Also, 4-bit has measurable intelligence loss. Sometimes worth it, but, at this size models are barely smart enough at 8 or 6.
The output quality is higher. It's held at full precision (not quantized).
These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-par...
OpenRouter does a lot of great work and I really enjoy being able to use different models so easily. I like when a provider is phasing out an older model that still works for my needs and the price is much lower. It seems like such a good win-win.
However, the problem is that many Qwen models have almost no capacity or is so flaky you literally have to just litter your code with a blacklist/whitelist of providers. OpenRouter has some attempts to solve this, but they don't work. In fact, OpenRouter has a lot of really cool stuff that is documented, but if you read the code it's not yet implemented or isn't actually there yet, which is a shame.
I tried to get in contact with them at OpenRouter about this and I was interested in working with them in the past, but it's difficult to get in touch with the right people and they are growing very fast. I expect being acquired by Stripe will accelerate those problems in some ways. I have no doubt they will resolve all of these issues eventually and scaling that much that quickly is really hard, so kudos to them, but the road has been pretty lame and taken some wind out of my sails.
I have a few of them even a smart router called "agents" which will use local models but if it thinks the request might require higher reasoning it's routing to a different model
If all it takes for a competitive model to run locally at good speeds is a used 3090 and some DDR4, then we might be in for the year of local AI.
https://github.com/FlashML-org/FreeToken
Yeah I'm sometimes unsure how to get best perf out of llama.cpp, and honestly thought it already did what the FreeToken paper discusses, but from everything I've been able to find since llama.cpp has no dynamic expert cache for GPU. An RFC discusses adding such capability and there's impressive results some are claiming from a fork, but I had to stop reading the thread, reading all the LLM generated comments and summaries from people was making me dizzy.
RFC here https://github.com/ggml-org/llama.cpp/discussions/24528 which also links to some experimental implementations throughout the thread.
It is still slow, a lot slower than what you are used to with claude and co.
And as soon as you increase context size, your memory requirements jump.
Then when it runs for 30 minutes for something claude needs 5, your device will get hot.
And even a used 3090 is apparently now between 1-2k.
That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus.
In general we haven't reached the ceiling for what performance we can get out of consumer hardware. As evidence by FreeToken which hasn't even added MTP/speculative drafting support yet, which will add another boost.
> Then when it runs for 30 minutes for something claude needs 5, your device will get hot.
I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.
> And even a used 3090 is apparently now between 1-2k.
Yeah I guess the price went up significantly in the last couple months, used to be hovering around 1k. 3090 isn't the only option though.
Yes, a lot of Qwen3.8 27B setups are actually quite snappy, as long as they fit 100% in VRAM. In my testing, I wouldn't go below 32GB of VRAM, though—you really want a 6-bit quant and 8-bit K/V quants minimum. I've seen too much weirdness out of 4-bit quants since Qwen3.8 shipped. I think it may be damaged more than 3.6 at similar levels of quantization?
If hyperscalers hadn't bought up almost all the fast RAM production for the next several years, 32GB of VRAM would be tolerably cheap—a lot by "home PC" standards, but not terrible by "professional tools" standards. Sadly, the RAM market is amazingly ugly right now.
> I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.
Yeah, running inference on a laptop is likely to run quite hot. But in an ATX case with decent cooling, it's generally a lower load than gaming. One handy tip: Many Nvidia GPUs (and some from other manufacturers) support power limits. For example, limit a 5090 to 400W instead of 600W, and it will run much cooler. You might lose 11% off your tokens/sec (depending on the exact card).
The normal LLMStudio stuff works great, but then i tried out anything with subagent things or parallel stuff and it trashed my cache and got super sluggish/slowish.
What do you run and how?
I don't use a harness, all the mainstream ones I have tried have tanked my productivity. I know that's not a common sentiment, but it's been my experience. None seem built for the way I work. I program mostly in my head first away from keyboard, then go type it out (faster than it would take to describe the solution to an LLM). Also a perfectionist who likes to learn, and tends to work on out of distribution problems. All I need is a simple chat interface for light research, quick small scoped prototypes, and generating simple scripts.
Not saying a harness is out of the question for me, just all I have seen and tested so far are not for me. Maybe if someone builds a more deterministic harness that doesn't rely on plain English skill files that bloat context and only sometimes do what you want.
That gives me hope that "full family" means it will include smaller models like 4B.
Qwen3 5-4B is the biggest model I can find tune in my laptop. And when I upgraded the model from Qwen3-4B to Qwen3.5-4B, both vanilla and fine tuned performances jumped significantly on a classification task.
Those models are great when you have very little data or very low diversity of examples, where it's not possible to train a neural net from scratch as it will just memorize the data. The best you can do is fine tune a generalist model that can already do the task for small number of steps until it starts over-fitting, or on some cases you can do even better though RL.
It's been a pretty decent step up for me compared to Qwen3.6
https://news.ycombinator.com/item?id=49362401
Laguna-S-2.1:UD-Q4_K_XL (no MTP) pp=186.4 t/s tg=27.8 t/s
Qwen3.6-35B:UD-Q4_K_XL (with MTP) pp=404.4 t/s tg=83.2 t/s
Qwen3.6-27B:UD-Q4_K_XL (recorded pre-MTP) pp=343 t/s tg=12.1 t/s
Laguna actually performed better than I remembered. I thought it was slower.
Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6.
A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.
I’m personally considering retiring my MBP for a Studio + 15" Air whenever this MBP ages out.
I’m doing that. Mac Mini M4 Pro with 48G RAM as a headless llama.cpp server.
I much prefer using " thin clients " as the interface to the big VMs running in my homelab
qwen3.5:122b-a10b is significantly faster at around 60-65.
Does it work? yeah... But I'd pick a subscription anyday...
I've seen benchmarks that show 4-5x faster of M5 Max vs. M4 Max.
For local models you're likely using M5 Max, prefill is low thousands of tokens per second, as opposed to, say high hundreds with M4 Max.
For larger dense models, some fraction of that, but similar multiple.
They are just giving sleepless nights to the western tech giants.
Besides, as they explicitly wrote here, the main goal for this release is not performance, rather to serve as a reference for inference runtimes about what to implement. So that later Qwen 4 can be released with zero day support.
Guess Qwen 4 is the one to wait for.
Any idea where this model sits according toquality benchmarks? Pre-bubble MSRP on this hardware was $1400, and it draws 200-ish watts, putting it down into consumer territory.
I’m wondering if it can replace claude for llm-friendly coding tasks.
The only reason I stopped using it as much is I was getting 25-35tok/s on Intel B70 (non-quant) which made some responses slow. For a long running/autonomous task, it would probably be sufficient.
- check your temp settings vs. Qwen's recommendations, they specify what it should be in the model card for thinking on, and that reduces some "over" think.
- the model appears to be intentionally designed to do a lot more test-time compute, if you anthropomorphize the tokens, it looks like overthinking and anxiety, but it is just spending compute to get to the end result, so it may not actually help it to prompt down the token spend (depending on the problem)
If you follow that formula, you would expect a 125b-a6b model to match a 27b model (sqrt(125*6) = 27.3). That does not feel like a coincidence
It's obviously just a rough approximation. Actual scaling laws suggested in published papers are a lot more complex, and even then you run into issues (architecture changes, effects like better training, putting intelligence on a one-dimensional axis is stupid in the first place, etc). But as an approximation it holds up pretty well for normal-ish ratios between active and total parameters
4.6 ~= 4.8
4.7 much worse.
Fable and newer consistently tells me to pound sand, so I’m not sure what I’m paying $200/month for. 4.8 sometimes does too, but it’s at least usable most of the time.
So, I’d expect this to mostly replace Claude for my workflows. The main tradeoff for me should mostly be token throughput vs. no longer really trusting anthropic.
3.8 next is not really a 3.8 (but I guess they had to disambiguate from the previous next). It's a preview of qwen4 architecture (and it is an moe + ngram), released early as a preview, and to help the community sort out inference before qwen4 releases.
For context 35B on my m4 runs at 10 tokens a second, 27B moe runs 50-60 tokens a second.
Token generation is slow (and prefill is) but you will likely find it solves actual problems faster than Qwen 3.6 35B-A3B.
To change the reasoning strength you just put text in the system prompt.
From memory it is:
It also said 51B of n-grams and new attention (IIRC it said "Qwen Sparse Attention").
edit: here's a random screenshot https://x.com/AiBattle_/status/2092210011858460819/photo/1
> Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.
> Efficient Training and Inference: Significantly reduces training and inference costs. At ~1/9th the training cost，Qwen3.8-Flash-Next achieves comparable capability against Qwen3.7-Plus, while being more capable in areas of coding and cowork.
There was another paragraph about a new attention, but I didn't copy that.
This one is basically aimed at macs, Strix halo and DGX Spark.
its worked out to to 40 tokens/seconds on their 80b-a3b model. we'll see how much of a hit this is.
what a joke this resolver has become