While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI.
Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the present ROI crisis for LLM-based AI.
If you are into small local models I highly recommend vibe thinker. It's a model trained specifically for reasoning. Basically a problem solver. When compared with other models, on math problems benchmarks, it's closer to models hundred times its size than ten times its size which it beats comfortably.
It supports long contexts on limited VRAM and is blazing fast.
Tangential/meta: Holy shit, I've never seen a thread where almost half the comments are dead (and LLM written), especially for a post that's (currently) at 86 points and 20 comments (4x ratio is "pretty good quality" post signal generally for me).
This seems really interesting - I was curious about this line from the website.
“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”
How does soup auto tune the hyper parameters and make some of these more complex training decisions?
Do not buy a 4 GB card for this. Mine is an RTX 3050 Laptop, I picked it because it
is boring hardware that many people already have.
If you are buying, buy VRAM. At 0.5B where I could measure both, resident training
was 1.43x faster than streaming. And if you do stream, system RAM matters more, the
base sits there and has to page-lock. 16 GB is about the floor for 8B.
Because streaming only removes the decoder stack. The embeddings and lm_head stay
resident, that is 2.10 GB of the 3.32 GB peak on 8B. And the logits tensor scales
with batch x seq x vocab, not with depth.
So it goes from "whole model must fit" to "embeddings plus one layer plus logits
must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA.
Should be clearer, my fault.
The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.
Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.
Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.
Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.
It isn't. Kazakh and Russian.
I said this further down but that comment is dead so you would not have
seen it. The later replies are mine, written by me.
22 comments:
Small open weight local models are the future.
While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI.
Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the present ROI crisis for LLM-based AI.
If you are into small local models I highly recommend vibe thinker. It's a model trained specifically for reasoning. Basically a problem solver. When compared with other models, on math problems benchmarks, it's closer to models hundred times its size than ten times its size which it beats comfortably.
It supports long contexts on limited VRAM and is blazing fast.
https://github.com/WeiboAI/VibeThinker
this is great! thanks for sharing
Wat. Those are some crazy benchmark scores for a 3B model
There are some samples of training data in this folder - https://github.com/MakazhanAlpamys/Soup/tree/main/examples/d...
They're all very short though. Anyone got a good rule for how much data of this nature is needed to successfully fine-tune a model of this size?
Tangential/meta: Holy shit, I've never seen a thread where almost half the comments are dead (and LLM written), especially for a post that's (currently) at 86 points and 20 comments (4x ratio is "pretty good quality" post signal generally for me).
Those are mine. I used an LLM for my replies and that was a bad call, I said so further down.
Writing them myself now.
I run a fine-tuned 4B for AML compliance at community banks — the ROI math is exactly this
Could you describe more? Your process, resources, use cases, user feedback
How much data do you need to fine tune a model?
Depends what you change. Format or style, few hundred rows is often enough. A task the model already half knows, few thousand.
New facts is where people waste a week. The model comes back wrong in a new way. Use RAG for facts.
This seems really interesting - I was curious about this line from the website.
“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”
How does soup auto tune the hyper parameters and make some of these more complex training decisions?
Looks cool, I'll try this when I get home.
I have a couple of comments about https://trysoup.dev
> Get Started for Free
Does this mean that this will not be free at some point?
The website is difficult to read (gray on black doesn't work well for me).
This looks cool - what 4 GB GPU laptop do you recommend?
Do not buy a 4 GB card for this. Mine is an RTX 3050 Laptop, I picked it because it is boring hardware that many people already have. If you are buying, buy VRAM. At 0.5B where I could measure both, resident training was 1.43x faster than streaming. And if you do stream, system RAM matters more, the base sits there and has to page-lock. 16 GB is about the floor for 8B.
Why is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
Because streaming only removes the decoder stack. The embeddings and lm_head stay resident, that is 2.10 GB of the 3.32 GB peak on 8B. And the logits tensor scales with batch x seq x vocab, not with depth.
So it goes from "whole model must fit" to "embeddings plus one layer plus logits must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA. Should be clearer, my fault.
Author here.
The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.
Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.
Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.
Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.
Measurement records, including the ones I threw away: https://github.com/MakazhanAlpamys/Soup/tree/main/benchmarks
Write-up: https://doi.org/10.5281/zenodo.21771064
Happy to answer anything about the scheduler or the correctness protocol.
Can you write one answer that isn't AI-sourced?
Can you just get over it and read what's already there? Anyway, your loss.
If their native language isn’t English then I’ll come off my pedestal
It isn't. Kazakh and Russian. I said this further down but that comment is dead so you would not have seen it. The later replies are mine, written by me.