> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.
Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.
For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU.
For decode, you are bandwidth bound, and it is difficult to batch multiple output tokens together. There is no benefit to sending your weights to the GPU because even if it internally has insane bandwidth, you are still bottlenecked by system RAM. This is the number people usually talk about when they say they are getting a certain tk/s.
For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.
If compute is not the bottleneck, memory is easy-ish to produce (the hard part is mostly on the fab side); what stops a Chinese NVIDIA (huawei) from being 10x cheaper?
Making memory is easy. Packaging that memory within a few millimeters of a piece of silicon using TSVs and maintaining signal integrity on a 1024 bit bus is really, really hard.
LLMs aren’t all that compute constrained or even memory constrained. It’s just that pushing dozens of terabits per second through a piece of silicon is a physics problem.
> I guess they're just welding the memory to the CPU chip, but still curious.
Unified memory is more of an architectural and performance characteristic, and does not imply much about the physical layout of the machine. Most unified memory PCs not from Apple don't have the memory on the same package as the SoC. For stuff like AMD Strix Halo and NVIDIA DGX Spark, it's just standard LPDDR packages soldered on the motherboard in the general vicinity of the SoC, and the only difference from mainstream laptops for the past decade+ is that the memory bus is twice as wide.
Yes. The memory is just located very close to the cpu with wires "welded" directly to it. This allows the memory to be run as fast as possible but it's still a RAM component.
The cache parts of memory are on the CPU itself but they are on the order of MB not GB.
They are usually the same family, LPDDR is used for amd and macs, but the fabs are the same as the most expesive HBM memory, if they have a choice they are going to produce the ones that they can sell for more $$.
The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
Yeah this is such a funny thing going around. Try to run or train a small/medium sized model on a MacBook. It doesn’t go very good compared to a dedicated gpu
This is likely the right path in the future but it isn’t there yet today
Sorry, thought we were talking about tokens. M5 Max is great for bandwidth and I’m looking forward to seeing what Apple does for AI inference in the M7. The 6000 kills everything else when it comes to TTFT and tokens/s.
I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.
NAND[0] has a fun thing called "read disturbance" where repeated reads from disk will eventually flip 0s to 1s. You have to erase and rewrite the block before the bits flip[1], or you lose the data, but doing so is the same amount of wear as a write.
[0] I heard this being an issue with TLC, I don't know if it also applied to MLC or SLC.
[1] I suspect in practice they use an error correction code and rewrite blocks that read with corrected errors.
It's kinda irresponsible to talk about read disturbance without clarifying that it takes an extremely large number of reads to cause a read disturb error, and it can be corrected by a single rewrite of the data. Read disturb errors are something SSD engineers need to account for, but from an end user perspective it's a smaller problem by multiple orders of magnitude than write endurance, which is already rarely a real problem in practice.
31 comments:
> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
https://www.microcenter.com/product/711961/amd-ryzen-ai-halo...
I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.
Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.
There are two phases to LLMs:
1) prefill
2) decode
For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU.
For decode, you are bandwidth bound, and it is difficult to batch multiple output tokens together. There is no benefit to sending your weights to the GPU because even if it internally has insane bandwidth, you are still bottlenecked by system RAM. This is the number people usually talk about when they say they are getting a certain tk/s.
It's slower than the 4:1 ratio would imply, but it does indeed work.
Things get really slow if the model doesn't for in vram + ram and you have to go from disk to ram to vram.
For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.
Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically
If compute is not the bottleneck, memory is easy-ish to produce (the hard part is mostly on the fab side); what stops a Chinese NVIDIA (huawei) from being 10x cheaper?
Making memory is easy. Packaging that memory within a few millimeters of a piece of silicon using TSVs and maintaining signal integrity on a 1024 bit bus is really, really hard.
LLMs aren’t all that compute constrained or even memory constrained. It’s just that pushing dozens of terabits per second through a piece of silicon is a physics problem.
I think it's mostly the ramp-up time, but ChangXin Memory Technologies (CXMT) is basically aspiring to do just this.
"Can't" is not really correct.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
Do unified memory CPUs suffer from the same memory shortages as normal memory?
I guess they're just welding the memory to the CPU chip, but still curious.
> I guess they're just welding the memory to the CPU chip, but still curious.
Unified memory is more of an architectural and performance characteristic, and does not imply much about the physical layout of the machine. Most unified memory PCs not from Apple don't have the memory on the same package as the SoC. For stuff like AMD Strix Halo and NVIDIA DGX Spark, it's just standard LPDDR packages soldered on the motherboard in the general vicinity of the SoC, and the only difference from mainstream laptops for the past decade+ is that the memory bus is twice as wide.
Yes. The memory is just located very close to the cpu with wires "welded" directly to it. This allows the memory to be run as fast as possible but it's still a RAM component.
The cache parts of memory are on the CPU itself but they are on the order of MB not GB.
They are usually the same family, LPDDR is used for amd and macs, but the fabs are the same as the most expesive HBM memory, if they have a choice they are going to produce the ones that they can sell for more $$.
The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.
Uhh the 5090 alone is double the cost of their quoted PC prices.
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
Yeah this is such a funny thing going around. Try to run or train a small/medium sized model on a MacBook. It doesn’t go very good compared to a dedicated gpu
This is likely the right path in the future but it isn’t there yet today
To be fair it's "only" half the throughput of a 4090 and a third of an RTX 6000. Significant but not an order of magnitude.
Those are the ratios for memory bandwidth, but the GPUs have a much higher ratio for compute, and that affects prefill rate / TTFT, right?
An old ada Rtx 6000 maybe. A Blackwell RTX Pro 6000 is an order of magnitude faster and has 96gb.
That's not what I'm seeing. It is much faster but not an order of magnitude. Not trying to be pedantic, only setting expectations.
"The Blackwell RTX PRO 6000 provides up to 1,792 GB/s of memory bandwidth, while the 40-core Apple M5 Max tops out at 614 GB/s"
Sorry, thought we were talking about tokens. M5 Max is great for bandwidth and I’m looking forward to seeing what Apple does for AI inference in the M7. The 6000 kills everything else when it comes to TTFT and tokens/s.
For sure. Clearly Nvidia mops the floor with the competition. I'm looking forward to M6/M7 and to see if Apple wants a bigger piece of the pie.
Let's also ensure the SSD doesn't age prematurely.
I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.
You'd need your OS to support, and be configured to use, a disk mounting option that disables file access timestamps, otherwise reads ARE writes.
NAND[0] has a fun thing called "read disturbance" where repeated reads from disk will eventually flip 0s to 1s. You have to erase and rewrite the block before the bits flip[1], or you lose the data, but doing so is the same amount of wear as a write.
[0] I heard this being an issue with TLC, I don't know if it also applied to MLC or SLC.
[1] I suspect in practice they use an error correction code and rewrite blocks that read with corrected errors.
It's kinda irresponsible to talk about read disturbance without clarifying that it takes an extremely large number of reads to cause a read disturb error, and it can be corrected by a single rewrite of the data. Read disturb errors are something SSD engineers need to account for, but from an end user perspective it's a smaller problem by multiple orders of magnitude than write endurance, which is already rarely a real problem in practice.
It is and it doesn't. You only get into disk writes if the system starts paging out to disk.