"Processor-in-Memory", is an age-old idea which was even explored in a doctoral thesis from 1998: "Computational RAM: A Memory-SIMD Hybrid"*. At the time, the motivation was, "Between a DRAM chip and a microprocessor mounted on a circuit board, we can only get a few hundred megabytes per second of bandwidth. However, inside the memory chip itself, there's hundreds of gigabytes (terabits) of bandwidth available, so why not move some of the logic there?". And then the author pushed it to the extreme where there was a wide array of 1-bit processors at each column of DRAM. (Note: the reason why true PIM isn't viable is because of the radically different semiconductor process used for DRAM vs. high-speed digital logic, i.e.: your digital logic circuits will perform poorly on a "DRAM process").
The designers of HBM5 had the same observations and thought, "why don't we put a logic chip at the bottom of the stack of HBM DRAM chips and get extremely high memory bandwidth?" But, in looking at this, you'll also quickly realize this is almost exactly what Intel and Micron pitched 15 years ago as Hybrid Memory Cube** (which lost on the market compared to the simpler, less costly, and more flexible alternative, HBM)
I think it goes back further than 1998. Several old systems had special memory locations where the numbers written would then read back changed. The Apollo Guidance computer for instance didn’t have a shift-right or shift-left cpu instruction. They would write data to a specific area of memory that would modify the data to shift bits around.
Although I’m sure it’s quite different. The cpu and memory hardware is so completely different. And basic bit manipulation is hardly computation.
inigyou
That's transport triggered architecture and it's the same hardware with a different ISA. This is different hardware.
hn_acc1
Duncan Elliott taught my graduate (IIRC) computer architecture course while writing that thesis - we all talked about C-RAM for a while.. And I think it was a rehash of an even older paper / discussion about that idea.
jhgorrell
I have been reading about this for the last 25 years.
So, yea, in another 25 years "this will be a thing".
timcobb
Age-old... 1998... :grimace:
Bratmon
Remember: 1998 is as old now as 1970 was in 1998.
schaefer
I don’t think he was asking you to twist the knife.
timcobb
Eh it's okay. It's just that, man, computers were already so well along by 1998, when I hear "age-old" I think like the 60s, 70s, _maybe_ 80s--but 90s?! oh well! I did a few double takes assuming my mind had flipped the 9 and 8 in 98, but no :D.
margalabargala
At this point, "age-old" means "64-bit processors were not the default".
honr
Compute In Memory has been most of what electronics research (more accurately, the parts I am interacting with) talking about this year. There are a few trends and I am curious to see which ones survive in the market. RAM designers are hot in the market and job openings at the moment.
Traubenfuchs
> RAM designers
How many RAM designer jobs are there in the world? 100? 1000?
KellyCriterion
Id guess its a mixture: "electronic designs in general" then with "a focus on DRAM", so switching to other components is possible?
jcranmer
Processing in memory was one of the extra topics we covered in my grad computer architecture class. Fifteen years ago. And even then, it was very clearly an old-hat idea.
One of the problems with trying to move a lot of ALUs close to the DRAM logic is that ALUs are pretty hot, and DRAM is some of the most sensitive components to heat. There are some other issues too, like the process manufacturing differences mentioned by a sibling comment.
StilesCrisis
"DRAM banks already provide most of the bandwidth; the external pins cannot expose it." Closed the tab.
jmalicki
Why would you want to read something someone was so sloppy at they didn't improve it with AI?
superb_dev
"Improving" it with AI is the sloppy way. Take some time and flush out your ideas the old fashioned way. I promise you the end result will be better and you will come away with a more thorough understanding.
MBCook
So they built a very specific operation into the RAM. This RAM would only be useful for AI, otherwise the transistors are just wasted.
And if we fund a different set of operations actually work better for AI you can’t adjust to that either.
Seems extremely limited. But otherwise wouldn’t you just end up with something like a massive number of tiny microcontrollers each accessing a set of memory? The Transputer again.
senshan
Why does not the article say anything on how KV-cache is handled with in-DRAM processing? Is this trivially obvious?
peter_d_sherman
I like the idea of a CPU/GPU/TPU on RAM (look at those gorgeous bandwidth numbers!) -- but I also wonder what problems it may potentially bring in the future... I think it would be a good idea that if RAM is to implement processors / PIM on it in the future, that each RAM stick should have an external DIP "safety" switch on the side which switches off such processing. If this switch is flipped by the user, then the stick simply falls back to operating as normal "dumb" RAM, driven by external CPU control. Also, it would be a good idea to have debug ports, debug proxying, single step capability, etc., in these futuristic RAM sticks, as well...
Related:
"SIMDRAM: A Framework for Bit-Serial SIMD Processing Using DRAM Extended Abstract" (2020):
>"We leverage the SIMDRAM framework to accelerate seven application kernels from machine learning, databases, and image processing (VGG-13 [63], VGG-16 [63], LeNET [38], kNN [42], TPC-H [66], BitWeaving [45], Brightness [28]). SIMDRAM provides up to 2.5× speedup for the kernels compared to Ambit [58]. Compared to a CPU and a high-end GPU, SIMDRAM is 257× and 31× more energy efficient, while providing 93 × and 6× higher throughput, respectively."
wtallis
You seem to think that "compute in memory" would mean the disappearance of commodity DRAM parts and their replacement with parts that had compute capabilities. This is impossible. Adding compute to memory will make it more expensive, so it will get zero adoption outside of the niches where it actually proves to be useful and valuable. Nobody is going to be putting compute in memory chips onto standard consumer DIMMs.
petra
Maybe it would make sense for a single type of a desktop/laptop, for some use cases.
I would be more expensive, but maybe the performance would be worth it.
peter_d_sherman
CPU's are silicon where some of the gates, some of the circuits that are etched into that silicon are turned into RAM.
That is, Modern CPU's (as opposed to truly ancient ones, around 50 or more years ago) implement caches -- regions of gates/circuits/transistors or mostly transistors -- which implement the equivalent of RAM.
When a silicon wafer specifically destined to become a stick of consumer RAM is etched, most of its space is etched in a gate/circuit/transistor (or mostly transistor!) pattern that implements RAM.
There may be other timing/coordination/refresh etc. circuits etched on the region of the silicon wafer which is to become the RAM, but those are usually a small portion of the space.
But, fundamentally CPU's and RAM -- are created from the same substrate, transistors and other very small electronic components etched into a silicon wafer, although these days due to manufacturing constraints and cost issues, RAM usually uses a larger process node, a larger (less expensive, older) size, i.e., the latest CPU might use 2nm process, and consumer RAM might use 10nm or 12nm process.
But, fundamentally, at the lowest level, both RAM and CPU are mostly transistors and other minature electronic components.
So, if they're both the same at that level (ignoring process node size), then my engineering question to you is as follows:
Why exactly can a CPU contain RAM electronic circuitry, but RAM can't contain CPU electronic circuitry, if both, at the lowest level, the level of the minaturized transistor, the minaturized electronic circuit, are created from the same substrate, the same basic building blocks?
>You seem to think that "compute in memory" would mean the disappearance of commodity DRAM parts and their replacement with parts that had compute capabilities.
You seem to imply that I think that, but I am merely speculating on possible futures, possible outcomes...
>This is impossible.
That might be speculation on your part...
>Nobody is going to be putting compute in memory chips onto standard consumer DIMMs.
That also might be speculation on your part...
Seymour Cray was heralded as a genius back in time in computer history, because he put memory physically closer to the CPU. Since his time, compute and memory have been getting closer and closer together, in various forms, including, but not limited to CPU caches, or say, 64GB HBM on Intel Xeon Max 9480... Cerebras, for example, mixes as much memory and compute as they can, on the same wafer...
But historically, memory and compute have been getting closer and closer together... that's because higher performance (which is the goal, the continual holy grail of the computing industry) typically requires less space, less latency, less delay, between the two...
Is putting a CPU or other processing unit/computing device/computing circuitry into a stick of DRAM a good idea?
Well, that I don't know, but the industry, always in search of better performance, might head that way, in whole or in part, in the future... we don't as-of-yet-know...
articulatepang
There’s a big difference between DRAM and SRAM. SRAM stores bits in the state of a couple transistors. DRAM stores bits in the charge of a capacitor.
A CPU is mostly transistors, so SRAM is made of the same “stuff” as the CPU logic itself.
This makes it easy to put SRAM on the same chip as a CPU, but difficult to put dense amounts of DRAM on that same chip. You could etch CPU and DRAM with the same process but both would suffer in density and therefore speed, cost and power efficiency. That’s the fundamental reason why DRAM and CPU/GPU are on different chips with a fast interconnect between them.
This is so even in devices like the iPhone where one designer controls and designs the whole thing and could, in theory, fab a single design with both. But they don’t.
(There are some old exceptions to this like the PS2 shipping with a small amount of on-die DRAM. And process incompatibility is not the only reason for keeping the two chips separate. But if the processes weren’t incompatible there would certainly be high-end fast CPUs/GPUs with big gobs of DRAM.)
wtallis
I'm going to ignore the first half of your comment, because it really only serves to disclose the limits of your knowledge.
The problem with your rambling and speculation is that you're not making any attempt to think about what conditions would be necessary for a technology to succeed in the market and gain widespread adoption across many market segments. I stand by my assertion that it is impossible for compute-in-memory chips to entirely replace commodity DRAM, especially in any market segment where the compute portion of those memory chips is not critically valuable. Any remotely plausible, useful compute-in-memory design will have significantly lower capacity and higher cost than pure commodity DRAM; this is not a low-overhead added feature. Any market segment where the compute capabilities of compute-in-memory were so marginally valuable that a non-trivial number of the customers wanted to disable the compute is a market segment where everybody would still be buying mostly ordinary DRAM.
peter_d_sherman
>"I'm going to ignore the first half of your comment, because it really only serves to disclose the limits of your knowledge."
"The true knowledge consists of knowing that one knows nothing..."
- Socrates
See, I'm agreeing with you(!), at least on that part of your message, at least on that comment!
You see, I truly don't know...
And, I don't truly know what the future as-of-yet holds...
I suggest and submit to you that this is the foundation of intellectual humility, which is the starting point of all forms of inquiry which have the potential to lead to genuine knowledge, at least in the specific areas of inquiry, and at least for specific amounts of time ("the only constant is change", aka, many forms of specific knowledge, if acquired, are time bounded, bounded in time -- but I digress!)
>"The problem with your rambling and speculation..."
My inquiry into the nature of this phenomena, but sure, your (perceived!) "rambling and speculation" on my part (I'm chuckling to myself, because that's rather funny from my perspective(!) -- yours is the first message on HN that has truly made me laugh!)
>is that you're not making any attempt to think about what conditions would be necessary for a technology to succeed in the market and gain widespread adoption across many market segments. I stand by my assertion that it is impossible for compute-in-memory chips to entirely replace commodity DRAM, especially in any market segment where the compute portion of those memory chips is not critically valuable. Any remotely plausible, useful compute-in-memory design will have significantly lower capacity and higher cost than pure commodity DRAM; this is not a low-overhead added feature. Any market segment where the compute capabilities of compute-in-memory were so marginally valuable that a non-trivial number of the customers wanted to disable the compute is a market segment where everybody would still be buying mostly ordinary DRAM.
I see...
So the premise of your argument is The Market (specifically the future consumer commodity DRAM market)... correct?
jcranmer
> Why exactly can a CPU contain RAM electronic circuitry, but RAM can't contain CPU electronic circuitry, if both, at the lowest level, the level of the minaturized transistor, the minaturized electronic circuit, are created from the same substrate, the same basic building blocks?
There's different kinds of RAM. CPUs are using SRAM for their caches (which usually have ~6 transistors per bit cell, IIRC, although I'm not a hardware guy, so take this stuff with a grain of salt). Your RAM sticks are using DRAM, which are 1 transistor and one capacitor per bit cell.
Now, this is where my understanding of the actual physical issues really gets sketchy, but the processes that are good at making the CPUs just aren't good at making DRAM, and vice versa. (I would guess that it has specifically to do with the capacitor, but that's purely a guess.) SRAM isn't a problem for the CPU processes, because SRAM cells are closer to the other kind of logic you set up in CPUs.
inigyou
That's like saying your CPU should come with a switch that makes it a four-function calculator. Which isn't how any of this works.
RetroTechie
AI focussed - as so much these days. Some corp. please put general-purpose cores in that RAM?
MadnessASAP
AI inference is basically 1 operation iterated over every single model weight. That's what's makes computer in memory attractive and feasible.
General purpose workloads would require much more silicon real estate and are not (typically) nearly as bandwidth hungry. Just like not all CPU workloads benefit from being adapted to a GPU, even fewer will benefit from being adapted to processing in RAM.
Lerc
AI is not intrinsically matmul. Current models are a framing of the problem as matmul to enable the bulk operations necessary. There are many calculations that can (and have been) reframed into matrix form to achieve similar results. More than once I have seen a matmul version of something that I thought couldn't have been representable that way. I'm not sure if the greater benefit is the performance or the ability to conceptually manage the problem in you head.
Not all algorithms can be converted of course, but a lot of that is because there are some inherently unparallelizable problems.
Tuna-Fish
Many algorithms can be reasonably converted into matmul. Not many algorithms can be reasonably converted into very low precision matmul. And that's all that these things are going to do.
ghaff
I looked at this for a client close to 20 years ago. The problem is that you need to get software support for it. Arguably easier these days with open source but still doesn't magically happen.
jmalicki
The nice thing is that for GPUs you're rarely writing directly against the metal but instead pseudocode lowered by the driver; the driver could know to just lower it this way. On an NVidia GPU they control the whole stack and don't need to get a further ecosystem going there. Possibly the same for Apple.
ghaff
Apple is such a vertical stack, I'm not sure how to generalize from that. Certainly the same was true on IBM mainframes. And, when I wrote open source, I could probably have written abstraction more broadly.
jmalicki
I am thinking the first place this appears is on GPUs and maybe Apple with unified memory, it is not going to be on PCs.
monster_truck
Bro there's an IANUS acronym in this article and nobody has said anything. Shame on you all
PIM (XBM, ZAM, etc) are all cope. This 600GB of throughput is ~4.5x higher than the 130 you get from LPDDR5X on a 128 bit bus yet their deltas are only 2.2-3x? I just don't see this being a corner worth cutting over simply paying the cost of a wider bus. Think of all the fun new rowhammer shit this will be able to do to itself
moron4hire
What ever happened to the idea of using memristors for this?
api
Integrated RAM with tensor processing seems like the end game for silicon that's actually efficient at running AI models.
Or integrated flash with TPUs? That would be awesome. Flash your model onto a model-processor and run it.
wtallis
I wonder if we might eventually see stacking of NAND, logic, and DRAM dies together so each can still be fabbed on their specialized process. Getting all three on one wafer is extremely impractical, but having them all connected with TSVs or other bonding might be feasible, if the density of the respective dies works out right to give you the right ratio of storage for weights, working memory, and compute.
sroussey
It’s SRAM that’s expensive, and more so as the nodes get smaller since the logic gets smaller but not the SRAM, which means it takes up a larger percent of die area over generations.
wtallis
In case you haven't been paying attention, DRAM and NAND are also pretty expensive these days, relative to where they were a year ago.
The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.
jeffbee
Isn't Intel effectively betting the farm on this right now? Other than stacking technology, what do they have that's even a little interesting?
The designers of HBM5 had the same observations and thought, "why don't we put a logic chip at the bottom of the stack of HBM DRAM chips and get extremely high memory bandwidth?" But, in looking at this, you'll also quickly realize this is almost exactly what Intel and Micron pitched 15 years ago as Hybrid Memory Cube** (which lost on the market compared to the simpler, less costly, and more flexible alternative, HBM)
* https://www.eecg.toronto.edu/~stumm/Theses/Elliott-PhD98.pdf
** https://en.wikipedia.org/wiki/Hybrid_Memory_Cube
Although I’m sure it’s quite different. The cpu and memory hardware is so completely different. And basic bit manipulation is hardly computation.
How many RAM designer jobs are there in the world? 100? 1000?
One of the problems with trying to move a lot of ALUs close to the DRAM logic is that ALUs are pretty hot, and DRAM is some of the most sensitive components to heat. There are some other issues too, like the process manufacturing differences mentioned by a sibling comment.
And if we fund a different set of operations actually work better for AI you can’t adjust to that either.
Seems extremely limited. But otherwise wouldn’t you just end up with something like a massive number of tiny microcontrollers each accessing a set of memory? The Transputer again.
Related:
"SIMDRAM: A Framework for Bit-Serial SIMD Processing Using DRAM Extended Abstract" (2020):
https://arxiv.org/html/2012.11890v1
>"We leverage the SIMDRAM framework to accelerate seven application kernels from machine learning, databases, and image processing (VGG-13 [63], VGG-16 [63], LeNET [38], kNN [42], TPC-H [66], BitWeaving [45], Brightness [28]). SIMDRAM provides up to 2.5× speedup for the kernels compared to Ambit [58]. Compared to a CPU and a high-end GPU, SIMDRAM is 257× and 31× more energy efficient, while providing 93 × and 6× higher throughput, respectively."
I would be more expensive, but maybe the performance would be worth it.
That is, Modern CPU's (as opposed to truly ancient ones, around 50 or more years ago) implement caches -- regions of gates/circuits/transistors or mostly transistors -- which implement the equivalent of RAM.
When a silicon wafer specifically destined to become a stick of consumer RAM is etched, most of its space is etched in a gate/circuit/transistor (or mostly transistor!) pattern that implements RAM.
There may be other timing/coordination/refresh etc. circuits etched on the region of the silicon wafer which is to become the RAM, but those are usually a small portion of the space.
But, fundamentally CPU's and RAM -- are created from the same substrate, transistors and other very small electronic components etched into a silicon wafer, although these days due to manufacturing constraints and cost issues, RAM usually uses a larger process node, a larger (less expensive, older) size, i.e., the latest CPU might use 2nm process, and consumer RAM might use 10nm or 12nm process.
But, fundamentally, at the lowest level, both RAM and CPU are mostly transistors and other minature electronic components.
So, if they're both the same at that level (ignoring process node size), then my engineering question to you is as follows:
Why exactly can a CPU contain RAM electronic circuitry, but RAM can't contain CPU electronic circuitry, if both, at the lowest level, the level of the minaturized transistor, the minaturized electronic circuit, are created from the same substrate, the same basic building blocks?
>You seem to think that "compute in memory" would mean the disappearance of commodity DRAM parts and their replacement with parts that had compute capabilities.
You seem to imply that I think that, but I am merely speculating on possible futures, possible outcomes...
>This is impossible.
That might be speculation on your part...
>Nobody is going to be putting compute in memory chips onto standard consumer DIMMs.
That also might be speculation on your part...
Seymour Cray was heralded as a genius back in time in computer history, because he put memory physically closer to the CPU. Since his time, compute and memory have been getting closer and closer together, in various forms, including, but not limited to CPU caches, or say, 64GB HBM on Intel Xeon Max 9480... Cerebras, for example, mixes as much memory and compute as they can, on the same wafer...
But historically, memory and compute have been getting closer and closer together... that's because higher performance (which is the goal, the continual holy grail of the computing industry) typically requires less space, less latency, less delay, between the two...
Is putting a CPU or other processing unit/computing device/computing circuitry into a stick of DRAM a good idea?
Well, that I don't know, but the industry, always in search of better performance, might head that way, in whole or in part, in the future... we don't as-of-yet-know...
A CPU is mostly transistors, so SRAM is made of the same “stuff” as the CPU logic itself.
This makes it easy to put SRAM on the same chip as a CPU, but difficult to put dense amounts of DRAM on that same chip. You could etch CPU and DRAM with the same process but both would suffer in density and therefore speed, cost and power efficiency. That’s the fundamental reason why DRAM and CPU/GPU are on different chips with a fast interconnect between them. This is so even in devices like the iPhone where one designer controls and designs the whole thing and could, in theory, fab a single design with both. But they don’t.
(There are some old exceptions to this like the PS2 shipping with a small amount of on-die DRAM. And process incompatibility is not the only reason for keeping the two chips separate. But if the processes weren’t incompatible there would certainly be high-end fast CPUs/GPUs with big gobs of DRAM.)
The problem with your rambling and speculation is that you're not making any attempt to think about what conditions would be necessary for a technology to succeed in the market and gain widespread adoption across many market segments. I stand by my assertion that it is impossible for compute-in-memory chips to entirely replace commodity DRAM, especially in any market segment where the compute portion of those memory chips is not critically valuable. Any remotely plausible, useful compute-in-memory design will have significantly lower capacity and higher cost than pure commodity DRAM; this is not a low-overhead added feature. Any market segment where the compute capabilities of compute-in-memory were so marginally valuable that a non-trivial number of the customers wanted to disable the compute is a market segment where everybody would still be buying mostly ordinary DRAM.
Read my profile: https://news.ycombinator.com/user?id=peter_d_sherman
On there, you will find the following quote:
"The true knowledge consists of knowing that one knows nothing..."
- Socrates
See, I'm agreeing with you(!), at least on that part of your message, at least on that comment!
You see, I truly don't know...
And, I don't truly know what the future as-of-yet holds...
I suggest and submit to you that this is the foundation of intellectual humility, which is the starting point of all forms of inquiry which have the potential to lead to genuine knowledge, at least in the specific areas of inquiry, and at least for specific amounts of time ("the only constant is change", aka, many forms of specific knowledge, if acquired, are time bounded, bounded in time -- but I digress!)
>"The problem with your rambling and speculation..."
My inquiry into the nature of this phenomena, but sure, your (perceived!) "rambling and speculation" on my part (I'm chuckling to myself, because that's rather funny from my perspective(!) -- yours is the first message on HN that has truly made me laugh!)
>is that you're not making any attempt to think about what conditions would be necessary for a technology to succeed in the market and gain widespread adoption across many market segments. I stand by my assertion that it is impossible for compute-in-memory chips to entirely replace commodity DRAM, especially in any market segment where the compute portion of those memory chips is not critically valuable. Any remotely plausible, useful compute-in-memory design will have significantly lower capacity and higher cost than pure commodity DRAM; this is not a low-overhead added feature. Any market segment where the compute capabilities of compute-in-memory were so marginally valuable that a non-trivial number of the customers wanted to disable the compute is a market segment where everybody would still be buying mostly ordinary DRAM.
I see...
So the premise of your argument is The Market (specifically the future consumer commodity DRAM market)... correct?
There's different kinds of RAM. CPUs are using SRAM for their caches (which usually have ~6 transistors per bit cell, IIRC, although I'm not a hardware guy, so take this stuff with a grain of salt). Your RAM sticks are using DRAM, which are 1 transistor and one capacitor per bit cell.
Now, this is where my understanding of the actual physical issues really gets sketchy, but the processes that are good at making the CPUs just aren't good at making DRAM, and vice versa. (I would guess that it has specifically to do with the capacitor, but that's purely a guess.) SRAM isn't a problem for the CPU processes, because SRAM cells are closer to the other kind of logic you set up in CPUs.
General purpose workloads would require much more silicon real estate and are not (typically) nearly as bandwidth hungry. Just like not all CPU workloads benefit from being adapted to a GPU, even fewer will benefit from being adapted to processing in RAM.
Not all algorithms can be converted of course, but a lot of that is because there are some inherently unparallelizable problems.
PIM (XBM, ZAM, etc) are all cope. This 600GB of throughput is ~4.5x higher than the 130 you get from LPDDR5X on a 128 bit bus yet their deltas are only 2.2-3x? I just don't see this being a corner worth cutting over simply paying the cost of a wider bus. Think of all the fun new rowhammer shit this will be able to do to itself
Or integrated flash with TPUs? That would be awesome. Flash your model onto a model-processor and run it.
The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.