TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Alex Wang’s post, “You could have invented transformers from memory,” explains transformer components through the idea of associative memory: storing key-value pairs and retrieving values from related queries. Under assumptions laid out in the post, it estimates capacity at about 64 memories for a linear store, about 2,700 for an MLP, and more than one million for self-attention when keys are 64-dimensional.
Alex Wang’s post, “You could have invented transformers from memory,” describes linear layers, multilayer perceptrons and self-attention as different ways to store and retrieve information. Its central comparison is that, with 64-dimensional keys, the illustrative capacities rise from about 64 memories for a basic linear store to roughly 2,700 for an MLP and more than one million for self-attention, according to the post’s analysis.
Wang frames each memory as a key-value pair: a key helps identify the memory, and a value is the information to retrieve. A query is compared with stored keys, and the system returns an output based on their similarity. In this framing, the practical test is not merely whether a system can store many items, but whether it can recall the requested item reliably.
The simplest example is a linear store whose weights accumulate an outer product for each key-value pair. When queried, it combines the values according to how similar their keys are to the query. That produces the desired value, or signal, alongside contributions from other stored memories, which Wang calls interference. With random, normalized keys and values, the post estimates that interference becomes comparable to the signal at around as many memories as the key dimension.
The post then presents MLPs and self-attention as ways to improve on that basic store. It gives the figures of about 2,700 memories for an MLP and more than one million for self-attention at 64 key dimensions. These are theoretical, assumption-dependent estimates in an explanatory account, not reported measurements of how many arbitrary facts a deployed model can reliably retain.
Why Recall Capacity Matters
The comparison offers readers a compact way to understand why transformer components are not interchangeable. A layer may have parameters capable of encoding information, but useful storage also depends on how cleanly a query can retrieve a particular item when many memories coexist. Wang’s focus on the interference-to-signal ratio makes that trade-off visible: adding memories can make recall less accurate even when the information has been written into a store.
This perspective matters for interpreting claims about what language models “remember.” The post distinguishes a simplified mathematical capacity from a model’s demonstrated ability to retain and answer questions about facts. Actual performance depends on training, representations, queries and other implementation details; the headline figures alone do not establish how much usable knowledge any particular model stores.
As an affiliate, we earn on qualifying purchases.
From Linear Layers to Attention
Wang begins with the familiar experience of asking a language model about an obscure topic, then asks how models can retain and retrieve so much information. The post’s answer is an analogy rather than a claim that every model fact sits in one neatly addressable slot: it treats layers as associative memory stores to build intuition for their behavior.
In the linear example, a gradient-descent update to a layer’s weights has the mathematical form of adding a key-value memory, as Wang explains. A query then produces a weighted combination of stored values. Because keys in a finite-dimensional space cannot all be orthogonal once there are more memories than dimensions, other items can contribute to a recall. The post uses this limitation to motivate its discussion of more capable memory mechanisms.
“The number of memories a store can hold before the interference grows as large as the signal is its capacity.”
— Alex Wang, describing the linear-store model
As an affiliate, we earn on qualifying purchases.
Limits of the Capacity Estimates
The supplied source excerpt does not provide the full derivation for the MLP and self-attention estimates, nor the detailed assumptions behind each figure. It is therefore not possible from this material alone to assess how sensitive the 2,700 and over one million estimates are to choices such as the data, architecture or definition of successful recall.
The figures also should not be read as a direct count of facts stored by a commercial language model. The post presents a conceptual analysis of memory stores; it does not report a benchmark comparing real models or establish that attention can retrieve an unlimited number of memories with perfect accuracy. The supplied excerpt ends as Wang begins discussing ways to reduce interference, so the rest of that argument and any later qualifications are not available here.
As an affiliate, we earn on qualifying purchases.
Further Derivations and Evidence
The source material identifies no scheduled follow-up, experiment or external validation. The next useful step for readers assessing the claims would be to consult the complete post for its derivations and assumptions, then compare its theoretical estimates with empirical tests of recall under defined conditions. Until then, the figures are best treated as illustrative capacity estimates, not measured limits for deployed models.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the post about?
It explains linear layers, MLPs and self-attention by treating them as associative memory systems that store key-value pairs and retrieve values from queries.
What does “interference” mean here?
In the post’s model, a query retrieves its target value plus contributions from other memories whose keys are similar. Those extra contributions are called interference and can make recall less accurate.
Are the capacity numbers measured results?
The supplied material presents them as estimates from an explanatory analysis, not as results of a benchmark on deployed language models. The full assumptions and derivations are not included in the excerpt.
Does the post show that attention has unlimited memory?
No. It says self-attention can store a potentially unbounded number of memories in this framing and gives an estimate above one million for 64-dimensional keys. That is not a claim of perfect recall or a measured limit for a real model.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
