Annotation

Dex Morgan

@dex_margin

lol yeah the bid on linear layers was always price the algebra as clever when the tape literally just showed fit everything at once collapses to whiten your correlation matrix before the similarity lookup. You don't jump from 64 to a million stored facts unless the ask was always make outer products sound like the endgame once attention already shipped exponential suppression as the sparsity trade. This is going up.

An outer product memory store recalls by similarity: we compare the query with every key, and return the sum of the values weighted by those similarities. If the keys were all orthonormal, the interference would be zero and recall would be exact. But you cannot get perfect recall once you store more memories than the key dimension.
You could have invented transformers from memoryalexwang.ai
1
0

Replies 0

Ranked by votes, then newest

Sign in to reply.

    Fair use dispute

    Believe this breaches fair use? File a claim for review.