A discussion exploring alternatives to gradient-based training using lossless compression schemes, locality-sensitive hashing, and algebraic structures (lattices, type systems, category theory, elliptic curves) that could enable true incremental learning without the "holographic effect."
Current LLM training suffers from a fundamental mismatch:
Each new parameter affects almost the entire graph, effects are discrete and thus imprecise, and there's a limit to precision based on memory bandwidth. This raises the question: if language is inherently discrete, why use approximation-based methods?
| Type | Examples | Characteristics |
|---|---|---|
| Lossy | DCT (JPEG, MP3), wavelets (Dirac) | Approximation, quantization |
| Lossless | Huffman, arithmetic coding, LZ77/78, RLE, FLAC, PNG | Exact reconstruction, algebraic |
For discrete data like human language or computer code, approximation may be the wrong approach if we want to minimize computation, memory copying, and rounding errors.
The most principled connection to compression. Solomonoff induction states: the best model is the shortest program generating the data. This IS deterministic. Work by Rissanen and Grünwald on MDL formalized this. The limitation: computing Kolmogorov complexity is uncomputable in general, so we approximate.
Classical Hopfield networks store patterns via outer products—algebraically exact for capacity up to ~0.14N patterns. Recent work (Ramsauer et al., 2020) connects modern Hopfield networks to transformer attention. These have more algebraic flavor—energy minimization with discrete attractors.
Theoretical work (Merrill, Sabharwal, and others) analyzes what formal languages transformers can represent. If a transformer IS a finite automaton, there might be ways to construct/train it more directly. Related: whether transformers can be represented as tensor networks with exact decompositions.
Binary neural networks (Courbariaux et al.), ternary quantization—keeping weights in {-1, 0, +1}. These ARE discrete, but typically still trained via gradient descent with straight-through estimators. Not truly algebraic training.
If language models are "compressed programs," why not search program space directly? Work on neural program induction, differentiable interpreters. The DREAMCODER system (Ellis et al.) learns compositional libraries. More algebraic, but doesn't scale to LLM-level complexity yet.
A well-trained language model IS optimal compression via arithmetic coding. If a model assigns probability p(xi | x{<i}), then arithmetic coding achieves entropy rate. Recent work ("Language Models are Compressors," Delétang et al.) formalizes this.
The implication: training a better model = learning better compression. But we still learn the compressor via gradient descent rather than constructing it algebraically.
CP decomposition, Tucker decomposition, tensor-train. These give algebraic factorizations of weight matrices. Mostly used post-training for compression, but some work on training directly in decomposed form.
Rather than retraining everything, store differences:
For lossless schemes like FLAC/PNG, the key insight is: exploit predictable structure to encode residuals cheaply. In language:
These are deterministic transformations. An analogous approach for transformers might:
The core issue: generalization. Deterministic/lossless schemes excel at reproducing seen patterns. Language requires:
These seem to require the "soft" interpolation that continuous representations provide. But perhaps there are algebraic structures (lattices, type systems, category theory) that could capture semantic relationships discretely.
In cryptographic hashes, collision = failure. In locality-sensitive hashing (LSH):
Nilsimsa specifically:
| Hash Type | Similarity Preserved | Properties |
|---|---|---|
| Nilsimsa | Edit distance (text) | 256-bit, trigram-based |
| SimHash | Cosine similarity | Bit-sampling of projections |
| MinHash | Jaccard (set overlap) | Permutation-based |
| GeoHash | Spatial proximity | Hierarchical, prefix-comparable |
| TLSH | Byte distribution | Locality-sensitive, diff-friendly |
Transformer attention as "find similar keys for this query" can be compared to LSH:
Neural attention: softmax(QK^T / √d) → learned similarity
LSH attention: hash(q) collides with hash(k) → designed similarity
The Reformer paper (Kitaev et al., 2020) exploited exactly this—using LSH to reduce attention from O(n²) to O(n log n). But they still learned the embeddings; LSH just accelerated the lookup.
Replace learned embeddings entirely with deterministic similarity hashes:
This resembles older approaches:
If collisions encode similarity, then:
This is essentially product quantization or vector quantization, but deterministic rather than learned.
For retrieval/classification, hash-based approaches work. "Gzip is All You Need" showed compression-distance classifiers are surprisingly competitive.
For generation:
The challenge: LSH captures surface similarity (n-grams, character patterns). Semantic similarity ("happy" ≈ "joyful") requires either:
1. Hash-addressed memory + minimal learned integration
context → LSH hash → retrieve candidates → small learned model → output
RETRO, REALM, RAG do variants of this, but with learned retrievers.
2. Hierarchical hashing
3. Collision-based training signal Instead of gradient descent, update based on whether the right things collide:
To test this approach:
The "blend" step is where something learned might still be needed, or might admit an algebraic solution.
In current neural networks, representations are distributed:
This is analogous to a hologram: every piece contains information about the whole, and you cannot modify one region without affecting the global image.
The practical consequences are immediately visible when using LLMs:
~/.ssh/id_ed25519 goes with git.mleku.devThis is not a minor inconvenience—it's a fundamental architectural limitation. A useful assistant needs:
We want a representation where:
update(knowledge_base, new_fact)
= knowledge_base ∪ {new_fact} # Set union, not gradient descent
Not:
update(knowledge_base, new_fact)
= retrain(knowledge_base, corpus + new_fact) # Global perturbation
A concept lattice is a mathematical structure from Formal Concept Analysis (Ganter & Wille) that represents objects, attributes, and their relationships:
Example lattice for git configuration:
⊤ (all repos)
/ \
(uses GitHub) (uses gitea)
| |
(public repos) (mleku's repos)
| |
plebeian-market git.mleku.dev/mleku/*
| |
⊥ (specific repo instances)
| Operation | Meaning | Example |
|---|---|---|
| Meet (∧) | Most specific common generalization | repo ∧ uses_gitea = mleku's gitea repos |
| Join (∨) | Most general common specialization | GitHub ∨ gitea = all hosted repos |
| Complement | What doesn't have attribute | repos without SSH key configured |
| Implication | If A then B | if mleku's repo → use id_ed25519 |
The challenge: language is not just object-attribute relations. It requires:
But lattices can be extended:
A deep connection between logic and computation:
| Logic | Type Theory | Programming |
|---|---|---|
| Proposition | Type | Specification |
| Proof | Term | Implementation |
| Implication (A → B) | Function type | Function |
| Conjunction (A ∧ B) | Product type | Struct/tuple |
| Disjunction (A ∨ B) | Sum type | Enum/union |
If we represent knowledge as types, then:
ssh_key : Key; git_remote : Remotekey_for : Remote → KeyDependent types allow types to depend on values:
-- A repository parameterized by its host
Repository : Host → Type
-- An SSH key valid for a specific host
SSHKey : (h : Host) → Repository h → Type
-- The relationship is encoded in the type
git_mleku_dev : Repository Gitea
id_ed25519 : SSHKey Gitea git_mleku_dev
This makes invalid states unrepresentable. You cannot use the wrong key for the wrong remote—it's a type error.
Montague semantics and its successors treat natural language as a typed lambda calculus:
e → t (entity to truth value)e → e → t (two entities to truth value)(e → t) → (e → t) → t (two predicates to truth value)Composition is function application. "Every cat sleeps" is:
every : (e → t) → (e → t) → t
cat : e → t
sleeps : e → t
every(cat)(sleeps) : t
This is fully algebraic—no gradients, no approximation.
Research connecting types to neural models:
Category theory is the "mathematics of composition." It provides:
For our purposes: category theory can describe how semantic structures compose without reference to gradients or continuous optimization.
A functor F : C → D maps:
Application: A functor from "syntax" to "semantics" that preserves compositional structure.
A natural transformation α : F ⇒ G relates two functors:
Application: Systematic translation between representations (e.g., surface form to meaning).
A monad packages:
Application: Context-dependent semantics, where T represents "in context C."
The DisCoCat framework (Coecke, Sadrzadeh, Clark) combines:
The key insight: grammatical types form a pregroup (a kind of category), and meaning composition follows the categorical structure.
Sentence: "dogs chase cats"
Grammatical types:
dogs : n (noun)
chase : n^r · s · n^l (transitive verb: consumes noun on right and left)
cats : n (noun)
Composition (pregroup reduction):
n · (n^r · s · n^l) · n → s (sentence type)
Semantic composition:
meaning(dogs) ⊗ meaning(chase) ⊗ meaning(cats)
→ contracted via grammatical structure
→ meaning(sentence)
Elliptic curves (EC) provide a rich algebraic structure:
The hypothesis: EC groups could serve as an algebraic substrate for semantic representations, replacing continuous vector spaces.
An elliptic curve over a finite field F_p:
E: y² = x³ + ax + b (mod p)
Points on this curve, plus a "point at infinity" O, form an abelian group:
All operations are exact—no floating point, no rounding.
Embedding tokens/concepts as EC points:
Example structure:
Let E be a curve with generator G of order n.
Token embedding:
embed(token) = H(token) * G # Hash to scalar, multiply generator
Relationship encoding:
related(A, B) iff embed(A) + embed(B) = embed(C) for some meaningful C
Composition:
compose(context, query) = pairing(embed(context), embed(query))
A pairing is a bilinear map e : G₁ × G₂ → G_T where:
This is structurally similar to attention:
Attention: softmax(Q · K^T) · V # Bilinear in Q, K
EC Pairing: e(Q, K) → scalar → select V
But the EC version is:
┌─────────────────────────────────────────────────────┐
│ EC-Based Language Model │
├─────────────────────────────────────────────────────┤
│ │
│ Input: token sequence [t₁, t₂, ..., tₙ] │
│ │
│ 1. Embed: Pᵢ = H(tᵢ) * G (hash-to-curve) │
│ │
│ 2. Context: C = Σᵢ wᵢ * Pᵢ (weighted sum in EC) │
│ or C = P₁ + P₂ + ... + Pₙ │
│ │
│ 3. Attend: For each position j, │
│ aⱼ = e(C, Pⱼ) (pairing) │
│ → discrete "attention scores" │
│ │
│ 4. Retrieve: Use aⱼ to index into stored │
│ associations (hash table lookup) │
│ │
│ 5. Output: Retrieved token or composition │
│ │
└─────────────────────────────────────────────────────┘
EC points can be related to:
This suggests a unified algebraic framework where EC provides the computational substrate, lattices provide knowledge organization, and types/categories provide compositional structure.
The holographic effect in neural networks stems from:
To eliminate this, we need:
Combining the algebraic structures discussed:
┌─────────────────────────────────────────────────────────────────┐
│ ALGEBRAIC COMPOSITIONAL MEMORY (ACM) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ LAYER 1: ELLIPTIC CURVE SUBSTRATE │
│ ───────────────────────────────────────────────────────────── │
│ • Tokens → EC points via hash-to-curve │
│ • Exact, discrete, parallelizable operations │
│ • Pairings for bilinear composition │
│ │
│ LAYER 2: LATTICE-STRUCTURED KNOWLEDGE │
│ ───────────────────────────────────────────────────────────── │
│ • Concept lattice organizing entities and attributes │
│ • Incremental updates via lattice insertion │
│ • Inheritance and implication automatic │
│ │
│ LAYER 3: TYPE-THEORETIC COMPOSITION │
│ ───────────────────────────────────────────────────────────── │
│ • Grammatical structure as types │
│ • Semantic composition as type-directed evaluation │
│ • Invalid combinations rejected at type level │
│ │
│ LAYER 4: CATEGORICAL INTEGRATION │
│ ───────────────────────────────────────────────────────────── │
│ • Functors between layers preserve structure │
│ • Natural transformations for systematic translation │
│ • Monadic context handling │
│ │
│ MEMORY INTERFACE │
│ ───────────────────────────────────────────────────────────── │
│ • LSH for fast approximate retrieval │
│ • Exact lookup for known associations │
│ • Incremental insertion without retraining │
│ │
└─────────────────────────────────────────────────────────────────┘
| Property | Current LLMs | Proposed ACM |
|---|---|---|
| Update locality | Global (all params) | Local (affected region only) |
| Precision | Float16/32, lossy | Exact (finite field/integer) |
| Memory model | Implicit in weights | Explicit, addressable |
| Composition | Learned (attention) | Algebraic (type-directed) |
| Generalization | Statistical interpolation | Structural composition |
| Interpretability | Opaque | Transparent (lattice/type structure) |
Current LLMs fail at persistent memory:
Failure mode: "Use id_ed25519 for git.mleku.dev"
→ scrolls out of context
→ model forgets
→ repeatedly uses wrong key
ACM solution:
1. Parse: SSH key assignment statement
2. Type: key_for(git.mleku.dev) : SSHKey
3. Lattice: Insert (git.mleku.dev, uses_key, id_ed25519)
4. Store: EC point for this fact in hash-addressed memory
5. Retrieve: On any git operation, query lattice for key association
6. Persist: Fact remains until explicitly removed
This is not context-dependent—it's a structural modification to the knowledge base.
function learn(fact):
# 1. Parse to typed representation
typed_fact = parse(fact) : FactType
# 2. Compute EC embedding
point = embed_to_curve(typed_fact)
# 3. Insert into lattice
lattice.insert(typed_fact.objects, typed_fact.attributes)
# 4. Store in hash-addressed memory
memory[hash(point)] = typed_fact
# No gradient descent
# No global parameter update
# No risk of forgetting unrelated facts
- Possibility: Bootstrap from small neural model, then freeze - Possibility: Learn from multimodal alignment (vision, action)
- Possibility: Maintain multiple lattice branches - Possibility: Type-theoretic disambiguation
- Possibility: Fuzzy lattice membership - Possibility: LSH buckets for approximate retrieval
- Possibility: Hierarchical lattice structure - Possibility: EC curves with large group order
- Possibility: Type-directed synthesis - Possibility: Compositional template filling
Goal: Establish theoretical framework and minimal viable prototype
- Define hash-to-curve function with semantic locality property - Prove properties of composition via group operations - Implement efficient EC arithmetic (use existing crypto libraries)
- Implement Formal Concept Analysis algorithms - Design incremental update procedures - Connect to simple NL understanding tasks
- Implement pregroup or CCG parser - Connect grammatical types to semantic operations - Test on compositional generalization benchmarks
Goal: Combine layers into working system
- Map lattice concepts to EC points - Use pairings for lattice queries - Benchmark retrieval speed and accuracy
- Grammatical structure guides EC operations - Composition produces EC points for phrases/sentences - Test on semantic similarity tasks
- LSH-based indexing of EC points - Exact retrieval for known associations - Incremental insertion benchmarks
Goal: Compare to neural baselines
- Design benchmarks requiring persistent memory - Compare ACM vs. RAG vs. fine-tuning vs. in-context learning - Measure memory capacity, retrieval accuracy, update cost
- SCAN, COGS, CFQ benchmarks - Compare structural composition vs. neural interpolation - Analyze failure modes
- Memory footprint per fact - Compute per update vs. gradient step - Inference latency comparison
Goal: Demonstrate viability at useful scale
- Multi-level lattices for large knowledge bases - Distributed EC computation - Caching and indexing strategies
- Ingest structured knowledge bases (Wikidata, etc.) - Handle natural language variation - Connect to retrieval systems
- ACM for long-term memory + small neural model for generation - Determine minimal neural component needed - Compare hybrid vs. pure approaches
The holographic effect in neural networks—where every parameter encodes everything and updates propagate globally—is not a law of nature. It's an artifact of the specific architecture (distributed representations) and training method (gradient descent) we've chosen.
Algebraic alternatives exist:
The synthesis of these approaches could yield machine intelligence that is:
This is not merely an optimization—it's a different paradigm. Rather than learning a compressed statistical summary of training data, we would construct an algebraic structure that represents knowledge explicitly and composes it systematically.
The path forward requires collaboration across:
The prize: machine intelligence that actually remembers what you told it.