Gravitational Cache Dynamics

Gravitational Cache Dynamics

Testing CDR theory predictions via Hopfield-inspired particle simulation

Headline result: a token converges into one cluster, detaches when its signal source changes, and reconverges into another — resolving polysemy through physics alone.

Abstract. We model cache entries as particles in high-dimensional space, driven by co-occurrence signals with recency-weighted partner memory. Five experiments test whether compression (cluster formation), temporal partitioning (timescale separation), and renewal (optimal decay rate) emerge from the dynamics without being explicitly programmed. All three core CDR theorems find empirical support. The headline result: a polysemous token (“bank”) converges into a river-sense cluster, detaches when its signal source changes, and reconverges into a finance-sense cluster.

The question

Transformer attention is mathematically equivalent to one step of a modern Hopfield network. But transformers strip out a key Hopfield property: in classical Hopfield networks, retrieval mutates state. Reading a memory changes the network. In transformers, the KV cache is append-only and never consolidates.

CDR theory argues that long-lived memory systems face three structural pressures:

1. They must compress — bounded capacity forces lossy representation
2. They must partition by timescale — fast and slow signals need different handling
3. They must renew — stale representations degrade and eventually need rebuilding

Can these properties emerge from simple physics rather than explicit design?

The model

No transformer is involved. Each cache entry is a particle in 8-dimensional space with position, velocity, and mass. Co-occurrence data drives the dynamics directly.

The key mechanism is partner memory with a sliding window. Each particle maintains a record of its recent co-occurrence partners. On each tick:

1. Record signals: new co-occurrence pairs are added to each particle’s partner memory. Mass is boosted for particles that appear in a signal.
2. Trim memory: associations older than the sliding window are dropped. This is what makes old relationships fade — the force that held a particle near its former partners literally disappears.
3. Compute forces: for each particle, compute gravitational attraction toward ALL remembered partners, weighted by recency (newer = stronger). Force = G · strength · m_a · m_b / (r² + ε). Both particles move toward each other proportionally to their relative mass — heavier particles move less.
4. Integrate: velocity updates with damping, then position updates.
5. Decay: mass decreases proportionally each step (mass -= decay_rate × mass). This creates a natural carrying capacity — particles stabilize at a mass proportional to their signal rate rather than growing without bound. Unused particles shrink and eventually die.
6. Merge: particles closer than a threshold coalesce into a mass-weighted centroid.

The sliding window is what distinguishes this from a standard N-body simulation. Gravity only acts between particles with recent co-occurrence history — not between all pairs. When a token stops co-occurring with a partner, that partner falls out of the window, the attractive force disappears, and the token is free to drift toward new partners.

Results

Five experiments, each probing a different CDR prediction. All three core theorems found empirical support.

Concept drift: “bank” changes meaning

Setup: 7 tokens — bank, river, water, flow, money, account, deposit. Both clusters maintain strong internal cohesion throughout. Bank’s co-occurrence signal is with the nature cluster for steps 0–1000, then switches sharply to the finance cluster for steps 1050–5000. Each particle maintains a sliding window of its recent co-occurrence partners — old associations fade as they leave the window.

What happened:

Bank converged firmly into the nature cluster during the river phase (distance ratio 1.34 → 0.00 by step 600). After the signal switched at step 1000, the old river-bank memories gradually fell out of the sliding window. Bank began to detach from nature and drift toward finance. By step 1500, bank was already closer to finance than nature. By step 2000, bank had fully converged into the finance cluster (bank→finance = 0.00). Mass stabilized at ~20 for active tokens and ~13 for background-only tokens — proportional decay creates a natural carrying capacity. Both clusters stayed intact throughout — no merges, no deaths, all 7 particles alive.

Bank (gray) converges into the nature cluster, detaches when signal changes, and converges into the finance cluster. Both clusters remain stable throughout.

The system resolved polysemy without any semantic knowledge — purely through signal-driven gravitational dynamics with memory decay. The key mechanism: each particle remembers its recent partners in a sliding window. When the signal source changes, old associations fade and new ones take over, causing the particle to drift to its new attractor.

Attractor formation: clusters from noise

Setup: 8 tokens (cat, purr, dog, bark, fish, swim, animal, pet) at random initial positions. Consistent co-occurrence pairs: cat–purr, dog–bark, fish–swim. Weak cross-cluster signals from animal and pet.

What happened: Semantic clusters formed rapidly — paired tokens converged and merged within the first 500 steps. All five merges completed by step 500, leaving 3 particles (one per semantic group). Mass stabilized around 20 for each — proportional decay prevents unbounded growth. Cluster count dropped from 8 to 3, then to 1 as the surviving heavy attractors continued to drift closer.

Eight tokens converge from random positions into three semantic clusters. Circle size indicates mass.

Cluster count shows clear stepwise convergence. Total mass grows linearly from continuous signal.

Temporal partitioning: CDR Theorem 2

Setup: 11 tokens with two signal regimes — background pairs always active, bursty pairs alternating in 100-step ON/OFF phases. Run for 2000 steps.

What happened: Background tokens (weather, animal) accumulated mass monotonically. Bursty tokens (cat, fish) showed clear sawtooth oscillation — gaining mass during ON phases, losing during OFF. Cat and fish oscillated in antiphase. The system spontaneously partitioned into stable and transient attractors based on signal frequency, with no explicit timescale configuration.

Background tokens (purple, orange) grow monotonically. Bursty tokens (red, green) show sawtooth mass oscillation in antiphase.

Capacity and interference

Setup: 50 tokens across 10 semantic groups. Within-group co-occurrence always present. Cross-group noise density increases linearly from 0 to 1 over 2000 steps.

What happened: Cluster count collapsed from 50 to 1 over 2000 steps as cross-group noise overwhelmed within-group cohesion. 49 merges occurred. With proportional mass decay bounding particle mass, no token becomes heavy enough to resist merging indefinitely — the system shows complete collapse under sufficient noise, demonstrating a clear capacity limit.

Progressive cluster collapse as cross-group noise density increases. The system shows graceful degradation rather than a sharp phase transition.

Renewal: CDR Theorem 3

Setup: 20 tokens — 10 domain A, 10 domain B. Signal switches from domain A (steps 0–300) to domain B (steps 300–1200). Decay rate swept across 6 values.

What happened: A clear tradeoff exists. At low decay (0.001), domain A stays tight but domain B can’t consolidate — old attractors interfere. At high decay (0.020), domain B dies entirely. The sweet spot is 0.008–0.010 where B forms reasonable structure while A fades. This is the empirical τ* — the optimal renewal rate predicted by CDR Theorem 3.

Decay Rate	A Clusters	B Clusters	Interpretation
0.001	3	8	A blocks B formation
0.005	3	5	Some interference
0.008	3	4	Sweet spot
0.010	3	4	Sweet spot
0.015	2	3	A fading too fast
0.020	2	0	B dies

Cluster count and total mass over 1200 steps. The kink in mass at step 300 marks the domain switch.

What this means for CDR

A particle system with recency-weighted partner memory — no semantic knowledge, no training, no explicit clustering — produces memory structure that aligns with CDR theory:

• Compress manifests as merging: nearby particles collapse, reducing redundancy
• Divide manifests as cluster formation and mass differentiation: different timescales naturally separate into heavy stable and light transient attractors
• Renew manifests through two mechanisms: mass decay (particles lose weight without signal) and memory decay (old partner associations fall out of the sliding window, releasing particles from stale attractors). The combination produces the converge→detach→reconverge behavior seen in the bank experiment.

The optimal mass decay rate is the empirical τ* predicted by Theorem 3. The memory window length controls how quickly old associations fade — a second temporal parameter that interacts with τ*.

Limitations

• No transformer in the loop. Uses raw co-occurrence, not actual attention weights. The signal is cleaner than a real model would produce.
• Parameter sensitivity. Different token counts and densities need different physics constants.
• Scale. Largest experiment used 50 particles. Real KV caches have 2K–128K entries.
• Merging disabled for bank experiment. The bank experiment required disabling merges to prevent bank from being permanently absorbed. A more sophisticated merge criterion (e.g., only merge particles with sustained co-occurrence) would be more robust.

Next steps

1. Replace synthetic co-occurrence with actual attention weights from a small transformer
2. Evaluate whether a gravitationally-managed KV cache improves downstream task performance
3. Add Lennard-Jones repulsive force at short range for stable orbits
4. Scale test with 1K+ particles using approximate nearest neighbor
5. Explore LoRA overlay variant — gravitational dynamics as continuous updates to low-rank attention adapters

Code

The simulation is open source: gravitational-cache (Python, ~600 lines).