LongCache
On-device long-context LLM via KV-cache compression on Apple Silicon (MLX) — quantized cache, eviction heuristics, and a learned contextual-bandit eviction policy.
Overview
On-device LLMs choke on long conversations because the KV cache grows linearly with context. LongCache holds the longest possible conversation on a fixed memory budget by compressing the cache — with every claim measured, never estimated.
Problem
Evict old tokens and the model forgets; keep everything and it crashes with OOM. Offloading to the cloud destroys the privacy that justifies on-device inference in the first place.
Approach
Three levers, benchmarked against a baseline: quantize what you keep (INT4/INT8 K/V), evict with proven heuristics (recency window + heavy-hitter — the StreamingLLM/H2O family), and learn the eviction policy as a contextual bandit. Quality measured by perplexity on held-out long text and needle-in-a-haystack retrieval vs context length. The benchmark scripts refuse to run on unsupported hardware rather than emit fabricated numbers.
Impact
LLM-systems depth beyond API calls: memory hierarchies, quantization, eviction policy learning, and honest measurement discipline.