You are very much ON TIME
Record a decent little poem that I saw~
Reading Notes of Selective Context: Compressing Context to Enhance Inference Efficiency of Large Language Models
TL;DR: using perplexity on a small LLM as the heuristic. The assumption is that context tokens with lower surprisal can be deleted. Not really optimize the policy model to minimize the cost.
Reading Notes of LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
TL;DR: using perplexity on a small LLM as the heuristic. The assumption is that context tokens with lower surprisal can be deleted. Not really optimize the policy model to minimize the cost. The diffe
Reading Notes of JLens: Verbalizable Representations Form a Global Workspace in Language Models
TL;DR: An interpretability method that unembeds middle hidden states transformed with a Jacobian matrix. I don’t know why it works in math.
Reading Notes of Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
TL;DR: random KV eviction works better than heuristics ones.
Reading Notes of ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution
Something similar with the KVP (and arxived at similar time). Use SFT to train on “oracle” tokens related to future attention. Use RL/GRPO to penalize loss spikes on low-entropy (tokens that should be
Reading Notes of RLKV: Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
TL;DR: Some of heads should not do KV cache eviction. This paper predicts which heads to do KV cache eviction and which heads to do full KV cache. MDP process trained with RL/GRPO based on task succes
Reading Notes of DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference
Last layer hidden state already has information related to uncertainty.
