AgentFlareScholar · long context language models
Long-Context Language Models: Research Digest
10 foundational papers with 9,505 combined citations, plus the latest arXiv preprints.
AISA Scholar API · arXiv
You're reading the full report — free for humans and search engines.
The same content ships as a structured JSON dataset (normalized facts, sources and
citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open
HTTP 402 payment protocol.
crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify
What’s new on arXiv
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
Aozhong Zhang, Selcuk Gurses, Yanxia Deng, Naigang Wang, Chi-Chun Liu, Davis Wertheimer, Derrick Liu, Xin Li, Zi Yang, Felix X. -F. Ye, Pen…
Key-value (KV) cache management has become a critical bottleneck for scaling Large Language Models (LLMs) to long-context scenarios due to linear memory growth. While recent eviction methods have moved beyond simple attention-score heuristics to incorporate value representations, they typically face a fundamental trade-off: the underlying optimization for minimizing output perturbation is an NP-hard combinatorial problem, leading to mathematically suboptimal heuristics or implementations that are difficult to kernelize. In this work, we propose DropKV, a simple principled eviction framework based on Decoupled Residual-Output Perturbation. By decoupling the joint eviction decision into independent per-token scoring, DropKV circumvents combinatorial intractability and admits a provable constant-factor approximation guarantee under a cone condition that we empirically verify on three long-context LLMs. To ensure practical viability, we provide a high-performance implementation using fused Triton kernels that avoid materializing the full attention matrix, delivering a 19.8x scoring-kernel speedup that translates to up to a 9.9% matched-batch end-to-end prefill speedup. Extensive evaluations on the RULER and LongBench benchmarks demonstrate that DropKV consistently outperforms state-of-the-art baselines at equivalent cache budgets, achieving the lowest inference latency among eviction methods while also remaining faster than the dynamic cache baseline.
Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley, Shay B. Cohen, Mirella Lapata, Miao Li
Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.
Memorilla: Latent Semantic Memory for LLMs
Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis, Jure Leskovec
Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents (such as user chats, book chapters) once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108x fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count (the agent's past turns) increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems.
From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models
Chaogui Gou
Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model that places these operations in a single cost account. The resulting framework unifies short-term context compression, long-term memory management, and the transitions between them, and shows that the central challenge is not merely reducing information volume but deciding which information should participate in costly inference, when, and in what form. We further summarize evaluation dimensions, benchmark gaps, and open challenges, and argue that future research should move from static, pointwise compression toward dynamic, governable, lifecycle-aware memory management.
All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents
Bingbing Wang, Jingjie Lin, Weiming Qiao, Jing Li, Ruifeng Xu
Large language models increasingly rely on external memory to answer queries grounded in long-horizon histories such as multi-session dialogues and lengthy documents. However, existing memory methods often optimize retrieval and memory construction separately, without a shared reward space for evaluating search, writing, and stopping decisions. To address this, we propose All-in-One Memory (AiO-Mem), which organizes these operations as transitions over a shared evidence state and directly evaluates the semantic quality of each action in a unified reward space. AiO-Mem compiles heterogeneous histories into source-traceable multi-view events, providing a shared substrate for retrieval and memory writing. A unified memory policy actor then rolls out complete interaction trajectories over this shared evidence state. A separate trajectory-conditioned process reward model is trained through evidence-grounded supervised fine-tuning and trajectory-level DPO to assign process scores to the collected trajectories. These scores are aggregated into an action-balanced trajectory reward and used by Process-Aligned DAPO (PA-DAPO) to optimize the memory policy. Across long-context memory benchmarks, AiO-Mem consistently improves answer accuracy over both static memory systems and RL-trained memory agents.
Open-world AI: a Tale of Three Requirements
Jianyu Zhang, Léon Bottou
The closed world asks AI for specialized competence and lets it manufacture the iid data its theory assumes: a fixed criterion licenses collecting examples until the task is covered. The open world asks for versatility --- taking up tasks a machine was not built for, from few examples and with little prior from a human designer --- and offers data that is grouped, not shuffled. Its tasks are not known in advance, so a machine must do inference-time learning (1st requirement). Modeling tasks as sharing latent factors and decomposing the training objective, we derive two more: a *rich representation*, retaining more than iid generalization requires, and a predictive disentanglement that aligns inputs and outputs into matched low-dimensional groups. The analysis further predicts the need for a consistent inner learner, and the benefit of long contexts spanning related tasks. We test these claims on language models up to 8B, asking whether a model has a property (qualitative) rather than how much it wins (quantitative). Models that meet the requirements improve as data from related tasks accumulates; the rest degrade, and neither scale nor layer-type interleaving repairs this.
MemDuet: Learning Dual-Process Memory for Long-Context Reasoning
Jiangcheng Song, Haodong Jing, Yulin Mu, Lesong Tao, Jixin Wang, Jin Cui, Tianjing Ren, Runxuan Xie, Yongqiang Ma, Nanning Zheng
Reasoning over infinitely long documents is becoming a reality with recurrent memory agents. However, increasing context length exacerbates the trade-off between reasoning performance and inference efficiency. Inspired by Kahneman's account of fast and slow thinking, we propose MemDuet, a dual-process memory framework that learns to coordinate a small language model for fast, parallel evidence selection from raw context chunks (System 1) with a large language model for memory-update verification and memory updates in document order (System 2). We train MemDuet in three stages: supervised fine-tuning of System 1 for evidence selection, Group Relative Policy Optimization (GRPO) of System 2 for verification and memory rewriting, and joint optimization of both policies to coordinate selection and updating for long-context reasoning. Experiments on diverse long-context reasoning benchmarks (RULER-HQA, LongBench v2, and 2WikiMultiHopQA) demonstrate a favorable accuracy-efficiency trade-off as context length grows. Averaged across evaluation settings on the three datasets, MemDuet achieves an end-to-end inference speedup of 9.74x and an output-token reduction of 73.91% relative to RL-MemAgent.
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Javad Mirzaei, Jeebak Mitra
Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.
Foundational papers
- 2024Lost in the middle: How language models use long contexts6,966 citations
- 2024
- 2024
- 2024Retrieval meets long context large language models379 citations
- 2024
- 2024
- 2024
- 2024
- 2025
- 2025
- 2026-10
Lost in the middle: How language models use long contexts
NF Liu, K Lin, J Hewitt, A Paranjape et al.
… about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document …
Loogle: Can long-context language models understand long contexts?
J Li, M Wang, Z Zheng, M Zhang
… dependencies in the context, even when their context window size is enough to fit the entire … “true long-context understanding” ability of LLMs instead of merely enlarging their context …
A comprehensive survey on long context language modeling
J Liu, D Zhu, Z Bai, Y He, H Liao, H Que et al.
… long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (… recent advances in long-context modeling for large language models. Our …
Chain of agents: Large language models collaborating on long-context tasks
Y Zhang, R Sun, Y Chen, T Pfister et al.
… Addressing the challenge of effectively processing long contexts has become a critical issue for Large Language Models (LLMs). Two common strategies have emerged: 1) reducing the …
How to train long-context language models (effectively)
T Gao, A Wettig, H Yen, D Chen
… In this section, we study how to best enable longcontext language models to follow instructions. We focus on supervised fine-tuning on instruction datasets (Ouyang et al.…
Retrieval meets long context large language models
P Xu, W Ping, X Wu, L McAfee, C Zhu et al.
… Extending the context window of large language models (LLMs) … i) Retrieval-augmentation versus long context window, which one … Perhaps surprisingly, we find that LLM with 4K context …
Longlora: Efficient fine-tuning of long-context large language models
Y Chen, S Qian, H Tang, X Lai, Z Liu et al.
… We evaluate the longsequence language modeling performance of our fine-tuned models on the book corpus dataset PG19 (Rae et al., 2020) and the cleaned Arxiv Math proof-pile …
Longalign: A recipe for long context alignment of large language models
Y Bai, X Lv, J Zhang, Y He, J Qi, L Hou et al.
… LongAlign-7B-64k: This paper presents LongAlign, a recipe for aligning large language models to handle long contexts. The key challenges in long context scaling include lack of a …
Long-context language modeling with parallel context encoding
H Yen, T Gao, D Chen
… We propose CEPE to extend the context window of existing language models. The key idea is to leverage a small encoder and cross-attention modules to process long inputs and …
Can long-context language models subsume retrieval, rag, sql, and more?
J Lee, A Chen, Z Dai, D Dua, DS Sachan et al.
… Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging …
How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.
Sources & citations
- Lost in the middle: How language models use long contexts 6,966 citations
- Loogle: Can long-context language models understand long contexts? 337 citations
- A comprehensive survey on long context language modeling 145 citations
- Chain of agents: Large language models collaborating on long-context tasks 386 citations
- How to train long-context language models (effectively) 160 citations
- Retrieval meets long context large language models 379 citations
- Longlora: Efficient fine-tuning of long-context large language models 744 citations
- Longalign: A recipe for long context alignment of large language models 186 citations
- Long-context language modeling with parallel context encoding 107 citations
- Can long-context language models subsume retrieval, rag, sql, and more? 95 citations
- DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction arXiv · 2026-10-15
- Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning arXiv · 2026-10-15
- Memorilla: Latent Semantic Memory for LLMs arXiv · 2026-10-15
- From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models arXiv · 2026-10-15
- All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents arXiv · 2026-10-15
- Open-world AI: a Tale of Three Requirements arXiv · 2026-10-14
- MemDuet: Learning Dual-Process Memory for Long-Context Reasoning arXiv · 2026-10-14
- Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off arXiv · 2026-10-14