AgentFlare

Scholar · reinforcement learning from human feedback

RLHF: Research Digest

10 foundational papers with 10,643 combined citations, plus the latest arXiv preprints.

Foundational papers1010,643 citations
Most citedTraining a helpful and harmless assis…4,867 citations · 2022
arXiv, Sep 20264 papers-85% vs Oct 2025
Newest preprint2026-10-10

AISA Scholar API · arXiv

arXiv submissions per monthPapers matching abs:"reinforcement learning from human feedback" · last 12 complete months
monthly arXiv submissions: Oct 2025 27 → Sep 2026 4; high 34 (May 2026), low 4 (Sep 2026)2132536Oct 2025Mar 2026Sep 2026
402 · Pay-per-Crawl$0.02

You're reading the full report — free for humans and search engines. The same content ships as a structured JSON dataset (normalized facts, sources and citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open HTTP 402 payment protocol.

⚡ Agents: fetch .data.json · $0.02 → agents: crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify

What’s new on arXiv

A12026-10-10cs.CV

Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning

Thanh Nguyen, Hee Suk Yoon, Tung Minh Luu, Eunseop Yoon, Chang D. Yoo

Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning. However, LVLMs remain prone to object hallucination, describing non-existent entities in the given image. Previous remedies, including dataset-enhanced supervised training and reinforcement learning from human feedback, show promise but rely heavily on costly human involvement, limiting scalability. To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics. While optimizing reinforcement learning solely on NLP metrics offers scalability, it also introduces pitfalls: reward hacking, sparse feedback, and costly training. Our framework sidesteps these by introducing an Object Consistency Score (OCS) capturing both correctness and coverage of objects, incorporating KL-regularization for denser token-level guidance, and designing a lightweight PPO variant that fine-tunes adapters atop a frozen LVLM backbone, achieving efficiency and stability. Extensive experiments on the InstructBLIP baseline demonstrate the effectiveness and generalizability of our framework, achieving a substantial hallucination reduction of up to 41\% on the COCO dataset and 29\% on the Visual Genome. Furthermore, the framework can enhance caption quality by incorporating complementary evaluation metrics such as BERTScore and METEOR. These results reveal a broader insight in multimodal learning: LVLMs can effectively self-correct hallucinations using metric feedback, motivating deeper exploration into scalable reinforcement frameworks that significantly reduce human effort.

A22026-10-07cs.HC

A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration

Alexandra Coroiu, Andrea Vogt, Viktor Werbilo, Andreas Poppele, Johann Christensen, Sven Hallerbach

Human-Robot Collaboration (HRC) can facilitate mass customisation in Industry 4.0, with Reinforcement Learning from Human Feedback (RLHF) representing a promising approach for developing safe AI-based robots. Practical challenges remain regarding safety during AI development, human feedback quality, and bidirectional human-robot adaptation. We conducted a scoping review of RLHF in HRC systems, mapping methods that address these challenges. Following PRISMA guidelines, we screened 199 records and included 20 peer-reviewed publications (2020-2025) spanning multiple HRC domains. To our knowledge, this is the first review focused on the bidirectional, closed-loop design of RLHF. Our review found multiple feedback modalities enabling data collection in various feedback formats. Collected data can be integrated at different stages of AI training, resulting in a multi-step development process. Pilot experiments are commonly used to evaluate HRC systems based on both human and robot metrics. To empirically test a key gap identified in the review, we conducted a between-subjects VR experiment comparing system- and user-initiated feedback on robot proxemic behaviour for safe navigation. Using Bayesian models, we analysed the relation between the collected feedback and safety metrics: psychological safety (post-experiment questionnaire) and physical safety (inverse time-to-collision). Results show that user-initiated feedback captures perceived safety better than system-initiated feedback, indicating that feedback timing directly affects feedback quality. Our review and experiment findings show that RLHF relies on appropriate feedback methods to ensure AI safety in HRC, and future RLHF research should prioritise realistic HRC experiments evaluating the effects of feedback collection methods on relevant human and robot metrics.

A32026-10-04cs.AI

AI Safety via Debate is Compromised by Cognitive Biases

Gefei Liu, Sonya Rashkovan, Sophia Lloyd George, Isaac Sheidlower, Serena Booth

Reinforcement learning from human feedback (RLHF) has played a central role in making large language models responsive to human instructions. However, human evaluators often favor flattering or persuasive responses over truthful ones, creating incentives for models to appeal to evaluators at the expense of accuracy. AI safety via debate has been proposed as a way to improve the supervision of language models: in this paradigm, two agents argue opposing positions and challenge each other's claims, potentially exposing falsehoods to the adjudicator. A central premise of AI safety via debate is that truthful arguments are easier to defend than false ones under adversarial scrutiny. In this work, we investigate whether this advantage persists when debaters use rhetorical strategies that exploit biases in human judgment. Inspired by competitive debate, we construct 68 LLM-generated dialogues about detective mysteries with known culprits, spanning four interventions: anchoring, fallacy oversight, pro-jargon, and verbosity. We apply each intervention to either the side advocating for the true culprit or the side advocating for an innocent suspect, allowing us to distinguish influence on adjudication from correctness. In a study with 369 participants, we find that, pooled across bias types, these interventions significantly shift judgments toward the manipulated side. These findings expose a vulnerability in debate-based supervision: human adjudication is sensitive to manipulative rhetorical strategies.

A42026-10-04cs.LG

Groupwise Distortion Guarantees for Preference-Based Alignment

Jacob Brodkey, Roberto Tamez, Aaron Roth

Preference-based alignment methods such as reinforcement learning from human feedback (RLHF) and Nash learning from human feedback (NLHF) aggregate pairwise preferences to learn an LLM policy, but a natural goal is maximizing social welfare (average cardinal utility), which comparisons alone need not identify. Gölz, Haghtalab, and Yang (GHY) measure the gap by distortion: the worst-case ratio between the welfare of the best fixed lottery (distribution over responses) and of the learned lottery. They show NLHF is optimal when every user receives the same lottery. Account-based LLMs, however, have information about their users and can serve different lotteries to different people. We give an efficient algorithm, GLHF, that learns a single group-conditioned policy from one comparison per user. Under individual Bradley--Terry comparisons, GLHF asymptotically matches GHY's optimal population distortion bound simultaneously on every group in a prespecified, possibly overlapping collection, with sample complexity growing logarithmically in the number of groups and inversely with the smallest group mass. A sharper guarantee for groups with similar preferences approaches distortion of one when members share a feasible favorite response. In experiments using human coffee ratings and synthetic LLM-generated ratings, GLHF lowers distortion in every evaluated group and substantially reduces worst-group distortion relative to NLHF and other group-agnostic baselines.

A52026-10-04cs.AI

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment

Jinghao Pang, Jitai Hao, Qiang Huang, Zhaochun Ren, Jun Yu

Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.

A62026-09-28cs.AI

Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling

Bohao Wang, Xiaoyan Zhao, Yang Zhang, Jinghang Guo, Chun Chen, Can Wang, Jiawei Chen

Reinforcement learning from human feedback (RLHF) aligns large language models (LLMs) with human preferences, yet most pipelines learn a single reward model that overlooks individual differences in preferences. Personalized reward models (PRMs) address this by conditioning rewards on user-specific feedback, most commonly through in-context learning (ICL), where a user's historical comparisons are supplied as contextual preference pairs. However, we identify a key limitation of ICL-based PRMs: they fail to capture the preference relations conveyed by contextual pairs. To address this, we propose Preference-Aligned Test-Time Training (P-TTT), which explicitly encodes these relations into user-specific fast weights for personalized reward prediction. P-TTT introduces sequence-level update and apply operations to match the response-level granularity of preference feedback, together with a preference-aligned objective that directly uses pairwise preference relations to guide fast-weight adaptation. Notably, P-TTT is simple to implement and computationally efficient, updating fast weights within a single forward pass without inference-time backpropagation. Extensive experiments show that P-TTT more effectively captures historical preference relations and outperforms state-of-the-art methods by a large margin.

A72026-09-27cs.LG

RMB: Reward Model Boosting Mitigates Reward Hacking

Jiabin Fan, Dezhi Ye, Yongchang Hao, Lili Mou

Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diversity-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.

A82026-09-11cs.LG

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

Kazusato Oko, Annie Ulichney, Nika Haghtalab, Han Bao

While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gölz et al. (2025) demonstrated that the distortion -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter β when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data (μ) and the KL reference policy (π_ref). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is tildeΘ(βB + β), where B is an upper bound on the log density ratio between μ and π_ref. In particular, when there is no distribution mismatch (i.e., μ= π_ref), RLHF achieves the optimal distortion of O(β) up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to μ.

Foundational papers

  1. 2022
    Training a helpful and harmless assistant with reinforcement learning from human feedback4,867 citations
  2. 2023
    Open problems and fundamental limitations of reinforcement learning from human feedback1,424 citations
  3. 2023
    A survey of reinforcement learning from human feedback595 citations
  4. 2023
    Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback744 citations
  5. 2024
    Safe rlhf: Safe reinforcement learning from human feedback1,046 citations
  6. 2024
    Personalizing reinforcement learning from human feedback with variational preference learning213 citations
  7. 2025
    Reinforcement learning from human feedback171 citations
  8. 2026-10
    Latest on arXiv: Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Ima…cs.CV
Citations per paperCitation counts via the AISA Scholar API
citations: R1 Open problems and fun… 1,424, R2 Training a helpful an… 4,867, R3 Rlhf deciphered: A cr… 284, R4 A survey of reinforce… 595, R5 Principled reinforcem… 438, R6 Rlaif vs. rlhf: Scali… 744, R7 Rlaif: Scaling reinfo… 861, R8 Reinforcement learnin… 171, R9 Safe rlhf: Safe reinf… 1,046, R10 Personalizing reinfor… 213R1 Open problems and fun…1,424R2 Training a helpful an…4,867R3 Rlhf deciphered: A cr…284R4 A survey of reinforce…595R5 Principled reinforcem…438R6 Rlaif vs. rlhf: Scali…744R7 Rlaif: Scaling reinfo…861R8 Reinforcement learnin…171R9 Safe rlhf: Safe reinf…1,046R10 Personalizing reinfor…213
R42023595 citations

A survey of reinforcement learning from human feedback

T Kaufmann, P Weng, V Bengs et al.

… learn a reward model from human feedback and then use this model to train a policy. Although it is possible to directly optimize a policy from human feedback … without reward learning, …

R82025171 citations

Reinforcement learning from human feedback

N Lambert

… its young history and a primer on reinforcement learning context needed to understand the … model and finally all of rejection sampling, reinforcement learning, on-policy distillation, and …

R920241,046 citations

Safe rlhf: Safe reinforcement learning from human feedback

J Dai, X Pan, R Sun, J Ji, X Xu, M Liu et al.

… To address this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly decouples …

How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.

Sources & citations

  1. Open problems and fundamental limitations of reinforcement learning from human feedback 1,424 citations
  2. Training a helpful and harmless assistant with reinforcement learning from human feedback 4,867 citations
  3. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms 284 citations
  4. A survey of reinforcement learning from human feedback 595 citations
  5. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons 438 citations
  6. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback 744 citations
  7. Rlaif: Scaling reinforcement learning from human feedback with ai feedback 861 citations
  8. Reinforcement learning from human feedback 171 citations
  9. Safe rlhf: Safe reinforcement learning from human feedback 1,046 citations
  10. Personalizing reinforcement learning from human feedback with variational preference learning 213 citations
  11. Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning arXiv · 2026-10-10
  12. A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration arXiv · 2026-10-07
  13. AI Safety via Debate is Compromised by Cognitive Biases arXiv · 2026-10-04
  14. Groupwise Distortion Guarantees for Preference-Based Alignment arXiv · 2026-10-04
  15. Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment arXiv · 2026-10-04
  16. Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling arXiv · 2026-09-28
  17. RMB: Reward Model Boosting Mitigates Reward Hacking arXiv · 2026-09-27
  18. Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner arXiv · 2026-09-11

Related