Inverse RL & Preference Learning

Foundational IRL paper formalizing reward extraction from observed optimal behavior.

Extends IRL to practical apprenticeship learning with feature expectation matching.

Resolves IRL ambiguity using maximum entropy principle; now the standard IRL formulation.

Frames value alignment as cooperative game; foundational for AI safety.

First Bayesian framework for IRL; provides posterior distributions over rewards capturing uncertainty.

First deep IRL method learning arbitrary neural network cost functions; enabled learning from raw images.

Treats designed rewards as noisy observations of true objectives; addresses reward hacking and negative side effects.

Pioneering behavioral cloning; first end-to-end neural network steering for autonomous vehicles.

Solves distribution shift in behavioral cloning; reduces imitation to online learning with O(T) error.

GAN-style adversarial training directly learning policy without reward recovery.

Foundational work showing demonstrations accelerate RL; established paradigm for robot skill acquisition.

Learning from state-only observations without action labels; enables learning from video demonstrations.

Industry-defining paper: CNNs mapping pixels to steering; achieved 98% autonomous driving in road tests.

Foundational theorem: data is rationalizable iff it satisfies GARP; basis for computational revealed preference.

Makes Afriat computationally tractable; shows how to test and recover preferences nonparametrically.

Comprehensive modern treatment covering GARP extensions, complexity, and mechanism design applications.

Combines revealed preference with nonparametric estimation; sharp bounds on counterfactual demands.

Nobel Prize-winning random utility framework; foundation of discrete choice models used throughout tech.

Axiomatic foundations for perturbed utility models; generalizes logit choice to capture bounded rationality.

Extends random utility to sequential choice with preference correlation; applicable to session-based user modeling.

Foundational RLHF paper learning rewards from preference comparisons with ~1% feedback.

1.3B InstructGPT outperforms 175B GPT-3 on human preferences; foundation for ChatGPT.

RLAIF using AI self-critique against constitutional principles; Claude's training methodology.

Eliminates reward model and RL loop; preference optimization via simple classification loss.

Stable policy gradient algorithm with clipped objectives; THE optimizer underlying RLHF in all major LLMs.

Demonstrated reward model + PPO pipeline for text; direct precursor to InstructGPT methodology.

Unifies RLHF/DPO theoretically; Identity Preference Optimization fixes DPO overfitting issues.

Aligns LLMs using binary good/bad signal via Kahneman-Tversky prospect theory; no preference pairs needed.

Introduced dueling bandits for pairwise preference learning; enables online learning without absolute labels.

Extended dueling bandits to K arms with regret bounds; Interleaved Filter algorithm for search evaluation.

Comprehensive 108-page survey of dueling bandits variants, algorithms, and applications.

Safe preference-based optimization separating exploration from exploitation; applicable to clinical/robotics.

Propensity-weighted learning from logged actions; foundation for offline policy learning in recommendations.

Seminal click modeling paper; introduced cascade and position-based models; foundation for bias correction.

DBN click model capturing examination chains and satisfaction; enables unbiased relevance estimation.

Counterfactual framework for unbiased LTR; Propensity-Weighted Ranking SVM; highly influential for debiasing.

Comprehensive survey of click models, estimation methods, and applications to search evaluation.

Taxonomy of five safety problems: side effects, reward hacking, scalable oversight, safe exploration, distributional shift.

Demonstrates agents can pursue wrong goals even with correct specifications; distinct from reward hacking.

First systematic study of Goodhart's Law in RLHF; provides predictable scaling for safe optimization bounds.

Shows GPT-2 can supervise GPT-4; core empirical work on scalable oversight for superhuman AI alignment.

Netflix Prize winners' tutorial; latent factor models, implicit feedback, temporal dynamics; 14,000+ citations.

Weighted matrix factorization for clicks/views; confidence-weighted preference learning; industry standard.

Pairwise ranking optimization from Bayesian principles; first method optimizing ranking directly for implicit data.