Uncertainty-in-RL

The repository is for Reinforcement-Learning Uncertainty research, in which we investigate various uncertain factors in RL.

1. Uncertainty in Reward

1.1. Inverse Reinforcement Learning

• Apprenticeship Learning via Inverse Reinforcement Learning, Paper (2004)
• Maximum Entropy Inverse Reinforcement Learning, Paper (2008)
• Adversarial Inverse Reinforcement Learning, Paper (2018)
• Inverse Reward Design, Paper (2017)

1.2. Generative Adversarial Imitation Learning

• Generative Adversarial Imitation Learning, Paper(2016)
• A Connection Between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models, Paper(2016)
• Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets, Paper(2020)

1.3. Preference-based RL

• Deep Reinforcement Learning from Human Preferences, Paper(2017)
• Reward learning from human preferences and demonstrations in Atari, Paper(2018)
• End-to-End Robotic Reinforcement Learning without Reward Engineering, Paper(2019)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training, Paper(2021)
• Reward uncertainty for exploration in preference-based RL, Paper(2022)

1.4. Meta-Learning

• MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning, Paper (2021)

2. Uncertainty in Transition

2.1. Gaussian Process, Bayesian Neural Network

• PILCO: A Model-Based and Data-Efficient Approach to Policy Search, Paper(2011)

• Improving PILCO with Bayesian Neural Network Dynamics Models，Paper(2016)

• Weight Uncertainty in Neural Networks, Paper(2015)

• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, Paper(2016)

2.2. Model-Ensemble

• Deep Exploration via Bootstrapped DQN, Paper(2016)

• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, Paper(2017)

• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models, Paper(2018) Code

• Model-Ensemble Trust-Region Policy Optimization, Paper(2018) Code

2.3. Variational RL

• Auto-Encoding Variational Bayes, Paper(2013)

• Exploring State Transition Uncertainty in Variational Reinforcement Learning, Paper(2020)

• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Plannin, Paper(2022)

2.4. Robust RL

• Robust Control of Markov Decision Processes with Uncertain Transition Matrices, Paper(2005)

• Reinforcement Learning in Robust Markov Decision Processes, Paper(2013)

• Robust Adversarial Reinforcement Learning, Paper(2017)

• Robust analysis of discounted Markov decision processes with uncertain transition probabilities, Paper(2020)

• Robust Multi-Agent Reinforcement Learning with Model Uncertainty, Paper(2020)

• RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning, Paper(2022) Code

3. Uncertainty in State

3.1. Approximation of belief-states with Bayesian Filtering

• Deep Reinforcement Learning with POMDP, Paper(2015)

• QMDP-Net: Deep Learning for Planning under Partial Observability, Paper(2017)

3.2. Approximation of belief-states in vector representation with RNN

• Deep Recurrent Q-Learning for Partially Observable MDPs, Paper(2015)

• On Improving Deep Reinforcement Learning for POMDPs, Paper(2018) Code

• Shaping Belief States with Generative Environment Models for RL, Paper(2019)

• Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDP, Paper(2021)

• Memory-based Deep Reinforcement Learning for POMDP, Paper(2021)

3.3. Approximation of belief-states with variational inference

• Deep Kalman Filters, Paper(2015)

• A Recurrent Latent Variable Model for Sequential Data, Paper(2015)

• TEMPORAL DIFFERENCE VARIATIONAL AUTO-ENCODER, Paper(2018)

• VARIATIONAL RECURRENT MODELS FOR SOLVING PARTIALLY OBSERVABLE CONTROL TASKS, Paper(2020)Code

• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model, Paper(2020)

• Flow-based Recurrent Belief State Learning for POMDPs, Paper(2022)

3.4. Approximation of belief-states with Particle Filter

• DESPOT: Online POMDP Planning with Regularization, Paper(2013)

• Intention-Aware Online POMDP Planning for Autonomous Driving in a Crows, Paper(2015)

• Deep Variational Reinforcement Learning for POMDPs, Paper(2018)

• Particle Filter Recurrent Neural Networks, Paper(2020)

4. Uncertainty in Observation

• A Bayesian Method for Learning POMDP Observation Parameters for Robot Interaction Management Systems, Paper(2010)

• Modeling Humans as Observation Providers using POMDPs, Paper(2011)

• Adversarial Attacks on Neural Network Policies, Paper(2017)

• Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks, Paper(2017)

• Robust Deep Reinforcement Learning with Adversarial Attacks, Paper(2017)

• Whatever Does Not Kill Deep Reinforcement Learning, Makes It Stronger, Paper(2017)

• Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning, Paper(2019)

• Certified Adversarial Robustness for Deep Reinforcement Learning, Paper(2020)

• A Two-Stage Reinforcement Learning Approach for Multi-UAV Collision Avoidance Under Imperfect Sensing, Paper(2020)

• ADVERSARIAL POLICIES: ATTACKING DEEP REINFORCEMENT LEARNING, Paper(2020)

• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations, Paper(2021)Code

• ROBUST REINFORCEMENT LEARNING ON STATE OBSERVATIONS WITH LEARNED OPTIMAL ADVERSARY, Paper(2021)

• Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning, Paper(2021)

• Incorporating Observation Uncertainty into Reinforcement Learning-Based Spacecraft Guidance Schemes, Paper(2022)

• POLICY SMOOTHING FOR PROVABLY ROBUST REINFORCEMENT LEARNING, Paper(2022)

• RORL: Robust Offline Reinforcement Learning via Conservative Smoothing, Paper(2022)

5. Uncertainty in Constraint

• Budget Allocation using Weakly Coupled, Constrained Markov Decision Processes, Paper(2016)

• A Fitted-Q Algorithm for Budgeted MDPs, Paper(2018)

• Inferring geometric constraints in human demonstrations, Paper(2018)

• Learning Constraints from Demonstrations, Paper(2018)

• Safe Exploration in Continuous Action Spaces, Paper(2018)

• Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints, Paper(2019)

• Budgeted Reinforcement Learning in Continuous State Space, Paper(2019)

• Learning Parametric Constraints in High Dimensions from Demonstrations, Paper(2019)

• Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning, Paper(2020)

• Learning Constraints from Locally-Optimal Demonstrations under Cost Function Uncertainty, Paper(2020)

• WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning, Paper(2021)

• Inverse Constrained Reinforcement Learning, Paper(2021) Code

• Safety‑constrained reinforcement learning with a distributional safety critic, Paper(2022)

• Learning Behavioral Soft Constraints from Demonstrations, Paper(2022)

• Constrained Markov decision processes with uncertain costs, Paper(2022)