Azuma–Hoeffding inequality Bellman optimality equation Markov decision process Algorithm 1: Q-learning exploration–exploitation trade-off optimism in the face of uncertainty upper confidence bound by Hoeffding's inequality, Thompson sampling regret minimization Lai–Robbins lower bound w.p. at least 1 − δ, minimax regret contraction mapping theorem temporal-difference learning by the contraction mapping theorem, policy gradient theorem actor–critic methods trust-region policy optimization Lemma 1 (performance difference). maximum-entropy reinforcement learning Boltzmann exploration eligibility traces telescoping over t = 1, …, T: stochastic approximation martingale difference sequence sub-Gaussian concentration taking expectation over s′ ∼ P(·|s,a), union bound Chernoff bound importance sampling exploration vs. exploitation: off-policy evaluation occupancy measure contextual bandits Q.E.D. □ adversarial bandits online convex optimization mirror descent Algorithm 1: Q-learning follow-the-regularized-leader Lyapunov stability sample complexity by Hoeffding's inequality, value iteration · policy iteration dynamic programming principle Bellman operator w.p. at least 1 − δ, Robbins–Monro conditions Bellman backup · bootstrapping generalized advantage estimation by the contraction mapping theorem, natural policy gradient Fisher information matrix Kullback–Leibler divergence Lemma 1 (performance difference). successive elimination doubling trick best-arm identification telescoping over t = 1, …, T: pure exploration confidence radius Bernstein inequality taking expectation over s′ ∼ P(·|s,a), McDiarmid's inequality optional stopping theorem Publication Info
- Hengquan Guo, Xin Liu, Honghao Wei, Lei Ying
- Advances in Neural Information Processing Systems (NeurIPS 2022)
- 2022
- Published