Azuma–Hoeffding inequality Bellman optimality equation Markov decision process Algorithm 1: Q-learning exploration–exploitation trade-off optimism in the face of uncertainty upper confidence bound by Hoeffding's inequality, Thompson sampling regret minimization Lai–Robbins lower bound w.p. at least 1 − δ, minimax regret contraction mapping theorem temporal-difference learning by the contraction mapping theorem, policy gradient theorem actor–critic methods trust-region policy optimization Lemma 1 (performance difference). maximum-entropy reinforcement learning Boltzmann exploration eligibility traces telescoping over t = 1, …, T: stochastic approximation martingale difference sequence sub-Gaussian concentration taking expectation over s′ ∼ P(·|s,a), union bound Chernoff bound importance sampling exploration vs. exploitation: off-policy evaluation occupancy measure contextual bandits Q.E.D. □ adversarial bandits online convex optimization mirror descent Algorithm 1: Q-learning follow-the-regularized-leader Lyapunov stability sample complexity by Hoeffding's inequality, value iteration · policy iteration dynamic programming principle Bellman operator w.p. at least 1 − δ, Robbins–Monro conditions Bellman backup · bootstrapping generalized advantage estimation by the contraction mapping theorem, natural policy gradient Fisher information matrix Kullback–Leibler divergence Lemma 1 (performance difference). successive elimination doubling trick best-arm identification telescoping over t = 1, …, T: pure exploration confidence radius Bernstein inequality taking expectation over s′ ∼ P(·|s,a), McDiarmid's inequality optional stopping theorem Publication Info
- Hengquan Guo, Qi Zhu, Xin Liu
- Learning for Dynamics and Control Conference (L4DC 2023)
- 2023
- Published