On the Power of Optimism in Constrained Online Convex Optimization

Azuma–Hoeffding inequality Bellman optimality equation

Markov decision process Algorithm 1: Q-learning

exploration–exploitation trade-off optimism in the face of uncertainty

upper confidence bound by Hoeffding's inequality,

Thompson sampling regret minimization

Lai–Robbins lower bound w.p. at least 1 − δ,

minimax regret contraction mapping theorem

temporal-difference learning by the contraction mapping theorem,

policy gradient theorem actor–critic methods

trust-region policy optimization Lemma 1 (performance difference).

maximum-entropy reinforcement learning Boltzmann exploration

eligibility traces telescoping over t = 1, …, T:

stochastic approximation martingale difference sequence

sub-Gaussian concentration taking expectation over s′ ∼ P(·|s,a),

union bound Chernoff bound

importance sampling exploration vs. exploitation:

off-policy evaluation occupancy measure

contextual bandits Q.E.D. □

adversarial bandits online convex optimization

mirror descent Algorithm 1: Q-learning

follow-the-regularized-leader Lyapunov stability

sample complexity by Hoeffding's inequality,

value iteration · policy iteration dynamic programming principle

Bellman operator w.p. at least 1 − δ,

Robbins–Monro conditions Bellman backup · bootstrapping

generalized advantage estimation by the contraction mapping theorem,

natural policy gradient Fisher information matrix

Kullback–Leibler divergence Lemma 1 (performance difference).

successive elimination doubling trick

best-arm identification telescoping over t = 1, …, T:

pure exploration confidence radius

Bernstein inequality taking expectation over s′ ∼ P(·|s,a),

McDiarmid's inequality optional stopping theorem

Publication Info

This paper studies how optimism can improve algorithm design and performance guarantees in constrained online convex optimization.