Understanding the Bellman operator part5(Artificial Intelligence + Reinforcement Learning) | by Monodeep Mukherjee | Apr, 2024

1. Finite-Sample Analysis of Off-Protection TD-Learning by means of Generalized Bellman Operators(arXiv)

Creator : Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, Karthikeyan Shanmugam

Abstract : In temporal distinction (TD) learning, off-policy sampling is considered additional smart than on-policy sampling, and by decoupling learning from data assortment, it permits data reuse. It’s acknowledged that protection evaluation (along with multi-step off-policy significance sampling) has the interpretation of fixing a generalized Bellman equation. On this paper, we derive finite-sample bounds for any primary off-policy TD-like stochastic approximation algorithm that solves for the fixed-point of this generalized Bellman operator. Our key step is to point that the generalized Bellman operator is concurrently a contraction mapping with respect to a weighted ℓp-norm for each p in [1,∞), with a common contraction factor. Off-policy TD-learning is known to suffer from high variance due to the product of importance sampling ratios. A number of algorithms (e.g. Qπ(λ), Tree-Backup(λ), Retrace(λ), and Q-trace) have been proposed in the literature to address this issue. Our results immediately imply finite-sample bounds of these algorithms. In particular, we provide first-known finite-sample guarantees for Qπ(λ), Tree-Backup(λ), and Retrace(λ), and improve the best known bounds of Q-trace in [19]. Moreover, we current the bias-variance trade-offs in each of these algorithms.