Convergence of Two-Timescale Markovian Stochastic Approximations with Applications in Reinforcement Learning
:::info Stub — Full Engineering Breakdown Coming This paper was auto-fetched from arXiv on 2026-06-01. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Vagul Mahadevan et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2605.31172 |
| Download | |
| Categories | cs.LG, stat.ML |
Abstract
This work studies the convergence of two-timescale stochastic approximations (SA), a class of iterative algorithms that update two sets of parameters in fast and slow timescales respectively. Notable examples of two-timescale SA in reinforcement learning (RL) include temporal difference learning with gradient correction (TDC) and actor-critic methods. Previously, the stability (i.e., boundedness) and convergence of two-timescale SA were only established under i.i.d. noise. This work instead establishes the stability and convergence of two-timescale SA under Markovian noise, a setup that is more realistic in RL. Notably, we do not need to use any projection operator and the noise does not need to live in a compact space. Our key technical novelty is to control the fast timescale parameter with the running max of the slow timescale parameter, instead of with the current slow timescale parameter, as most prior works do. As a key application, we establish the first almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation.
Engineering Breakdown
The Problem
Notably, we do not need to use any projection operator and the noise does not need to live in a compact space.
The Approach
This work studies the convergence of two-timescale stochastic approximations (SA), a class of iterative algorithms that update two sets of parameters in fast and slow timescales respectively. This work instead establishes the stability and convergence of two-timescale SA under Markovian noise, a setup that is more realistic in RL.
Key Results
As a key application, we establish the first almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation.
Research Areas
This paper contributes to the following areas of AI/ML engineering:
- Model training
- Generalization
- Optimization
- Supervised learning
- Deep learning
- Convergence
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::