¶¶Òõ´«Ã½ÔÚÏß

Duration: 2 hours, 48 minutes

This module introduces function approximation as the transition point from tabular reinforcement learning to deep reinforcement learning. In Course 1, we represented values explicitly using tables: V (s), Q(s, a). This works when the state and action spaces are small enough to enumerate. But many reinforcement-learning problems have large, continuous, high-dimensional, or image-like observations. In such settings, tables are not enough. Course 2 replaces tables by parameterized functions: Vθ(s), Qθ(s, a), πθ(a | s). The parameter vector θ may represent a linear model, a neural network, or another differentiable function class. The central question of this module is: How do we learn value functions when tables are too large? The module also explains why deep RL is not merely supervised learning applied to RL data. The targets are noisy, bootstrapped, policy-dependent, and often moving as the parameters change. These difficulties lead to the deadly triad: function approximation, bootstrapping, and off-policy learning. The module ends with fitted value iteration as a bridge from tabular value iteration to deep Q-learning.

Duration: 1 hour, 42 minutes

This module develops value-based deep reinforcement learning as bootstrapped regression. In the previous module, we replaced tabular value functions by parameterized functions: Vθ(s), Qθ(s, a), πθ(a | s). We also saw that function approximation changes the learning problem: values are no longer stored independently, targets can move as parameters change, and bootstrapped updates can become unstable. This module applies these ideas to deep action-value learning. We begin with fitted value iteration, which turns Bellman updates into regression problems. We then study Deep Q-Networks, or DQN, where a neural network represents Qθ(s, a). DQN combines Q-learning targets with two important stabilizers: replay buffers and target networks. Finally, we study common DQN variants: Double DQN, dueling networks, and prioritized replay. The goal is to understand DQN not as a mysterious deep-learning recipe, but as Q-learning plus function approximation, bootstrapped targets, replay, and stabilization.

Duration: 2 hours, 35 minutes

This module introduces policy-gradient methods, a family of reinforcement-learning algorithms that optimize a parameterized policy directly rather than deriving behavior from a learned value function. Starting from the motivation for direct policy learning, the module develops the policy-gradient objective, the score-function trick that makes this objective differentiable from sampled experience, and REINFORCE, the foundational Monte Carlo policy-gradient algorithm. The module then introduces baselines as a practical variance-reduction technique and closes by motivating actor-critic methods as the natural next step once a learned baseline is introduced.

Duration: 1 hour, 39 minutes

REINFORCE updates the policy directly from sampled Monte Carlo returns, but those returns are noisy — the same policy can produce wildly different outcomes from episode to episode. This module introduces actor–critic methods, which tame that variance by learning a second component, the critic, that estimates how good a state or action is and feeds that estimate back into the policy update as a baseline. Learners will see how subtracting a learned value function from the return produces an advantage signal, how that signal generalizes from the one-step TD error to the multi-step Generalized Advantage Estimator, and how actor and critic are jointly trained via separate policy and value losses. The module closes by tracing the conceptual line from basic actor–critic methods to PPO, motivating why controlling the size of policy updates matters for stable learning.

Duration: 2 hours, 21 minutes

This module surveys modern deep reinforcement learning algorithms through the lens of stability, exploration, and continuous control. In the previous module, we studied policy-gradient and actor–critic methods. ÌýVanilla policy-gradient updates can be brittle. If the policy changes too much after one update, the new policy may perform much worse than the old one, and the data collected under the old policy may no longer be reliable for updating the new one. This module studies three major algorithmic ideas. First, we study conservative policy updates through TRPO and PPO. The main idea is to improve the policy while preventing overly large policy changes. PPO implements this idea using a simple clipped surrogate objective. Second, we study DDPG, a deterministic actor–critic method for continuous-control problems. Third, we study SAC, an entropy-regularized actor–critic method that encourages exploration and often improves robustness.

Duration: 1 hour, 47 minutes

This module turns deep reinforcement learning algorithms into implementation patterns. Earlier modules introduced the main algorithmic ideas: function approximation, DQN, policy gradients, actor–critic methods, PPO, DDPG, and SAC. This module asks how those ideas become working code. A deep RL implementation is not just a neural-network training loop. In supervised learning, the data are usually given in a fixed dataset. In reinforcement learning, the data are generated by an agent interacting with an environment. This means the implementation must manage environment interaction, exploration, neural-network models, optimizers, replay buffers or trajectory buffers, target networks, logging, evaluation, and reproducibility.

Duration: 2 hours, 12 minutes

This module contains materials for the final exam. If you've upgraded to the for-credit version of this course, please make sure you review the additional for-credit materials in the Introductory module and anywhere else they may be found.Ìý

• The exam is non proctored.
• It is a two-hour exam.
• You may submit your exam only once.
• The exam contains only multiple choice questions.
• You are not allowed to use any notes or access other websites when you take your exam.