Training a double-jointed arm with DDPG — Deep Reinforcement Learning
Today we’ll see another powerful Deep Reinforcement Learning algorithm.

It’s important that you can know the basics about the Deep Reinforcement Learning, you can see my last article here (Collecting bananas with Deep Reinforcement Learning).

Before we start, I’ll share with you something more about Deep Reinforcement Learning. It’s fast, I promise!

Theory first

Reinforcement Learning is a subclass from Artificial Intelligence, and this subclass has different types of algorithms.

Types of reinforcement learning

It’s important to know about these types because each type has its own way of learning.

Value-based

Estimate the optimal value function Q* (s, a).

This is the maximum value achievable under any policy.

Policy-based

Search directly for the optimal policy π*.

This is the policy achieving the maximum future reward.

Actor-Critic

We combine Value-based and Policy-based methods.

The Actor is policy-based and Critic is value-based.

For this project I used an Actor-Critic Method. The algorithm choosed is DDPG (Deep Deterministic Policy Gradient).

DDPG works a little bit different comparing with the basic approach of Reinforcement Learning. We’ll have two neural networks inside the agent.

Now, let’s know more about the environment and the double-jointed arm!

Well, the environment that we have here it’s better than we have in the GIF above. I used Reacher Environment from Unity.

Environment

In this environment, a double-jointed arm can move to target locations.

The first version contains a single agent.

The second version contains 20 identical agents, each with its own copy of the environment.

DDPG is a good approach for this environment because we can have more than one agent. We can learn from all agents and share the knowledge with all agents as well.

State Space

The observation space consists of 33 variables corresponding to position, rotation, velocity, and angular velocities of the arm.

Actions

Each action is a vector with four numbers, corresponding to torque applicable to two joints. Every entry in the action vector should be a number between -1 and 1.

Reward

A reward of +0.1 is provided for each step that the agent’s hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.

To solve this environment we need to have 30+ score at last 100 episodes.
['Douglas Trajano']
['Python', 'Artificial Intelligence', 'Machine Learning', 'Deep Reinforcement', 'Deep Learning']