The REINFORCE tutorial 4 is for Google Colab, which runs on the Cloud. You are recommended to use it to utilize the free GPU acceleration (for a number of hours) and the setup simplicity, since deep RL can be compute intensive and sample-inefficient (see Reinforcement learning, fast and slow by DeepMind’s Matthew Botvinick et al., 2019).
If you’re running the notebook locally:

On a Linux machine, you might also need to add x11-utils:

!apt-get install -y xvfb x11-utils
!apt-get install -y ffmpeg


On a macOS machine, to install xvfb you may need to check the answer on StackOverflow: linux - xvfb-run on OS X - Stack Overflow which may have a good answer. To install ffmpeg, check out Which ffmpeg package I should download for macOS? - Super User - you might want to choose Homebrew (brew install ffmpeg).

The xvfb and ffmpeg dependencies are needed for video playback to watch the replays of your RL agent. “Xvfb (X virtual framebuffer) is a display server implementing the X11 display server protocol. It runs in memory and does not require a physical display. Only a network layer is necessary.” (source). (For more information, you can read about X virtual frame buffer/xvfb and FFmpeg. You can also read about X11.)
Regarding this step:



 F_J:

pip install ‘imageio==2.4.0’


If you use back ticks, make sure you use ' and not ‘ or ’. No back ticks should work, as @lgusm mentioned here:



 lgusm:

!pip install imageio==2.4.0



You may find this post useful: Rendering OpenAi Gym in Google Colaboratory. – StarAi – StarAi Applied Research Blog