R lengthy or contain blocking code, such as networking requests.
Using RQ is as simple as creating a queue, and enqueueing the desired function along with the arguments you want to pass to it, according to the code in their “Hello World” example:
from redis import Redis from rq import Queue from my_module import count_words_at_url q = Queue(connection=Redis()) result = q.enqueue(count_words_at_url, '')
Let’s walk through how to use RQ to execute a function that grabs data from the Mars Rover API.
Setting up your environment and installing dependencies
Before moving on, you will need to make sure you have an up to date version of Python 3 and pip installed. Make sure you create and activate a virtual environment before installing any dependencies.
We will be using the requests library to get data from NASA’s Mars Rover API, and RQ for handling asynchronous tasks. With your virtual environment activated, run the following command in your terminal to install the necessary Python libraries:
pip install rq==1.1.0 requests==2.22.0
In order for RQ to work, you'll need to install Redis on your machine. That can be done with the following commands using
wget:
wget tar xzf redis-5.0.5.tar.gz cd redis-5.0.5 make
Run Redis in a separate terminal window on the default port with the command
src/redis-server from the directory where it's installed.
Getting started with Redis Queue
Let’s start with an example function to hit the Mars Rover API and print out the URL to an image taken by the rover. Since this function includes making an HTTP request, that means the code in it is blocking, making it a great example of something that RQ would be useful for.
Create a file called
mars.py and add the following code to it:
from random import choice import requests mars_rover_url = '' def get_mars_photo(sol): params = { 'sol': sol, 'api_key': 'DEMO_KEY' } response = requests.get(mars_rover_url, params).json() photos = response['photos'] image = choice(photos)['img_src'] print(image)
Run this code by opening a Python shell and entering:
from mars import get_mars_photo get_mars_photo(1000)
This will print a URL to a random image taken by the Mars Rover 1000 Martian solar days into its journey. Notice that it takes some time for the HTTP request to resolve and for the URL to print. While your code is waiting for the remote server to respond to your HTTP request, it can’t do anything else until it’s finished. Here is a photo that my code printed.
If you pass this function to RQ to be processed as an asynchronous task, then it will no longer be blocking the rest of your code. This can be done as easily as importing RQ, creating a queue, and enqueueing the function. Let’s do this by putting the following code in a file called
print_mars_photos.py:
from redis import Redis from rq import Queue from mars import get_mars_photo q = Queue(connection=Redis()) for i in range(10): q.enqueue(get_mars_photo, 990 + i)
In order for this code to work, you will have to run an RQ worker in the background in another terminal window for processing tasks..
To run the code we just wrote, run the command
rqworker in the same directory as your code in another terminal window, and then run your code with the following command:
python print_mars_photos.py
You should see the URLs for pictures taken on Mars being printed one by one, along with other information about the task in the terminal window where the worker is running, but not in the window where your code is running.
To demonstrate the non-blocking nature of this, try adding print statements before and after the loop, and executing regular function calls instead of enqueueing them:
print('Before') for i in range(10): get_mars_photo(990 + i) # q.enqueue(get_mars_photo, 990 + i) print('After')
You will see each photo URL printed, and then the “After” message. If you revert to the original code, but keep the print statements in, you will see the messages both print pretty much immediately while the code executing the HTTP requests is processed in your other terminal window.
There is a lot you can do with RQ workers and it’s worth reading the documentation to find out. RQ also provides you with a host of utilities on getting insight into jobs, including a CLI that makes it easy to requeue failed jobs. Here’s an example of the latter:
# This command will requeue all jobs in myqueue's failed job registry rq requeue --queue myqueue -u redis://localhost:6379 --all
To Infinity and Beyond
The code we used in this post are just short examples of what you can do with RQ, but after going through this post and the RQ docs, hopefully you find it easy to add RQ to your other projects. After being able to execute functions as tasks like we did above, adding RQ to your Flask or Django app means only writing a few lines of code wherever it's needed!
You can also use RQ Scheduler if you want to be able to schedule tasks. It is quick to add it to projects that already use RQ and has a simple API that allows you to execute Python functions at a given datetime.
For more examples of projects that use RQ, check out this post on how to receive text messages when the International Space Station flies above your location, or how to create a phone number that plays computer generated music that sounds like the soundtracks of old Nintendo games.
- Twitter: @Sagnewshreds
- GitHub: Sagnew
- Twitch (streaming live code): Sagnewshreds