Yes, FasterRCNN expects the num_classes as:

num_classes (int): number of output classes of the model (including the background).

That’s a good hint by the way. Maybe the illegal memory access is created in a custom CUDA extension used by this model, so we would need to dig into it.
How exactly did you setup the classes, so that we could try to reproduce it?