Your input size is just too small for this model.
I’ve added the output shapes of the conv layers as comments:
conv_layers = nn.Sequential(
nn.Conv2d(3, 24, 5, stride=2), # [N, 24, 33, 158]
nn.ELU(),
nn.Conv2d(24, 36, 5, stride=2), # [N, 36, 15, 77]
nn.ELU(),
nn.Conv2d(36, 48, 5, stride=2), # [N, 48, 6, 37]
nn.ELU(),
nn.Conv2d(48, 64, 3, stride=2), # [N, 64, 2, 18]
...

As you can see, the spatial size if the last mentioned layer is 2x18, which is too small for a kernel size of 3 with stride=2.