Hi, i have the same idea about building a self-driving toy car. You may consider incorporating the Dronet http://rpg.ifi.uzh.ch/dronet.html 60.
They provided the datasets, including the training & collision datasets and the model weights in Github, but in TF. Need to port to pytorch & retrain the toy car. Should be fun. Here is the wheel for RPi3 torch, https://github.com/github-kk/RPi3-PyTorch.0.5.0 72.
Thanks.