Good advice, @Jason ! I think these are the best paths to follow.
From what you said @Cristian_Heleno Cristian, I guess that RefineNet will be able to solve your problem. The model was trained in the KITTI dataset and can recognize the following classes:

Airplane
Bicycle
Bird
Boat
Bottle
Bus
Car
Cat
Chair
CowDining Table
Dog
Horse
Motorbike
Person
Potted Plant
Sheep
Sofa
Train
TV

If you need to train a custom model, I recommend using the Computer Vision Annotation Tool (CVAT 3) on the Semantic Segmentation mode and follow this repo 15 instructions to retrain a RefineNet model using TensorFlow.
Then, you can easily convert the trained model to Tensorflow.js using the tf.js converter.