Hope you are doing well.
I am trying to build a following robot which follows a person.
I have a raspberry pi and and a calibrated stereo camera setup.Using the camera setup,i can find depth value of any pixel with respect to the reference frame of the camera. 
My plan is to use feed from the camera to detect person and then using the stereo camera to find the average depth value thus calculating distance and from that calculate the position of the person with respect to the camera and run the motors of my robot accordingly using PID.
Now i have the robot running and person detection using HOGdescriptor that comes opencv.But the problem is,even with nomax suppression, the detector is not stable to implement on a robot as too many false positives and loss of tracking occurs pretty often.
So my question is,can u guys suggest a good way to track only people. Mayb a light NN of some sort,as i plan to run it on a raspberry pi 3b+.
I am using intel d435 as my depth camera. 
TIA
Raspberry pi does not have the computational capacity to perform object detection and realsense driver support, check out the processor load once you start the realsense application. One of the simplest models for person detection is opencv's HOGdescripto that you have used.
fast light and accurate person-detection algorithm to run on raspberry pi