I failed to get the new person tracking feature of the RealSense SR300 camera working in this previous post but when I wrote that post I was struggling to get the SR300 camera working at all.
In my last post I had more success in that I managed to get the camera working on my Surface Pro 3 with some UWP code which did some facial work.
That made me think that it might be time to give person tracking a try on my Surface Pro 3 and I’ve had much more success there although it’s fair to say that I don’t have everything working just yet and it hasn’t been a completely ‘free ride’ so far.
Person Tracking isn’t part of what’s supported in the UWP SDK and so I went back to the .NET SDK in order to try and get something up and running and the release notes for the SDK say that person tracking is in preview, so that needs to be remembered when experimenting with it.
the idea here is that the image can display video frames from the camera and the Canvas can overlay data picked up from the person tracking modules.
I’ll come back to why I ended up doing this in a moment.
// 1) Provides methods to 'copy/capture' per-frame data.
// 2) Provides methods to render that data to a canvas.
// Hence, we construct it with our canvas.
// Ask it to switch on person tracking.
// Configure person tracking to suit our needs.
// when it has frames of data for us.
// Mirror so that L<->R are reversed.
// We ask for a maximum of one person.
// I've yet to see that deliver anything that's not in the upper body.
// We check to see if our 'buffer' is busy. If so, we drop the frame and carry on.
// Copy the data for the color video frame.
// Copy the bounding boxes around the torso and the head.
// Copy any individual joints that we can find.
// Now, switch to the UI thread to draw what we have captured.
/// Draws the data that was previously captured.
// have the sizes from the video frames.
// Update with the latest video frame.
// Draw boxes for the head and torso.
// Draw ellipses for any joints we know about.
// missed frames in the meantime).
/// from the colour source.
Gather data from the frames of data as they arrive.
Render that data on the UI thread at a later point.
/// rendering a frame but not both at the same time.
/// dimensions of that frame on first use.
/// Captures the bounding boxes around the head and torso for later rendering.
/// Does the work of capturing the head and torso bounding boxes.
/// but it seems to be needed to get the joints.
/// showing/hiding them as I haven't tested.
// We shoot for the joints where there is at least some confidence.
/// Updates the writeable bitmap with the latest frame captured.
and then things started to work a bit better.
Note – I’m fairly certain that work could be done in the declaration of the external function that is called via PInvoke here in order to let the marshalling layer do more of the work around this array but, for now, I’ve tried to make the minimal change to fix the problem that I was seeing and, hopefully, the SDK will get updated and this error will go away.
As an aside, this post on the RealSense forums seems to have hit the same issue.
My performance isn’t great but I suspect I could do quite a bit to improve that.
I don’t seem to get too many joints reported from the SDK – usually left hand, right hand, mid spine and head.
I don’t often get the bounding box for the head reported, it seems quite sporadic.
but the bounding box for the torso seems rock solid.
I should also say that I’ve played around a few times with the SkeletonMode parameter that is passed to SetTrackingArea as part of the configuration here – I’ve varied it between its 4 options (upper body, upper body rough, full body, full body rough) to try and see which works best and I haven’t worked that out just yet.
I’m going to keep experimenting with this SDK as there’s a lot more data that can be obtained from it in terms of person tracking and, hopefully, more info and samples will come out around how it’s meant to work as I’m working somewhat in the dark with this at the moment but I thought I’d share this early experiment as an indication of what’s coming and so that it might help anyone else that’s experimenting with the preview too.