Calculate Mixed Reality Object Locations Using BoxCastAll
Calculate Mixed Reality Object Locations Using BoxCastAll
Learn how to calculate mixed reality object locations in this tutorial using BoxCastAll. Read this post to view this step-by-step guide on object locations.
Join the DZone community and get the full member experience.Join For Free
Introduction
It was quite difficult to name this article. Usually, I try to find a title that more or less describes a search term that I used when I was looking for information on the topic at hand, but I could not really find what I was looking for. What I have here is code that calculates locations for objects to be in front or on top of other objects and/or the spatial mesh. For this project, I use BoxCastAll, something I have tried to use before, but not very successfully. I have tried using Rigidbody.SweepTest, and although it works for some scenarios, it did not work for all. My floating info screens ended up half a mountain away (in Walk the World), or the airport could not move over the ground because of some tiny obstacle blocking it (in AMS HoloATC). So, I tried a new approach.
This is part one of a two-post blog post. In this post, I will explain how the BoxCast works and what extra tricks and calculations were necessary to get it to work properly.
BoxCast magic
So what is a BoxCast, actually? It’s comparable to a normal RayCast. But, where a RayCast gives you the intersection of a line and an obstruction, a BoxCast does that — surprise — with a box. You essentially throw a box from a point along a vector until it hits something — or some things — as a BoxCastAll potentially returns more than one hit. If you take the one that is closest to your camera (a hit has a convenient “distance” property), you potentially have a place where you can place the object.
However, it does not take into account the following things:
- An object’s (center) position and the center of it’s bounding box are not always the same; this will make the BoxCast not always happen at the place you itend for it to happen.
- The vector from the camera to the hit may or may not be parallel to the direction of the BoxCast; therefore, we need to project the vector from the camera to the hit on the vector of the BoxCast.
- The BoxCast hit detection happens at the edge of the casted box, and an object's position is determined by its center. So, we need to move back a little towards the camera; otherwise, about half of our object — determined by its actual orientation — will end up inside the obstruction.
My code takes all of that into account. It was quite hard-won knowledge before I uncovered all the lovely pitfalls.
First, a New Utility Method
For a BoxCast to work, you need a box. You typically accomplish that by getting the bounds of all the renderers in the object that you want to cast and combine those into one big bounding box. I hate typing or copying code more than once, so I created this little extension method called
GameObject.
public static class GameObjectExtensions { public static Bounds GetEncapsulatingBounds(this GameObject obj) { Bounds totalBounds = new Bounds(); foreach (var renderer in obj.GetComponentsInChildren<Renderer>()) { if (totalBounds.size.magnitude == 0f) { totalBounds = renderer.bounds; } else { totalBounds.Encapsulate(renderer.bounds); } } return totalBounds; } }
BoxCast Magic
In
LookingDirectionHelpers, a static class containing utilities to calculate directions and places the direction the user is looking (duh). I have created a method that does the BoxCast magic. It does quite a lot, and I am going to walk you -through it step-by-step. It starts like this:
public static Vector3 GetObjectBeforeObstruction(GameObject obj, float maxDistance = 2, float distanceFromObstruction = 0.02f, int layerMask = Physics.DefaultRaycastLayers, BaseRayStabilizer stabilizer = null, bool showDebugLines = false) { var totalBounds = obj.GetEncapsulatingBounds(); var headRay = stabilizer != null ? stabilizer.StableRay : new Ray(CameraCache.Main.transform.position, CameraCache.Main.transform.forward); var hits = Physics.BoxCastAll(GetCameraPosition(stabilizer), totalBounds.extents, headRay.direction, Quaternion.identity, maxDistance, layerMask) .Where(h => !h.transform.IsChildOf(obj.transform)).ToList();
As you can see, the method accepts quite a few parameters, and most of them are optional:
obj— the actual object to cast and place against or on top of the obstruction
maxDistance— the maximum distance to place the object from the camera (if it does not hit another object first)
distanceFromObstruction— the distance to keep between the object and the obstruction
layerMask— what layers should we ‘hit’ when we are looking for obstructions (default is everything)
stabilizer— used to get a more stable location and viewpoint source than the camera itself
showDebugLines— use some awesome help classes I nicked from the Unity Forums from “HiddenMonk” to show how the BoxCast is performed. Without these, I would not have been able to identify all issues that I had to address.
First, we get the total encapsulating bounds. Then, we check to see if we canuse the stabilizer that to define a ray in the direction that we want to cast. Then, we calculate a point dead ahead of the camera.
And, then, we do the actual BoxCast, or actually a BoxCastAll. The cast is performed:
- From the Camera position;
- Using the total extents of the object;
- In the direction of the viewing ray (so a line from your head to where the gaze cursor is);
- Using no rotation (we used the Render's bounds, that already takes any rotation into account);
- Over a maximum distance;
- Against the layers described by the layer mask (default is all).
Notice the Where clause at the end. BoxCasts hit everything, including child objects of the cast object itself, as it may be in the path of its own cast. So, we need to weed out any hits that apply to the object itself or its children.
The next piece visualizes how the BoxCast is performed. using HiddenMonk's code:
if (showDebugLines) { BoxCastHelper.DrawBoxCastBox(GetCameraPosition(stabilizer), totalBounds.extents, headRay.direction, Quaternion.identity, maxDistance, Color.green); }
This uses debug. Next, draw these lines so that they are only visible in the Unity editor and in Play mode. They will not show up in the game pane but in the scene pane. This makes sense, as you can them look at the result from every angle without affecting the actual scene in the game.
It will look something like this:
Now, to address the issues I listed on top of this article, we need to do a few things.
Giving it the Best Cast
The next line is a weird one but is explained by the fact that there may be a difference between the center of the actual bounding box (and thus the cast) and center of the object as reported by Unity. I am not entirely sure why this is, but, trust me, it's happened with some objects. We need to compensate for that.
var centerCorrection = obj.transform.position - totalBounds.center;
Below, you see an example of such an object. It typically happens when an object is composed of one or more other objects that are off center, especially when the object is asymmetrical, like this "floating screen." You will see it's an empty game object, containing a Quad and a 3DTextPrefab that are moved upwards in local space. Without the correction factor, you get the situation on the left — the BoxCast happens "too low."
On the right side, you see the desired effect. I opted to change the location of the object to the center of the BoxCast. However, you might also consider changing the start location of the BoxCast, but that a side effect: the ray won’t start at the user’s viewpoint (but, in this case, a little bit above it), which might be confusing or produce undesirable results.
Hit or Miss: Projection
We need to find the closest hit. But, it might not be right in front of us, along with the viewing vector. So, we need to create a vector from the camera to the hit, then make a (longer) vector that follows the user’s gaze, and, finally, project the ‘hit vector’ to the ‘gaze vector’. Then, and only then, we know how much room there is in front of us.
if (hits.Any()) { var closestHit = hits.First(p => p.distance == hits.Select(q => q.distance).Min()); var hitVector = closestHit.point - GetCameraPosition(stabilizer); var gazeVector = CalculatePositionDeadAhead(closestHit.distance * 2) - GetCameraPosition(stabilizer); var projectedHitVector = Vector3.Project(hitVector, gazeVector);
To show what happens, I have made a screenshot where I made Unity draw debug lines for every calculated vector:
if (showDebugLines) { Debug.DrawLine(GetCameraPosition(stabilizer), closestHit.point, Color.yellow); Debug.DrawRay(GetCameraPosition(stabilizer), gazeVector, Color.blue); Debug.DrawRay(GetCameraPosition(stabilizer), projectedHitVector, Color.magenta); }
This results in the following view (for clarity, I have disabled the code that draws the BoxCast for this screenshot)
A little magnification shows the area of interest a little bit better:
You can clearly see the yellow line from the camera to the original hit. The blue line is the viewing direction of the user, and the magenta line is projected on that.
Keep Your Distance, Please
Now, this all works fine for a flat object like a Quad (posing as an info screen here) but not on a box like this for instance (which I made partially translucent for clarity).
The issue here is simple. Although it took me some time to figure out what was causing it, the hit takes place at the edge of the shape. However, the object's position is tied to its center. So, if I set the object's position to that hit, it will end up halfway the obstruction. QED.
What we need to do is make yet another ray that will go from the center of the object to the edge, following the same direction as the projected hit vector (the magenta line). Now, RayCasts don't work from inside an object, but, fortunately, there's another way. The bounds class supports an IntersectRay method. It works a bit kludgy IMHO, but it does the trick:
var edgeRay = new Ray(totalBounds.center, projectedHitVector); float edgeDistance; if(totalBounds.IntersectRay(edgeRay, out edgeDistance)) { if (showDebugLines) { Debug.DrawRay(totalBounds.center, projectedHitVector.normalized * Mathf.Abs(edgeDistance + distanceFromObstruction), Color.cyan); } }
We will intersect the projected hit vector from the center of the bounds to the edge of the bounds. This will give us the distance from the center to the part of the object that hit the obstruction, and we can move the object 'backward' to the desired position. Since I specified a '
distanceFromObstruction ,' we can add that to the distance the object needs to be moved 'back' as well to keep a distance from an obstruction, instead of touching it (although for this object it's 0). Yet, another debug line can, this time, show what's happening:
The cyan line is the part over which the object is moved back. Now, the only thing left is to calculate the new position and return it. This time, using the
centerCorrection that we used before, we will make the object actually appear within the BoxCast's outlines:
return GetCameraPosition(stabilizer) + projectedHitVector - projectedHitVector.normalized * Mathf.Abs(edgeDistance + distanceFromObstruction) + centerCorrection;
Nobody Is Perfect
If you think "hey, it looks like it is not completely perfectly aligned," you are right! This is because Unity has its limits in determining volumes and bounding boxes. This is probably because the main concern of a game is performance, not 100 percent accuracy. If I add this line to the code:
BoxCastHelper.DrawBox(totalBounds.center, totalBounds.extents, Quaternion.identity, Color.red);
It actually shows the bounding box:
This explains a bit more of what is going on. With all the debug lines enabled, it looks like this:
Show and Tell
It’s actually not easy to properly show you how this method can be utilized. I will save that for the next post. In the meantime, I have cobbled together a demo project that uses the
GetObjectBeforeObstruction in a very simple way. I have created a
SimpleKeepInViewController that polls every so many seconds (2 is default) where the user looks, then calls
GetObjectBeforeObstruction and moves the object there. This gives a bit of a nervous result, but you get the idea.
public class SimpleKeepInViewController : MonoBehaviour { [Tooltip("Max distance to display object before user")] public float MaxDistance = 2f; [Tooltip("Distance before the obstruction to keep the current object")] public float DistanceBeforeObstruction = 0.02f; [Tooltip("Layers to 'see' when detecting obstructions")] public int LayerMask = Physics.DefaultRaycastLayers; [Tooltip("Time before calculating a new position")] public float PollInterval = 2f; [SerializeField] private BaseRayStabilizer _stabilizer; [SerializeField] private bool _showDebugBoxcastLines = true; private float _lastPollTime; void Update() { if (Time.time > _lastPollTime) { _lastPollTime = Time.time + PollInterval; LeanTween.move(gameObject, GetNewPosition(), 0.5f).setEaseInOutSine(); } #if UNITY_EDITOR if (_showDebugBoxcastLines) { LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance, DistanceBeforeObstruction, LayerMask, _stabilizer, true); } #endif } private Vector3 GetNewPosition() { return LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance, DistanceBeforeObstruction, LayerMask, _stabilizer); } }
There is only one oddity here – you see I actually call the
GetObjectBeforeObstruction twice. But, the first time only happens in the editor and only if you select the Show Debug Boxcast Line checkbox:
If I did not add this, you would see the lines flash for one frame every 2 seconds, which is hardly the point. This way, you can see them at all times in the editor
In the demo project, you will find three objects — in the images above you have already seen a single block (the default), a rotating ‘info screen’ that shows “Hello World,” and there’s also a composite object on the left (two cubes off-center). Here, it is displayed with all debug lines enabled. You can toggle between the three objects by saying “Toggle” or by pressing the “T." The latter will actually work in a HoloLens, if you have a Bluetooth keyboard attached. And, believe me — I tried!
Conclusion
Here is another way to make an object appear next to or on top of an obstruction. This code actually took me way too much time to complete, but I learned a lot from it and at some point, it became a matter of honor, once I got the thing to work! }}