Experiment: Find a Path for Lizards

Did you read my article on the work in progress for Lizards in My House? Did you? Good. With this experiment I will bring you deep into the development of an Augmented Reality Application. I'm using Unity AR Foundation here, an interesting tool which can deal transparently both with Google AR Core and with Apple ARKit. The objective is to investigate a possible algorithm based on Points Clouds (which can be easily retrieved from any of the platforms cited previously) to keep and update a reference to the shape of the floor on which the lizards will move, walk and build their nests.

About MushroomsLabs Experiments

Mushrooms Labs experiments are fast development activities, made in 4-10 hours with a clear objective, usually involving the exploration of new technologies or of new tools for Curved Poly or for the Shadow Framework. I usually go straight forward to the goal, and, in the end, try to figure out what I did wrong. I think that if I can learn from my errors, it's worth to share them so that everybody can learn as well as me.

Project Setup in Unity

This experiment is part of Lizards in My House, so I opened the project and created a new scene rather than creating a new project from scratch. If you want to follow my steps, first of all you need to activate AR Foundation for Unity, following the instructions you can find in Unity Packages Window Reference (for me: this step was done the first time I opened the project for Lizards in My House). Then, following the steps you can find in the AR Foundation Manual I added to the project an AR Session and an AR Session Origin with an AR Camera.

Next step was to activate Plane Recognition and Points Clouds. With AR Foundation for Unity you can easily add them from the menu. The options are GameObject > XR > AR Defalt Plane and GameObject > XR > AR Defalt Point Cloud, which instantiate two Game Objects with default settings. Then, I had to register them on the AR Plane Manager and AR Point Cloud Manager, which should be on the same Game Object with the AR Session Origin. I didn't change the default settings, because in this way the planes and the the points will be visualized correctly in the App. This is a choice which make sense for testing and development purposes. Furthermore, for this experiment I wanted to start working only with Points Clouds, but I still wanted to keep an eye on the Planes, so keeping them visualized was very important to me.

Last but not least. You don't need to use Curved Polys for this experiment, but I did (The general goal of these experiments is to show that you can do everything on Unity with Curved Polys and that they are great for real time applications!). I had already integrated Curved Poly - Runtime in the project and had already imported the Lizards Models which was already used for Lizards in My House.

Planning a Strategy

This time I wanted to try something not too complex. I wanted to find a good dynamic straight line which describes the best direction the lizards can take while moving on the floor keeping away from the obstacles. My algorithm follows this steps:

• Y Candidate: I needed to find a good value for the vertical (y) components of points of the cloud being on the floor. This could have been done also by accessing plane data. Since I wanted to achieve this with points, I wrote an algorithm where I first take the average of all y components for all points. Then I try to refine the result by discarding all the y-values which are too far from the avarage. Of course: this make sense since the objective of the APP is to find Lizards on the ground, so I suppose people will often try to watch the ground while playing. Furthermore: when the APP is launched, ARKit and ARCore will usually reset the y values to be zero at device height, and the floor level should be in a range of negative values between -0.5 and -1.5 (in meters). For this reason, I first discard every y-value above a predefined threshold (which I can change via script). Another alternative to this may be to take the lowest y value among all available ones and calibrate the threshold to be close to them.
• Main Direction: All points in the Cloud are filtered, so that only the ones near the Y-Height Candidate are kept. Then I solve a classical Least squares problem to find the best straight line passing through that points. Since all the points in the Cloud from the obstacles are filtered away, this simple solution often generates a good line which is far from the obstacles.
• Integration: This time I didn't work on this, but it will be an important upgrade for the future. The two previous steps are applied at each frame, and the line will change continuosly while the Point Cloud follows a device movements. The objective of Lizards in My House is to build a map of free areas on the floor, a map which shall be updated and corrected continuosly while the user is playing. So, potentially, the map should filter all the lines found at each frames, discarding the ones which doesn't look good; this can be accomplished in many ways, for example removing the lines which were evaluated on a little number of sample points, because the Point Cloud was too rarefied or because the Y Candidate filter discarded too much of them. Also, the lines found at different positions in the house should be integrated together to find possible robust path which the Lizards can follow safely. In order to do this, I plan to mix data from this Main Direction analysis with data from the Planes, because I believe that a comparison between two different set of data is the best way to track possible errors in the evaluation of paths (and now you know why this time I wanted to start working with the Point Cloud while keeping an eye on Planes).

Visualizing Data

I added a few things in the App to visualize the result of the algorithm.

• A Red Line shows the dynamic direction found by the algorithms.
• Three orange Lizards on a raw whose positions and directions are evaluated with and avarage of the last positions of the line at each frame, after a user press a button named Place.
• A Little GUI with same labels which show technical informations about the algorithm.

Walking Around

Moving the device on the floor updates the position of the Red Line and of the Lizards. Below them, the Points of the Cloud and the Planes are visualized. So I can walk around my house and I can look for possible issues making it hard for the dynamic line solution to work. these issues are usually related to reflections on the floor ( especially the ones due to the light coming from the windows ), thin shaped obstacles (like the legs of chairs) or obstacles with bad materials (objects with dominant metallic parts, where the Point Cloud fails to find a good set of points). Here a set of screenshots taken from the App during the tests of the experiments.

Next Step

This is all. Next step is to work on the integration with Planes. AR platforms can discover the surfaces around the floor which are modelled as vertical planes, so the idea is to compare vertical planes with the dynamic lines directions: I will surely work around the distribution (here I mean the average and the variance) of distances between the lines and the vertical planes, to find out both a measure of possible errors and the position of the most good (here I mean: free from obstacles) regions on the floor. The final goal is to write a robust algorithm which can build routing paths on the ground for my lizards. I also need to work around persitance of data between different executions of the App, but I first need to understand what is a good set of data to be stored on the device after the app is closed to accomplish such a thing.