Menu Close

03 July 2018

Augmented Reality Sandbox

Lachlan Smith developed an augmented reality sandbox back in 2016. He writes about how the sandbox has evolved since then, and why he keeps coming back to work on it.

When you think of the ways that people normally interface with computers there are a few main ones that come to mind: the mouse and keyboard, and more recently the touchscreen. These common forms of input bridge the divide between the user and the computer. The Augmented Reality (AR) Sandbox project, like many other SensiLab projects, aims to remove this interaction divide by letting users interact with the sandbox, and therefore the computer, in a more natural way.

The idea of an AR Sandbox was made popular back in 2014 when academics at UC Davis created a sophisticated watershed simulation. Using the combination of a Microsoft Kinect, a projector and a table full of sand, contour lines could be projected onto the sand whilst simulated water could flow freely across the table. This project, which went viral on YouTube, was the inspiration for what my project would become.

I started work on an AR Sandbox back in 2016 when I was fortunate enough to be accepted to work at SensiLab on a Winter Scholarship Project. During this time, I embraced learning many new concepts, such as Shaders and designing software for an ever-changing sandy environment. It was challenging, but I had amazing fun designing a simplified water simulation, a bushfire simulation and I even created a mode where virtual spiders could freely roam the sandbox surface. The AR Sandbox was featured later that year at Monash’s Future Finders exhibition where it received encouraging feedback.

This video highlights some of the current capabilities of the AR Sandbox

In 2017 I had to decide on a final year project for my Electrical Engineering degree and I felt that the AR Sandbox still had lots of potential for interesting applications beyond what I had already developed. The idea of an agent-based simulation (think autonomous individuals) was brought to my attention by SensiLab director Jon McCormack. We decided on developing a dynamic ant colony simulation based on real ant colony behaviour.

This project explored how ants use their immediate surroundings to navigate complex environments using virtual pheromone trails. At the simulation’s core, the ants seek food and try to return it to the nest. Trails emerge and disappear as the simulation runs, while human interaction with the sandbox can disrupt trails and send the ants running for cover. A paper written about this simulation has been accepted by the IEEE conference ICME (International Conference on Multimedia and Expo) and I’m excited to present it in late July 2018 in San Diego.

Weather systems can be simulated using the AR Sandbox
Detail of the pressure bars projected onto the sandy landscape

Finally, in 2018, I started working on behalf of SensiLab for the Monash Earth, Atmosphere & Environmental Science Department. The department saw the real potential of the sandbox as an immersive teaching aid, so I’ve returned to building more environmental simulations. Over the past 3 months I’ve rebuilt the software using some recently learnt Compute Shader techniques, and improved almost every aspect of the original software built in 2016, unfortunately without the spiders for now! I’ve been working closely with many academics in the department and have created far more realistic simulations to be used in classes starting next semester.

The biggest addition developed has been a new Geology Simulation. This simulation looks at the way that geological beds (represented with different colours) breach the surface of the sandbox. Realistic geological volumes can be defined simply, but robustly, using a touchscreen on the side of the sandbox. We’ve already had the opportunity to use the new simulation in a 3rd year geology class and feedback so far has been fantastic.

Over the past 2 years of iterative development on the AR Sandbox I’ve really started to appreciate the power of natural interfaces to computers and technology. This project enables an intuitive interface to many complicated concepts whether it be the spread of bushfires, an ant colony or even a geological simulation. I’m very excited to see what other applications can be built for the sandbox in the future.