Bringing the natural world into your living room

Micro Kingdoms: Senses provided us with the opportunity to tell the stories of two fantastic creatures: the Leaf Cutting Ants colony in the tropical rainforests of Central America and a Wandering Spider in the Atlantic Forest of Eastern Brazil.

Making use of the Natural History Unit’s detailed archives, we studied hours of video footage to select creatures based on their ‘senses’ and how they perceive the world around them. Leaf Cutter Ants use smell to forage leaves and produce pheromone trails to guide other workers. The Wandering Spider uses touch via tiny hairs on their bodies to detect vibrations of prey as it wanders the forest floor. Breaking down the key attributes and behaviours of each creature allowed us to establish the narrative journey and a story that worked in any space.

Once this foundational flow was created, and the senses weaved through it, our attention was turned to the player’s agency. From the start one of our key pillars was to create active storytelling: as a player you are taking part rather than being a passive observer.

Whilst the chapters have a distinct narrative arc, the way the story unfolds is linked to the user’s environment – the physical space they are in – and the way they choose to interact with it. Using a magnifying glass players can explore a specific scene and in other areas players can experiment with the props within the scene to see how the creatures react. An example of this is when the player is asked to lure out the Wandering Spider from under a log. Drop a rock near it and you’ll scare it, a leaf meanwhile will produce just the right vibration to coax it out. It’s all about how you use the environment.

Bringing the creatures and environmental props to life used a variety of techniques. First studying a large amount of visual material and footage of each of the creatures in their real world environments. We needed to be true to life, so detailing of the creatures’ joints, muscle weights and world textures were carefully created using hand crafted materials and photogrammetry assets.

Leaf floor litter, bark, fallen logs, banana plants and many other objects were created and brought into the experience true to real life scale. Visual and technical effects were applied to the props which allowed them to respond to a player’s touch and bend as they’re being plucked.

We knew this experience had to feel like it was alive so we made an early decision to drive the creatures using AI. This meant breaking down each possible locomotion movements and hand animate them, carefully checking against real footage. With all the animation clips created, these were linked up into an animator controller driven by the AI system.

The AI system acted as the brain thinking about if the creature was hungry, or could sense a phenomenon trail or just fancied a wander around. Added to this thinking was the player’s position to the creature, were they close or far away? Based on multiple decision inputs the AI system moved and played back the correct animation clip. The end result is a living breathing creature walking around your living room floor – and it knows where it’s going.

Sound effects pushed us to heighten motion, pull attention and drive the narrative. Using sounds effects spatially meant that if players knelt down they were rewarded with a richer sound scape of the creature’s noises against the environmental sounds. In the ants chapter, kneeling down revealed the small patter of ant footsteps in the sand coming closer to you.

With the wild track being formed from the folly sound effects, the next step was to add music scored against the narrative beats. The music, beautifully composed by Peter Mauder, was created in layers – danger, resolve, celebration moments, etc. This allowed us to sync and bring in different emotional musical layers on demand, allowing it to adapt to the narrative intent or a player’s agency.

With all these elements in place there was one missing piece: the voice over narration. For this Stephen Fry stepped in. One thing we were keen to do was make it feel as though Stephen was there with you commenting on the scene unfolding in front of your eyes.

A key part of the experience is the ability for players to trigger contextual information and additional voice over by using the magnifying glass. To avoid any abrupt interruptions, we built a system that transitioned players from the magnifying glass back to the main line of dialogue with the use of bridging – “Now where was I …”. This helped with the illusion of having Stephen there with you commenting on what you were both seeing.

The end result, a labour of love and challenges, brought together a wealth of disciplines within PRELOADED. Each working together to find visual, experience and technical solutions. The final product is a spatial computing experience which promises to be an exciting new way to tell interactive stories and engage people with the natural world.

If you’d like to learn more about BBC Earth – Micro Kingdoms: Senses, read the case study, or let’s talk!