Reef Digital Twins

Blog by Roberto Giudici, ESR of 4D-REEF at OceanMaps in Austria.

 

Scientific descriptions of reefs contain a wealth of information, but are difficult to communicate in a form that can be easily understood by the general public. Digital Twins programs have recently been introduced for assisting scientific communities in data fostering and outreach, turning complex experiments into pleasant stories. Born for creating digital models of industrial scenarios, these solutions are emerging as educational tools in science.  I, ESR Roberto, ventured in the development of an educational Digital Twin, showing the reconstruction of a reef on a coralline beach, in form of 3D immersive Serious Game, featuring images and descriptions of the corals and their state of health.

The blog contains links to download two developed Digital Twins (link 1 and link 2)

 

What is a Digital Twin

A Digital twin is a technology that allows scientists to create a virtual replica of a physical object or system. This replica can be used to simulate the behaviour of the physical object or system, and can fit a wide range of applications. Digital Twins have the potential to revolutionize the way scientists and researchers study and understand complex any dynamic Earth’s processes, bridging the gaps between research and dissemination and harnessing specialist expertise in topographic modelling with interactive maps.

Roberto modelled two Digital Twins; an entire coralline island scenario, similar as in the Spermonde field study area, featuring vegetation, sandy beach, coral reef and fish, to virtually unveil the secrets of this fascinating environment and describe its processes.

Digital Twin developed for turbid reef visualisation

The development of a Digital Twin of a selected reef at the Spermonde archipelago, required the creation of a fully interactive 3D replica of the most relevant entities, featuring the creation of a topographic map of the seabed and emerged islands, a set of layers of ancillary information, including photographic, descriptions, etc. The resulting scenario would be presented into 3D immersive form, to let the user diving in a virtual reef, plounging the user in the beauty of the underwater world helping raising awarness about these endangered ecosystems. I decided to target an audience of young students, showing them the reefs as games, as proposed by several ESRs and supervisors from 4D-REEF.

I chose to develop the Digital Twins with Unity 3D, a market leading 3D Advanced Gaming Engine. Unity 3D is multiplatform and Multi Target, allowing to generate PC, Tablet, and Web based Gaming solutions at once. Moreover, this engine features an extremely large Repository of 3D objects and Terrains, including, vegetation, rocks, buildings, fish and other underwater animated species that can dynamically be added to our scenes to create more realistic scenes. The terrain can be either digitally synthetized through a terrain generator application to create pseudo-random topographic and morphological features, or we can import existing terrains, issued of Topo-Bathymetric surveys. In this latter case, the original terrain will be first composed by a 3D Point-Cloud and has to be converted into a 3D Mesh, through an operation called Triangulation. The terrain needs to be georeferenced during acquisition or Post Processing, with Geographical Coordinates issued of a GPS receiver.

Digital Twins Use cases: Focus on CAO, BIM and GIS

Digital Twins have so far mainly been used for industrial applications, providing substantial assistance in industry and construction.  Seabed mapping and benthic classification developed only when reliable Multibeam Echo-Sounders and Underwater Laserscans came into market. Newer fully interoperable 3D engines are leveraging on advanced parallel computing to offer a fully immersive and interactive experience, based on first person’s view, where the users can freely navigate in virtual ocean depths and reach inaccessible regions, straight from their home’s pc screen. I tried to first characterise some potential applications of a Digital Twin of the Spermonde Islands, and identified three main options, ordered by growing complexity:

1. SCIENTIFIC DISSEMINATION TOOL: To visually and interactively make aspects of research insightful. Needs describing a mission of Research, by its geographical locations, position and type of Sampling operated on the site, Georeferenced Clickable Images and Pop-Up windows with Explanations. This model corresponds to navigable 3D Hypertext, with limited interoperability and realm. It cannot be used for simulations or serious games.
2. MARINE/COASTAL INFORMATION MANAGEMENT SYSTEM (MIMS/CIMS): This corresponds to the transposition in 3D of a GIS system connected to Geographical Data Bases for Real-Time Data Update. It features limited 3D navigability, but cannot be used for simulations, or serious games.
3. SERIOUS GAME/SIMULATION: This is a fully immersive simulated Marine environment, featuring Coastal, Underwater and Benthic entities, such as Terrain, Sky, Clouds, Waves, Fish, Seagrass, Mangrove, Coral, Rocks and CAO entities such as Pipelines, ROVS, etc. It can feature advanced simulation of the underlying processes and manipulations (operating a ROV in a Pipe Deployment/Inspection mission. Physics Laws may be applied on the natural and CAO entities, to model the behaviour of Coastal Erosion due to underwater sedimentary transport, the Zooplacton Daily Vertical Migration (DVM) in the water column. The Laws of physics may extend to Weather conditions, light and colour absorption in the water Colum and reflection from the seabed. This latter scenario may also dynamically model the natural Spawn of Corals, Sea grass and fish swimming in schools driven by Social Algorithms (Algorithm of the Boids), with different types of realm depending of the Targeted Academic/Industrial/Retail community, and the Target Device (Mobile Device, PC, Web, etc.…). One or more characters driven by user interaction or pseudo random (or AI) cinematics, may also be added, featuring First Person Cameras (where the scene is seen from the eyes of the character), of Third Person Camera, where the User can see the entire scene (or a portion of it), independently of the movement of the characters. The realisation of this latter scenario may require an extensive amount of resources and a well-defined Project build on interactive participation of the Customer and the technical Teams. Based on the requirements, the cycle of production for this scenario may require several months of development.

Development Process

I developed two Digital Twins. As Covid-19 travel restrictions prohibited the 4D-REEF ESRs from travelling to the Indonesian Spermonde archipelago, conducting bathymetric surveys in the field had to wait until summer 2022. Therefore I developed a pseudo-random Terrain using Gaia, one of the most advanced Generator working as Plugin from within the Unity Editor. This terrain would be replaced by the bathymetric maps of the Spermonde, once available.

 

3D Scene Hierarchy in game engine Unity 3D

Scene: A 3D Scene is made up by a set of “Entities” that represent the abstraction of the Real World. The distances in a Scene are expressed in Points. Our Digital Twin has been developed in a Projected Georeferenced Space. The creation of a Georeferenced Space Layer of has been built by Ocean Maps, through a special customized application, as Unity 3D, being a 3D Gaming Engine, does not provide this Layer of Abstraction. The vegetation, rocks and Human crafted entities are all added to the Terrain. The Terrain, Cameras, Lighting, Sea, Weather, Sound, etc., are all attached to the Scene.

1. Terrain: This is the digital reconstruction of the soil, featuring, flat land, hills and mountains. It can be built as Pseudo-Random or imported an external 3D Mesh, normally obtained by Laserscan Photogrammetric or Bathymetric reconstructions. Unity 3D offers several cutting-edge Terrain Editors, like Gaia, used in this project.
2.  Cameras: They represent the Viewport used by the user to see the terrain. They are defined by a Field Of View (FOV) that is the width of the region that can bene seen and its relative scale. They can be positioned freely by the user, and the FOV behaves as lenses of a real camera.
3. Characters: They represent the main actors of the Serious Game and they can either be animated Humans, Animals, or Machines. They can have different levels of Animation.
4. Lighting: Lighting sources are added to enlighten the scene and simulate real-environmental conditions. They can be freely deployed by the user, under Spot or Diffused, and beam width, direction and colour (light temperature) and their parameters can be changed during Run-Time.
5. Sea, Weather, Sound, etc.: These entities are commonly used in Serious Games provide a better realm and user experience to the scene.

Digital Twin 1: Serious Game: Dive in the reef

This Digital Twin can be downloaded here (link).

The process of creating a Digital Twin of a coralline island is illustrated by the pipeline shown at Fig. 1.  We decided to use the Gaia Terrain Generator that features a set of tools for creating all the entities required in the current project. The operations required for creating and populating the Terrain are shown at Fig. 2.

Figure 1: Process of Island Generation

Figure 2: Pipeline for the creation of a Digital Twin for Dissemination

 

Creation of the Terrain: The first step required the creation of a terrain featuring a tropical coralline sedimentary island. We created the terrain using a Stamp provided by the Gaia Editor (Fig. 3).
Procedure for Generating a Coralline island: Gaia provides existing Stamps featuring the most common morphologies: Mountains, Flat Land,  Hills, Island, Archipelago, etc. However, custom Stamps can also be imported.

Properties of the island:

• The Planimetry of the island is defined by a Stamp that can reproduce straight coastline, Bays, Gulfs, etc.).
• Morphology of the Island: Elevation
• Ruggedness: smooth, polished rock to rough crystalline
• Topographic Discontinuities: Valleys, Graben, etc.
• Sedimentary feature: Type and Grain size. Different grain types can be included and the transition to the different types can be set through linear or custom laws

Figure 3: Creating a Pseudo-Random Island with UNITY 3D – Gaia

 

Gaia generates patterns of sediment with different of Sedimentary Facies and Grain Size and Morphological Structures, featuring longshore ripples with different orientations and height. This allows building complex sedimentary architectures at once. Morphological structures can also be highly customised with Scars, Graben, etc. (Fig. 4, 5).

Figures 4, 5: Pseudo Random Sedimentary Rugosity and Scars

 

Vegetation: The origin of the plant 3D specimens had to be defined ahead of any addition. We needed to procure the right set of entities useful for the project on the Unity Asset Store (The Virtual Market where entities can be purchased and downloaded). As the Spermonde Islands are Coralline islands, with no volcanic, nor Karstic formations, we selected a coralline sand and lowered the topography to get a low-lying island. We chose the right assemblage of plants after studying the distribution of some coralline islands at the Spermonde, namely Samalona, Kodigareng Keke and Langkai.

Spawning the Entity on the Terrain: The Spawning operation on Gaia allows adding dozens of hundreds of specimens at once, with different pseudo-random spatial distributions, densities, sizes and orientations. We spawned first the vegetation of the emerged and riparian regions, adding palm trees along the beach and trees at the centre of the island. As these islands are just a few meters above the level of the sea, rainwater cannot accumulate, and therefore, mangrove forests cannot develop. In a second time, we added the seagrass and coral in the underwater in-shore region, above the reef flat (Fig. 6). The entire reef will be then covered by water, whose level is set during the construction of the model and can be freely changed on runtime, to simulate tides. Our model did not include tides, as the Spermonde has a micro tidal regime with a total tidal amplitude of approximately 50 cm on spring tide.

Figure 6: Coral Assemblage

Environmental and Weather Engines:  Simulate the water, with water level, water turbidity, water caustics, and waves, based on the direction and strength of the wind generated by the weather engine. We also simulated changes in Sun light penetration in the water column, based on changing weather conditions across time on the scene.

Daylight: Based on the month, time and weather conditions, the environmental engine sets the sun position, height, intensity and light temperature, recreating stunning environmental conditions.
The final assemblage featuring vegetation and coral (shown at Fig 7).

Figure 7: Final Terrain Assemblage

Fish and underwater living forms: The addition of underwater forms of life like fish that move within the scene, is a challenging process. In particular, the computation of the positions of fish swimming in a school requires the implementation of Algorithms driving the Social Behaviour of fishes, like the Algorithm of the Boids that reproduces the Flocking behaviour of fish (Fig. 8).

The algorithm for fish movement is composed of Four Rules:

1. Cohesion: Fish search for their neighbours in a radius defined as the Radius of Cohesion. The current positions of all neighbours are summed up. The result is divided by the number of neighbours. Thus, the centre of mass of the neighbours is obtained. This is the point to which the fish strive for cohesion.
2. Separation: Fish search for their neighbours in a radius defined as the Separation Radius. To calculate the motion vector of an individual fish in a specific separation direction from a school, the difference in the positions of the neighbours and its own position is summed up. The result is divided by the number of neighbours and then normalized and multiplied by -1 to change the initial direction of the fish to swim in the opposite direction of the neighbours.
3. Alignment: Fish search for their neighbours in a radius defined as the Radius of Alignment. The current speeds of all neighbours are summed, then divided by the number of neighbours.
4. Reversal: All of the fish can only swim in a given space, the boundaries of which can be specified. The moment a fish crosses a boundary must be identified. If a fish hits a boundary, then the direction of the fish is changed to the opposite vector (thereby keeping the fish within the defined space). (Source: Unreal 3D)

The Flocking algorithm needed to be adapted to Pelagic and Benthic fishes like Grazers.

• Pelagic fish: Oceanic pelagic fish typically inhabit waters below the continental shelf.
• Benthic fish: Fish that live on or near the sea-bottom, irrespective of the depth of the sea.

As all existing solutions provided by game engine Unity 3D cannot yet integrate this distinction, we rewrote the algorithm to simulate the behaviour of Reef fish like Parrotfish, Clown fish, Surgeonfish. We implemented a Pseudo-Random variant of the algorithm, where the parameters and the target followed by the leader of the fish school change over time, to create a better realm. We applied different sets of parameters and social rules for each species of fish, to account for their social behaviour. The first developed the Algorithm of the Boids on an empty scene, to validate the behaviour of the fish swimming inside the boundaries of a Bounding Box Cube (Fig. 9). The fish continuously watch ahead to check the eventual presence of obstacles that must be avoided, diverging their routes or bouncing back (Fig. 11).

Figure 8: Social behaviour of fish and anticollision (coloured lines).

Figure 9: Principle of the implementation of a Pseudo-Random Flocking Fish Algorithm.

Figure 10 (left): Social behaviour of fish and anticollision (coloured lines)
Figure 11 (right): Behaviour of a herd of fish at the Bounding Box Boundary

As such we have produced a Digital Twin of the type of islands in the Spermonde Archipelago. We can eventually complete the scene with a Character that simulates the presence of a diver or of a marine mammal. Now we are ready to embed the data originated by surveying sampling missions conducted by the 4D-REEF ESRs. These datasets are mainly composed by a set of images and geographically pinpointed locations of coral and sedimentary samples. Their addition to the scene could give a direct overview of the works done on the site, and may be enriched with coral and benthic maps, or results of other custom animations. This first application is meant to be a fully featured baseline for further developments.

Digital Twin 2: 3D Coral Surveying at the Kodingareng Keke island

This Digital Twin can be downloaded here (link).

This second demo concerns the development of a 3D environment built on the bathymetric map gathered during a bathymetric surveying mission held at the island of Kodigareng Keke summer 2022. We conducted the survey using a Multibeam Echo Sounder (MBES), to gather a high-resolution map of the seabed, over the lagoon and the reef slope, on waters not shallower than 1 m, as our surveying boat could not navigate on the 50 cm shallow reef flat. The raw bathymetric Dataset has been produced as 3D dense Point Cloud form, featuring hundreds of points per square inch (Fig. 12). We post processed the Dataset, removed outlier points, and added the 3D Point Cloud in the Unity 3D Gaming Engine, to create a terrain. The Point Cloud shows clearly the reef slope and some sedimentary architectures that would not be visible through underwater or satellite images and may worth further investigations, to help disclosing the evolution of the coral platform. We then developed two different views that will be detailed hereafter:

1. Satellite View: We wrapped the 3D terrain with a satellite orthophotography of the island, to build a realistic model of its emerged and underwater regions (Fig. 13).
2. Bathymetric View: In this latter view, we do not show any orthophotography of the islands, but we assign a scale of different colours for each different height, to give a better view of the depth of the seabed (Fig. 14).

We also added some tools for measuring distances and depth.  The point (1), (2) and (3) at Fig. 15 show respectively the altitude of the island sand cay (51.58 m), the altitude at the middle of the reef slope (51.06 m) and the altitude at the lagoon (41,87 m). These values have been computed above the average altitude of the Reference Geoid used to calibrate the Bathymetric map. Underwater images have been shot at some spotted points along long-shore and cross-shore transects (point (4).

Figure 12: Original Point Cloud at Kodigareng Keke.

Figure 13: 3D Satellite reconstruction of Kodigareng Keke.

Figure 14: 3D Bathymetric reconstruction of Kodigareng Keke.

 

Conclusion

I developed two Digital Twin solutions to help the scientific community develop novel tools for communicating science. We utilized some of the latest technologies, to create 3D realistic scenarios that can be extended for any educational purpose, ranging from 3D viewer to advanced Serious Games with first person view.  I always kept in mind that my product must be able to describe science through visual experiences and images, turning the scientific questions, contributions and results to pleasant interactives stories. Contributing to make science easy and understandable by communities of young students, should help them gathering a better comprehension of the marine life and promote their active participation to ongoing projects of research.

 

Further developments

The program could be extended to other islands of the Spermonde archipelago, to create a wider scenario and simulating the coral and sedimentary transport between islands. The current demo could be turned into a Serious Game based on the construction of a Reef, where the students would build and maintain their own reef.