The New Era of Simulation

In a time already dominated by screens, the idea of a virtual persistent world, like the metaverse, may seem like too much tech for humanity to handle. But rather than seeing digital worlds as a replacement for reality, UCF researchers see them as replications that can be integrated naturally with a technology called digital twin.

The concept, which has been evolving since the ’80s, centers on creating virtual replications of the physical world — in real time. Possibilities include mapping alternate realities of the future, creating time capsules of regional histories and bridging connections to our natural world.

“Digital twin is a transformative technology that holds the potential to improve everything from manufacturing plants to medical procedures,” says Grace Bochenek ’98PhD, director for UCF’s School of Modeling, Simulation and Training (SMST) and the Institute of Simulation and Training (IST).

Before they even enter an operating room, doctors can use digital twins to foresee every outcome of their decisions during surgery. Digital twins can also be scaled to represent entire transportation systems or cities, with constantly updated data and conditions allowing experts to test and prove their concepts before production.

As these virtual worlds come to fruition, there’s a physical place best suited to bring them to life — Orlando, with UCF playing a critical role in this new era of simulation.

“We’re the blinking light, nationally and internationally, for modeling and simulation,” Bochenek says. “We’re a sought out university to help support in workforce and research needs — not just in defense, but within entertainment, manufacturing and transportation. Those things are all merging together [here in Orlando].”

The World Hub for Simulation

Known as the nation’s capital for tourism, Orlando is also the world hub for modeling, simulation and training (MS&T), which is the foundation for digital twin. Orlando’s simulation sector has grown over 60 years and now accounts for $7 billion in regional economic impact and a workforce of 60,000, according to the Orlando Economic Partnership.

Arguably, Orlando wouldn’t be a simulation leader without IST, a 40-year-old research center based at UCF that is the largest of its kind, and its partnerships with government agencies.

Long before Facebook (now Meta) introduced the Oculus headset and made virtual reality (VR) mainstream, government agencies paved the way for MS&T, similar to ARPANET and its evolution to the World Wide Web. Creating a network of persistent, shared virtual experiences is likely going to require government support to prevent one tech giant from maintaining a monopoly.

“Orlando is interesting because we democratize the world in a way that Silicon Valley can’t at the current moment,” says Joseph Kider, an SMST associate professor.

UCF’s decades of MS&T partnership with the U.S. Department of Defense offers a strong example of how private and public collaboration can spark progress across other industries and the commercial space to promote accessible technology. In large part, Central Florida Research Park is the nexus for this growth.

“We’re the blinking light, nationally and internationally, for modeling and simulation.”

Adjacent to UCF’s main campus, it’s a 1,260-acre innovation playground for businesses, government agencies and research centers.

“Things don’t last that long unless they’re delivering [results] and contributing to research and the region,” Bochenek says with pride. With a 25-year career in defense and energy research — including her work with the U.S. Army, U.S. Department of Energy and the National Energy Technology Laboratory — she knows what it takes to make waves nationally.

To continue to push the field forward, the world needs new innovators with a strong academic background, which is why UCF launched the SMST in 2018. Located in Research Park, it’s home to graduate degree offerings and the IST. While technologies like digital twin, artificial intelligence (AI) and machine learning may automate advances, highly trained professionals are needed to maintain a humanistic touch on technology.

“We have one of the highest concentrations of (faculty) expertise in virtual environments, virtual reality, human performance and learning in the world,” says Bochenek, who earned a doctorate in industrial and systems engineering from UCF.

Collaborative Efforts for Impact

Among UCF’s accomplished simulation experts is Agere Chair Professor of Computer Science Carolina Cruz-Neira, a VR pioneer and SMST affiliate faculty member. Cruz-Neira revolutionized the field in the ’90s by helping develop the CAVE system, which allows people to be immersed in a physical space with VR. Throughout her career, she’s spearheaded advancement in simulation, and says she’s excited to use digital twin to take innovation to the next level.

“In a sense, it’s a duplication of reality, but it’s not constrained with the rules of reality,” Cruz-Neira says.

Creating digital twins involves much more than MS&T or computer science researchers. Optics experts are needed, such as UCF Trustee Chair Professor Shin-Tson Wu, whose work with liquid crystal display has revolutionized the hardware for AR and VR technologies. Psychologists provide understanding of human interaction with technology, and philosophers can theorize the ethics behind these interactions. Data scientists analyze information to thwart the “curse of dimensionality,” which UCF Endowed Assistant Professor of Statistics HanQin Cai says is a common problem of having too many data points. And experts across fields from healthcare to the arts can provide nuanced insight into the custom needs of their niche.

“We talk a lot about multidisciplinary processes, but the members of the multidisciplinary teams still think in terms of their own disciplines,” Cruz-Neira says. “We need to train the students to think beyond their own discipline … and truly understand others … so we can create a seamless change.”

To help create the change, UCF is investing $7.5 million in an initiative to expand research around digital twin, with funding allocated for hiring more faculty, evolving curriculum offerings and fostering more collaboration across the university. Cruz-Neira, who is leading the effort with Bochenek, believes UCF is the first university in the U.S. to dedicate a campus-wide integrated effort around the technology.

“Digital twin is a transformative technology that holds the potential to improve everything from manufacturing plants to medical procedures.”

UCF is also receiving external support for digital twin efforts. Last year, the U.S. Department of Commerce awarded UCF $8.8 million to support a digital twin project that’s part of a larger initiative to expand the potential of a semiconductor manufacturing facility in NeoCity, a 500-acre technology district in Kissimmee, Florida. The overall project was one of 22 winners of the Build Back Better Regional Challenge. It will benefit aerospace, engineering, healthcare and other industries nationally while bolstering Central Florida’s economy.

“[Digital twin] can be a game changer for industry,” says Sean Mondesire, an SMST assistant research professor working on the project. “[It can] use data, machine learning and AI in a way that would help drive down costs to make things more efficient.”

For this effort, UCF will replicate NeoCity’s production line to improve supply chain efficiency, increase microchip reliability and productivity, and enable cross-discipline collaboration. Through digital twin, UCF can also determine ways to lower risks and maintenance costs.

The cost benefit, Mondesire says, comes from digital twin’s ability to monitor real-time data and test what-if scenarios to get a glimpse of what can happen in the future, whether it’s one minute or one year from now.

Revolutionizing Preservation

Other faculty at UCF are using digital twin to defy the constraints of reality by traveling to the past.

Lori Walters is a UCF research associate professor whose home unit is the SMST, but her expertise is in a different area than most other faculty at the school: history. She’s interested in digital applications of oral history, the oldest form of cultural preservation. Her work with Kider is transforming the practice with their latest project, MemoryScan.

“We’re dealing with a different generation that has been exposed to immersive technology since their birth,” Walters says. “So text often may not fulfill their particular interest. You have to attract their interest with something [else].”

In its current iteration, MemoryScan provides a virtual drive through a place and time that’s pivotal to the history of Central Florida and the nation: Cocoa Beach during the ’50s to mid-’70s — the early era of the U.S. space program. One of the sites along the tour is the Cocoa Beach Glass Bank, an iconic five-story mid-century building that included the First Federal Savings & Loan Association of Cocoa, and a nightclub and restaurant. The top floor was used for watching launches from Cape Canaveral, which was about 15 miles away.

The building fell into disrepair in the ’90s and was demolished in 2015, but not before Walters and Kider could use a terrestrial laser scanner to capture the site. To restore it to its accurate state during the Space Race, the duo relied on the vivid memories of the late Vivian Lindauer, who worked in the building for many years and remembered the site down to the Barcelona chairs in the lobby.

“Visualization is important, but the human aspect behind it, to me, is even more important,” Walters says. “The recording of how individuals interfaced with that artifact or structure is really important to maintain for future generations so they can learn from the fantastic qualities and also the flaws.”

The effort has received internal support as well as a competitive National Endowment for the Humanities grant. A project like MemoryScan could be expanded to capture the histories of communities of all sizes across the nation. Or it could preserve national landmarks like the Montauk Point Lighthouse in New York, the oldest of its kind in the U.S.

More broadly, digital twin could be used to archive entire cultures, which may become increasingly valuable as the world faces issues like climate change. Digital twin could even be used to investigate solutions to climate change itself.

Last year, Tuvalu, a Pacific country encompassing nine islands, stated it would become the first “digitized nation.” With rising sea levels, the country is estimated to become uninhabitable by 2100. Before then, its leaders aim to create a digital twin of its landscape, as well as digitally preserve songs, stories, historical documents and other aspects of their culture.

“Pictures say 1,000 words so you don’t have to,” Kider says. “And simulations say 1,000 words that pictures [can’t].”

As Bochenek says, this is a transformative technology.

Exploring Virtual Nature

Digital twins also have the power to connect us to the natural world, which is increasingly important as technology becomes even more pervasive, says Maria Harrington, associate professor of digital media. For over 14 years, her research has focused on using technology to explore virtual nature.

“Experiencing the natural world inspires awe, wonder and curiosity. This is how we learned until relatively recently in the modern world,” Harrington says. “We can design tools intentionally to support that intrinsic desire to explore, discover and learn in very natural ways, and connect to something very deep in our evolutionary brains from over 200,000 years of human-environment interactions.”

A few years ago, Harrington and a team of students developed the AR Perpetual Garden app in collaboration with the Carnegie Museum of Natural History to modernize its dioramas. Through photorealistic 3D models and ambient sounds, museumgoers can learn about woodland ecosystems and how deer populations impact them when they become unbalanced. Harrington also studied the impact of the immersive experience versus a printed pamphlet. While both methods yielded learning gains of 40% in 10 minutes, the virtual experience produced more positive associations with the beauty of plant life, which led users to make stronger connections to the information around these objects.

“The power of these immersive tools is that they create highly salient emotional experiences and episodic memories that are anchored in your mind,” Harrington says.

To make the greatest impact, cartoon-like imagery won’t cut it, she notes. High-fidelity graphics are needed to provide realistic models that enhance the information people interpret and replicate real-life experiences.

“Pictures say 1,000 words so you don’t have to, and simulations say 1,000 words that pictures [can’t].”

Her most recent work is the Virtual UCF Arboretum, a digital twin of the 247-acre arboretum on the university’s main campus. The immersive digital twin application provides a virtual field trip for the public to learn about plants and ecosystems found in the Southeastern U.S.

“The natural world is really the highest-level structure in STEM curriculum because it’s physics, it’s chemistry, it’s biology and it’s mathematics,” Harrington says. “But by making those hard science subjects accessible through a field trip, [and even more accessible through a virtual one], it’s an incredibly natural way to learn because we all wish to understand our world.”

In 2020, Epic Games awarded Harrington a $25,000 grant to improve the immersive experience — and potentially broaden its coverage of plants in the Southeast and Northeast. As part of the grant, the Virtual UCF Arboretum is available through the gaming company’s Unreal Engine Marketplace, a digital store for photorealistic 3D visuals and immersive experiences.

When the world isolated during the pandemic, the Virtual UCF Arboretum provided people with an escape and productive distraction. Since then, more than 220,000 plants have been downloaded from the Unreal Engine

In the future, Harrington aims to use digital twins to improve environments in the real world.

“I think we are living in environments that are very hostile, from a sensory-field perspective — not very beautiful and very unnatural. Our sensitivity as organisms in [these] urban environments is largely ignored,” Harrington says. “Digital twins could help us simulate city planning and integrate sustainable urban landscape design to make cityscapes more beautiful and healthier environments for people to thrive, learn and grow in.”

Building a Model Smart City

As a major metropolitan city, Orlando is an ideal candidate to become a model district for the future. The strong simulation industry and roster of UCF faculty who can contribute to statewide solutions will be instrumental in achieving this.

“Orlando has a lot of potential because it has diverse populations, [including tourists], and diverse populations mean diverse needs,” says Natalia Barbour, an assistant professor and member of UCF’s Smart and Safe Transportation Lab (SST). “Having the [opportunity] to study the most efficient ways to accommodate those needs make Orlando a very unique environment in that sense.”

Barbour is among 25 researchers in the lab, including SST director and Pegasus Professor Mohamed Abdel-Aty, who are working to use simulation and digital twin to enhance transportation infrastructures and develop smart cities. The state has already begun improvements in this area, such as launching Brightline, a high-speed rail service, but more needs to be done. Orlando has long ranked among the worst cities for pedestrians and bikers.

Barbour says the group has one of, if not the, largest data sets of vehicle trajectories, with a distinct use of drones to capture data points rather than relying on the traditional method of sensors. Using this data set, researchers are working on simulation methods to find optimal solutions. Another value of this work is that it’s open source, with universities all across the globe leveraging the information to explore other transportation science solutions.

UCF Synthetic Reality Laboratory Co-Director Greg Welch is working on a project that’s another example of how advancements at UCF not only enable progress at the university, but across others. The U.S. National Science Foundation is sponsoring a platform Welch developed to allow researchers to conduct VR studies across the nation, and even the world. It removes the limitation of needing to be near an institution’s physical campus. The project can advance national research that may accelerate MS&T and digital twin breakthroughs.

One of the reasons I came here is because we have so much talent that the opportunity to do something that has a real societal impact, beyond academics, is huge,” says Cruz-Neira.

Making a societal impact may not seem likely with just one initiative at one university. But faculty, like Bochenek and Cruz-Neira, say they believe the digital twin initiative at UCF is the right effort at the right place to make it happen.

“Why UCF? Because we’re standing on 40 years of modeling and simulation,” Bochenek says. “We partner very well in the region. I’m still amazed at the strong collaborations that occur at all levels. And whether that’s state and local government, whether that’s within the university, between different colleges, between multiple industries, there are a lot of layers of engagement.”

It’s an investment that has a domino effect to unleash potential within changemakers across all industries.

“I think we already are, but eventually Orlando will be the smartest city,” Mondesire says.