Unlocking a Better HMI with Spatial Computing & AR

Written By: Nancy White
  • 3/30/2021
  • Read Time : 3 min

Take a closer look at PTC’s logo. Do you see an intertwined ‘P’ and a ‘D’? This is purposeful – the p stands for physical and the d for digital – at PTC, we’re all about how the digital transforms the physical.

As we support businesses in their digital transformation journey, it starts from this foundation. While we are a digital company, our software – from CAD to augmented reality – remains tied to the physical world.

The PTC Reality Lab, our innovation research group, is actively engaged in developing the next iteration of physical-digital convergence. Its newest member, Yuanzhi Cao, as recent PhD graduate from Purdue University, is keenly interested in how to improve the human experience with digital technologies.

“Much of the research I did for my PhD is in perfect synergy with what PTC is doing,” Cao said. His academic work involved spatial computing, a new exciting field combining augmented reality and IoT with artificial intelligence and spatial sensor technology. “The Reality Lab is thinking about the same things I am – in particular, how to apply augmented reality for physical targeting.”

Making technology accessible is a core tenet of his research. This manifests in a few different ways, including accessibility regardless of skill or expertise and leveraging hardware that’s commonly available. He’s currently experimenting with a mobile-based tutoring and skill transfer application.

“Everyone has a smartphone and we’re at the point now that the technology in those smartphones have capabilities to do more with augmented reality,” he says.

For example, a LiDAR (light detection and ranging) scanner, which can determine the depth between the phone and other objects, is a built-in feature of the iPhone 12 Pro.

While PTC currently offers Vuforia Expert Capture, which helps companies create powerful step-by-step AR instructions for frontline workers, his current research project is exploring a possible next-gen version.

The idea is to create and deliver work instructions exclusively via a smartphone using both 2D video capture and 3D augmented reality volumetric data. With depth-embedded video, he is developing a way to segment the human’s actions from the rest of the environment in real-time. Then combine those actions with other relevant data through spatial analytics to create 3D training content. This type of AR content gets much closer to the long-standing training method of on the floor job shadowing that’s regarded as the most effective.

With the complexities of modern industrial machines, especially for multi-step processes or tasks involving several machines within a space, this type of application could be highly effective for training and skills transfer. It would make it possible to do much more in-situ, a primary need for frontline workers.

Another goal with his research is to simplify the development of these work instructions to democratize the responsibility and accelerate the creation of content libraries.

“Augmented reality is a user interface capable of bridging the gap between digital interface and physical reality.” Cao says. With an increasing number of IoT-connected robots and machines working alongside humans – and a strong drive toward constant improvement – factories are an ideal location for this type of research and the collaborative future of work.

Bringing Ideas to Life

While his more recent research has explored spatial computing, he has an extensive background with CAD and engineering. He draws on this engineering skillset, along with art and drawing training, to bring to life ideas and designs.

“CAD is only a software tool; it’s difficult to create something from nothing – the idea must start in your imagination,” Cao says. From imagination, he often will draw a sketch of a design or process as a starting point. “What’s most important to a sketch is that it’s a tangible, visual record of an idea. For me, it’s part of the research process.”

In developing user interfaces – vital to furthering human-machine interaction (HMI) – good design is essential. “The UI needs to look good, feel good, and come natural (to the user),” he says.


At PTC, we’re actively engaged in researching and developing spatial technologies for a variety of industrial use cases. Stay tuned for more ideas and innovations coming out of the PTC Reality Lab. Keep up with the Reality Lab on Twitter: @PTCRealityLab.

Vuforia Spatial Toolbox

Explore the power of IoT, AR, and spatial computing.

  • Augmented Reality
  • Industrial Internet of Things
  • Digital Transformation
  • PTC Reality Lab

About the Author

Nancy White

Nancy White is a content marketing strategist for the Corporate Brand team at PTC. A journalist turned content marketer, she has a diverse writing background—from Fortune 500 companies to community newspapers—that spans more than a decade.