Turning the physical world into a digital canvas

Whether it is Snapchat using filters to make someone look like a puppy, Lowe's using the Hololens to help visualize a kitchen remodel, or Soldamatic training on welding technologies, augmented reality (AR) is already being used in many applications today. With Apple announcing its plans earlier this month to bring AR to iPads and iPhones and, Facebook’s announcements at their F8 developer conference in April, it is clear that almost every major technology company is involved with augmented reality in a big way. As the technology continues to mature, we will continue to find use cases across both consumer and enterprise customers.

The exact definition of augmented reality is still a moving target as the technology evolves, but at its core, “augmented reality” is used to describe experiences in which digital assets have been added to the physical world, and in such a way that one is able to interact with them.

Currently, there are two technology approaches used to create AR experiences: marker-based and markerless. While the end user doesn’t really need to know much about the technology powering their applications, the differences between marker-based and markerless approaches can affect the user experience, development cost, ease of implementation, training, maintenance, and upgrades, so understanding this is valuable when setting out to create something new.

Marker-Based

With marker-based AR experiences, there is a physical mark/fiducial in the real world that is recognized by the device camera and does two things.

First, when the mark is recognized by the camera, the application knows to trigger the digital experience. These are not unlike QR codes of the past, the difference being that information was only a digital experience, and was not integrated with the physical world in such a way that it could interact with it or be affected by it. Unlike the black and white blocks of the past, today’s AR markers can be branded and designed in such a way that they are easily noticed but also do not stand out and ruin design aesthetics. For example, triggering AR experiences on machinery using ThingMarks, a PTC branded version of Vuforia's VuMark, is an example of marker-based augmented reality.

PTC uses ThingMarks to trigger augmented reality experiences written in Vuforia. In addition to letting the application know to start the AR experience, a marker acts as an anchor to root the digital experience in the physical world, giving it volumetric definition and enabling interactivity. In addition, having a physical marker allows for a much more exact placement of an experience.

 An example of an augmented reality experience that is triggered by and anchored to a ThingMark marker

Markerless

With markerless AR applications, there is no specific visual indicator that triggers the experience. The entire world becomes an interactive canvas on which to paint an experience. The Microsoft Hololens uses an array of sensors and cameras to enable markerless augmented reality applications*. Snapchat uses the camera on someone’s phone to determine facial features and then augment them, another example of markerless AR. Free from the limitations of requiring markers strategically placed ahead of time, markerless AR applications will not only enable more applications, they will enable the user to have more control of their own experience, and get more value/delight out of it.

The algorithms powering this approach are used not only in AR applications, but in robotics as well. It is what allows robots to see and, to some degree, interact with the world around them. The most commonly heard term is SLAM, which stands for Simultaneous Localization And Mapping.

With SLAM and associated algorithms, one can solve issues related to geometric understanding of a scene, location, or environment. What these algorithms do not completely solve is perception, or the ability to understand what one is looking at.

Broadly speaking, the difference between marker-based and markerless applications is whether or not the system knows ahead of time what it’s looking for. As you can imagine, certain applications will benefit from a free-form exploration of the physical world, whereas others can be based off prior knowledge of the working conditions and types of information one would want to access. In my next post, we will dive further into understanding which approach to take for what kinds of applications.

*Note that the Hololens also supports marker-based AR experiences, as the two technologies are not mutually exclusive.
**Note that Vuforia also supports markerless experiences. Features like user defined targets and Smart Terrain on Tango devices are examples.