Augmented Reality Is Getting Real

When every worker in a plant has glasses on—not standard-issue safety goggles but wearable displays—Scott Montgomerie believes industry might look back at this time as the tipping point for adoption of augmented reality on the factory floor.

“2017 may go down in history as the year that augmented reality finally became viable,” explains Montgomerie, “I think the big shift has been in the power and capabilities of the hardware, with all these vendors really throwing their full weight behind it.”

Montgomerie has a first-hand view of that step change as the co-founder and CEO of Scope AR, a San Francisco-based creator of augmented-reality software (scopear.com). Founded in 2011, Scope AR launched its business by working with early-adopting companies like aerospace giant Boeing, eventually creating what it calls a “device-agnostic technology” that can be utilized by smartphones, tablets, and wearable glasses.

 

Mixing Realities

For the uninitiated, talk of augmented reality might conjure visions of unsuspecting victims at a mall kiosk donning oversized black goggles and reacting wildly to a stimulus only they can see. That “immersive experience” is the kind of virtual reality where a headset completely obscures the world around the wearer and drops him or her into a visual and aural environment he/she can interact with.

Along the mixed-reality spectrum, anchored by physical reality at one end and digital reality—think 360° videos—on the opposite end, augmented reality sits toward the middle, adding virtual elements to physical environs. Its auspicious introduction to most of the public came in the summer of 2016 via the game, Pokémon Go.

In a factory, instead of video-game monsters appearing alongside an injection molding machine, augmented reality would use the camera on smart glasses, a smartphone, or a tablet to scan the equipment within its view and add digital markers once the machinery is recognized. Alternatively, a remote technician could view the same camera image from his or her own smart device and mark up the screen, directing the worker in front of the piece of equipment to take action from afar. With video phone calling, that latter scenario could happen today, but minus a key element.

“The differentiator with every other video call is that we’re tracking the augmented-reality coordinates of the technician and allowing you to annotate on the world of the technician,” Montgomerie says. “So if you’re looking at a particular machine and you need to hit a series of switches or buttons, you can actually draw or add arrows on top of those switches and they’ll see marks in the real world. They can walk around it in three dimensions, and it’s like you’re drawing on the real world.”

And those drawings stick. If a remote technician marked up a specific button on the control panel of a molding machine, and then the individual servicing the press walked around the entire machine, when they came back to the control panel, the digital marker would still be there through the view of the connected device.

Scope AR has undertaken case studies, with plastics processors and others, that test its augmented-reality service-call offering, dubbed Remote AR, against Skype, finding that augmented reality sped up the maintenance process by at least 30 percent over standard video calls.

“In the past year or so, we’ve seen huge investments in augmented reality by all the major hardware vendors.”

“Skype would be like having somebody standing behind you while you’re doing a particular task, and he’s telling you what to do but his hands are tied behind his back, so he can’t actually show you what to do,” Montgomerie says. “With our technology, not only are the experts’ hands untied, and he can show you what to do, but he’s also got a magic 3D marker that can make marks in thin air or on top of equipment for you. So it’s just a much more efficient way of explaining and illustrating how to solve problems with machines and an expert technician.”

 

Rethinking the Instruction Manual & Training Video

Beyond remote augmented-reality service calls, where technicians could provide live assistance to equipment customers on multiple continents on the same day without leaving their office, the most immediate impact augmented reality is likely to have on manufacturing will come in how workers are trained and their interactions with equipment.

For Scope AR, this element of its business is called WorkLink, and it involves creation of interactive instructions that could be used, for example, to train employees on the steps for setting up a machine or show them how to address common maintenance issues. Using instruction templates, companies can enter in CAD data for machinery and layer on markers and animations that guide a user through a procedure.

For demonstration purposes, Scope AR has a circuit-breaker box it brings to trade shows. When WorkLink is opened and the camera of a device running the software is pointed at the box, it scans it, and walks the user through how to remove the broken circuit breaker. Arrows show where the box can be opened and the individual circuit breaker itself becomes animated, displaying in a loop how it can be physically removed from the box.

Tasked with replacing a circuit breaker, non-electricians could read from a printed manual with drawn pictures or photos of the elements in question; or these days, they might even seek out a video on YouTube, watching and listening to instructions. With augmented reality, however, the components that need to be manipulated come to life, as animated versions of the tools required walk a user through the task at hand while the task is literally at hand.

“You can create 3D models from manufacturing processes and then overlay them on top of real equipment,” Montgomerie says. “Using animations and a combination of text and voiceovers, you can show a technician what to do with these really intuitive instructions.”

Here, too, Scope AR has undertaken studies comparing the speed and efficiency of a worker using a paper manual versus augmented reality, with augmented reality coming in at 25 percent faster. “The reason they’re able to understand the instructions so much more intuitively is they’re seeing things in front of them versus having to do a mental mapping from their paper instructions.”

 

Upgraded Hardware Powers New Software

When Scope AR started business back in 2011, it was working with Apple’s iPhone 4S, whose single camera was a novelty, and tablets were even more rare. “I think one of the first demos we ever did was on the iPad 2,” Montgomerie says, “which was the first iPad that ever had a camera on it.”

Broader adoption and higher functionality for augmented reality had been limited by the hardware it was working with—at least until 2017. One week after Scope AR spoke with Plastics Technology, the company announced the launch of Remote AR for Microsoft’s wearable augmented-reality headset, the HoloLens, saying it marked the first cross-platform, live-support, video-calling program available for that product, which launched in 2016.

Microsoft’s HoloLens, which is described as a holographic computer and retails for $3000, is a long way from the smartphones and tablets of 2011. Weighing just over 1 lb, it features an inertial measurement unit (IMU)—which includes an accelerometer, gyroscope, and a magnetometer—and four environment-understanding sensors: a depth camera with a 120°-by-120° angle of view; a 2.4-megapixel photographic video camera; a four-microphone array; and an ambient-light sensor.

In terms of wearable tech, there are also Google Glass, Oculus Rift, and others. Apple and Google have both made big commitments to augmented reality in their devices. After acquiring augmented-reality software maker Metaio in 2015, Apple then acquired headset maker Vrvana last November. As part of the iOS 11 launch last September, Apple introduced ARKit—a development platform for creation of augmented-reality apps. For Google, newer Android-powered devices, including its latest Pixel, have augmented-reality capabilities “baked in,” according to Montgomerie.

“That’s really been the catalyst for what we’re seeing now—just the increase in capabilities of the devices,” Montgomerie says. “In the past year or so, we’ve seen huge investments in augmented reality by all the major hardware vendors.”

 

Reality’s Future

Now that capable hardware has arrived, usable tutorials will be required—the proverbial game console needs games—with Scope AR and others stepping in with their creation platforms. “The next hurdle is going to be content creation,” Montgomerie says. “That’s really where we focus.”

Investors and industry will be watching. Immersive-tech virtual reality and augmented reality revenue are expected to reach $150 billion by 2020, according to consulting firm Digi-Capital, with $120 billion of that total coming from augmented reality. According to the PwC/CB Insights MoneyTree Report, $1.5 billion of venture capital was invested in 71 deals in 2016 in augmented and virtual reality.

Much of that money is flowing into enterprise augmented reality for businesses rather than consumers; and in that space, wearable head gear will dominate, leaving workers’ hands-free as they service machinery, for instance. ABI Research expects head-worn devices to make up 90 percent of the enterprise augmented-reality device installed base by 2021.

Whether on people’s heads or in their hands, Montgomerie believes augmented reality will soon be ubiquitous. “I would see right now as being an inflection point, and if it’s not right now, it’s not too far away,” Montgomerie says, noting a spike in real-use cases and real investments by companies. “I think it’s just on the verge of explosion.”