This blog post is the second of a series on XR Technology for Enterprise. The first post is on NextGen’s website and Medium.
It is worth taking some time to define some of the terms used in the XR field and provide an overview of some of the enterprise-grade hardware used. There is some overlap and confusion when discussing these terms with our clients, and some of the technology might not be what they expect. “Mixed Reality” is a particularly troublesome term, and Extended Reality is a new one for most in the industry.
Virtual Reality
Virtual Reality (VR) refers to a completely immersive experience where a head-mounted display (HMD) covers your eyes, and everything you see is computer-generated. Headphones provide audio to complete the immersive experience. VR HMDs are typically 3D tracked in the real world by external sensors on walls or tabletops. The external tracking system and the connections to the computer limit the area the user has for walking around in the real world, however, you can use hand controllers to interact with the virtual world and to move around an unlimited virtual space.
There are two basic categories of VR equipment: Mobile VR solutions, and PC-based. For Enterprise, I’ll be covering only PC-based VR equipment.
Enterprise-grade VR systems require a separate high-performance computer (HPC) to drive stereo images to a head-mounted display at a high framerate, typically 90 frames-per-second (fps). (Numerous computer workstations and laptops say they are “VR Ready” and that is a good start when looking for a computer to drive VR.) A hard-wired tether connects the Oculus Rift or HTC Vive Pro to the computer with HDMI/DisplayPort 1.2 and USB3 connections. Future HMDs will have a single USB-C connection for power, video, and data that connects to the GPU.
For an un-tethered VR experience, you can either use a backpack VR computer like the HP Z VR Backpack, or a next-generation wireless connection like the VIVE Wireless adapter connected to an HPC desktop computer. There are no wireless systems for laptops at this point, as the VIVE adapter requires a free PCIe card slot on your motherboard.
The VIVE / VIVE Pro and Oculus systems both use external sensors for tracking the HMD and hand controllers. Tracking in other VR systems may be what’s called “inside-out” tracking, where the sensors are built-in to the HMD (see Windows Mixed Reality, below). These WMR devices are less complicated to use and are easy to set up, but conversely, have less accurate tracking.
The HTC VIVE Pro, with the latest external trackers, have sub-millimeter precision. The system supports up to six tracking devices for a much larger free-walking space and has the capability of supporting multiple users at one time.
The HTC VIVE Pro, shown above, is the more professional choice for commercial VR than the Oculus Rift or WMR devices. The Oculus Rift is a less-expensive consumer-grade device, and I’ve had mixed results with the Oculus – many experiences quite negative in fact. Conversely, the VIVE and VIVE Pro have nearly all been good experiences for me. The potential for larger operating space and multiple users makes VIVE Pro the best choice for commercial use.
About 20-25% of VR users experience “VR Sickness,” which is a form of multi-sensory motion sickness. Often, the cause of VR sickness is poor 3D content, poor motion control through the VR environment, and low framerate due to inadequate hardware. Suffice it to say, having people get sick from your VR experience is never a good thing. We’ll cover the causes and some solutions in a future post.
VR typically doesn’t support “pass-through” video, where you can see the outside world. The HTC VIVE includes a camera which allows you to peek at the outside world, but it is intended for momentary use for a quick check on your surroundings. Microsoft is looking to add this to their Windows Mixed Reality offerings as a “flashlight” to peer into the real world. Generally, you don’t want to break the immersion unless necessary.
Mobile VR
In a separate category of VR, and not enterprise-grade, are mobile-VR devices. These small mobile devices aren’t built for large models and complex applications, and won’t be covered in detail at this point. They have their place, and we’ve developed for these, but you need to manage the expectations of your client.
Some mobile VR solutions are complete stand-alone products, like the Oculus Go, and others require you to provide a mobile phone for the computing device, such as Google Cardboard and Samsung Gear VR.
A good indicator as to whether a device is suitable for your purpose is to take a good look at the demonstrations offered for the device. There are reasons these demos on mobile VR are relatively small and low-resolution, and it likely took a lot to get even that low-resolution data to perform at a good framerate. You don’t save anything when development time can be higher, and the experience is less compelling.
Augmented Reality
Augmented Reality (AR) is computer-generated content composited with a video feed onto the display of a mobile handheld device. The user views the 2D image on the mobile display. The device knows it’s position and orientation by using the accelerometers in the handheld device and optical tracking of the real world. The underlying mobile AR system (ARCore on Android and ARKit on Apple) identifies and tracks the real-world surfaces that are seen through the camera, allowing your 3D content to maintains its position relative to those surfaces and objects. AR experiences can be shared and collaborative, with the right software and cloud connections.
Apple iPad devices seemed ahead of Android with tablet device support for AR, but the newer Samsung Galaxy Tab S4 devices and others support native ARCore technology. Previously, you needed a Google Pixel or newer Android phone to get ARCore support. 2D AR on a handheld device it will not generate illness in the user if the display is lagging.
In addition to ARCore and ARKit, PTC’s Vuforia Engine is powerful and supports a wide variety of mobile and PC devices, and is native in Unity 3D. It supports and expands on ARCore and ARKit, and will use them when available. Vuforia supports more mobile devices than ARCore, and we’ve been able to expand our AR offerings by using Vuforia. Technologies like 6d.ai aim to make AR more natural to interact with the real world and include the ability to understand depth, occlude your 3D geometry with real-world objects, and physics.
Head-Mounted AR
AR can also be computer images visible in the lens of eyeglasses or other headgear. HMD AR includes devices like the discontinued Google Glass, the Microsoft HoloLens, or the Magic Leap eyewear. For HoloLens and Magic Leap, the computer-generated imagery (CGI) is a transparent holographic 3D image superimposed on the real world. See “Holographic AR/MR,” below.
Mobile and HMD-based AR are good choices for industrial applications, training, and field maintenance and repair. As someone that has had these responsibilities, I can see the incredible value AR brings. For industrial applications, the AR application has a cloud back-end to read data and then display diagnostic and other useful information on top of equipment in the view. They aren’t generally useful for displaying large models or complicated simulations, although that is an area of much research and development.
Holographic AR / MR
Mixed Reality may refer to an augmented reality-like experience within a self-contained head-mounted device, such as a HoloLens or the Magic Leap eyewear. Microsoft calls the HoloLens a Mixed Reality device and a holographic computer.
With these MR HMDs, you see through transparent lenses into the real world. Computer-generated content is projected onto the lenses so that it appears in your view. Your content appears pinned to real-world objects and surfaces, just as with AR. Magic Leap describes their technology as something that “lets in natural light waves together with softly layered synthetic lightfields.” These are ideal for industrial and enterprise applications, but the manufacturers also push the entertainment possibilities.
The above image from Microsoft is the original HoloLens, which sells for $3,000 for the Development Edition, or $5,000 for the Commercial Suite. An improved next-generation HoloLens is expected in early 2019 and will be priced lower, according to industry insiders. The HoloLens is certified as protective eyewear in North America and Europe.
The sensors on the HMD track your hands and fingers for controlling the experience in your view, and the Magic Leap also includes a 6DOF hand controller.
These devices are self-contained with computers and batteries on the HMD (HoloLens) or in a portable pack (Magic Leap) and do not require an attached HPC. As an “edge compute” device, HoloLens and Magic Leap aren’t built for large enterprise-level applications; all their compute capability is in whatever computing device they carry in or with the unit. The 3D model must fit in available memory and be small and fast enough to display at a fast rate. Large models often require a lot of work to split and create code to manage it in sections that work in the device. It is an evolving technology, and there is a lot of R&D around enterprise AEC applications for holographic devices.
Windows Mixed Reality
“Windows Mixed Reality” — WMR or simply MR — is a term that Microsoft coined to describe a range of technologies. This broad term can lead to some confusion as to what exactly WMR is providing in any one device. In my mind, “Mixed Reality” describes AR, but at this point, WMR devices are VR and not AR. The HoloLens is considered MR by Microsoft, but is not dependent on Windows so is not WMR — it falls into the AR category in my mind.
VR and WMR
Windows Mixed Reality is also a term used for relatively inexpensive headsets that are a Virtual Reality experience – you can’t see your environment through the HMD nor is a pass-through video feed possible as part of the experience. WMR headsets from HP, Lenovo, Dell, Samsung, and Acer, are all in this category. There is no mixing of realities.
These headsets require a Windows 10 PC with “Fall Creator’s Update” or newer, as WMR is part of the operating system. Microsoft has “Standard WMR” and “Ultra WMR” levels of minimum hardware. If your computer doesn’t meet the minimum spec, WMR won’t work at all. If you can’t meet the minimum spec, then you’ll likely get ill from the VR experience, so it is probably a good thing it doesn’t work.
You can go to this Link to get the Mixed Reality PC Check tool from Microsoft and ensure your machine is cable of driving WMR. You can also search on the Microsoft Store from a Windows 10 computer.
New WMR headsets are coming onto the market that includes pass-through video, so I expect the VR-only aspect of these WMR headsets to change rapidly.
This is the report for my main workstation:
You can click the Display Results button and drill-down to the technical details, including the value it was looking for and what it found. It looks at Bluetooth, CPU, Disk Space, GPU, System Memory, the Operating System, System Thermal, and USB. If you copy the results to the clipboard and view in a notepad app, you can see a bit more detail in XML code attached to the results. The XML file is likely only interesting if your hardware or software failed to pass the test.
If you are buying a new machine you can look for mixed reality certification with the “Mixed Reality” badge:
The PC Check tool also has a link to see what headsets are available and more information for enthusiasts.
My older (albeit high-end) AMD W9100 GPU fails the test, as does our ultra-thin HP 4K laptop with a mid-range NVIDIA GPU for external video. The WMR application will not run at all on this hardware. Neither system hardware was listed as “VR Ready,” but the fact that it won’t run at all was surprising.
These WMR VR devices use sensors on the visor to track the environment and optional controllers. Unlike the VIVE or Oculus, these WMR devices do not use external sensors, are easy to set up and use, and using the device is not limited to a small tracked area. With a wireless connection, you could work in unlimited space, if the device can continue to track its position. These sub-$500 systems are poised to be an economical and easy way to get into virtual reality. None of these HMDs list “pass-through video” as a feature as of this writing, but again, it’s coming.
The Samsung Odyssey HMD, in the above image from Samsung’s site, is purpose-built for WMR. This headset has two 1440×1600 pixel OLED displays and is around $500 list price. We use the Samsung Odyssey headset for a lot of VR our needs – it is much more portable than the HTC VIVE Pro when meeting with clients, and for most VR we demonstrate we don’t need the more accurate tracking provided by external sensors. The Odyssey (and others) can do 90fps at high quality, and you only need a decent computer feeding it images.
Pass-Thru Stereo Video for WMR
The current crop of WMR headsets does not sport pass-through stereo video to provide an AR-like mixed-reality experience to the user. That is changing, as Microsoft adds that capability to the Win 10 OS and manufacturers start to add stereo cameras. Just check the specifications of whatever you buy, and ensure it is doing what you expect. The Samsung device, for instance, list “6DOF Camera *2” in its features, which is two cameras that track the environment and hand controllers, and are not stereo pass-through.
The Zed Mini camera is described as the first pass-through stereo camera made for Mixed Reality experiences. It attaches to your WMR/VR goggles and provides video to your application for further processing.
AR and WMR
The term Windows Mixed Reality is also used for what is essentially Augmented Reality but on a Windows PC. If you have a Windows 10 WMR-capable computer, you can point a connected camera at the real world and use the MR Viewer app to find surfaces and display your content. This is Microsoft’s answer to ARCore and ARKit on mobile devices like the Microsoft Surface tablet but works with any connected camera and the right OS.
Above is a tiny T-Rex on my keyboard, as seen through a cheap webcam and the Mixed Reality Viewer in Windows 10.
Extended Reality
XR – or Extended Reality – is an umbrella term that is intended to cover all the above. For some uses, it may just cover AR and MR, but my feeling is that XR is a good overall term to capture the industry.
VR CAVE Systems
A CAVE system (Cave Automatic Virtual Environment) is a walk-in area with multi-wall and floor stereo projections for displaying a VR environment. The system tracks the user’s eyes and hand controllers through an external motion tracking system. The user typically wears wireless active stereo glasses, and the system displays the proper stereo perspective from the one user’s point of view. One or more high-performance computers and high-end GPUs are needed to drive the multiple displays. Active-shutter glasses allow the user to see the alternating 3D images on the displays. The glasses are tracked with high precision, and the wall displays are adjusted and synchronized to match the user’s point of view. There are mobile and fixed-installation versions of the CAVE.
Users of a CAVE system aren’t generally prone to VR sickness, as the user has a continuous frame of reference at the edges of the screens and the floor.
The above image – showing an Autodesk Revit model in an EON Reality VR CAVE system – was typical of the work we did in our CAVE. In this project, we set up buttons to drop the walls to expose the framing and MEP design, peeling back the layers to see the Revit MEP model in-context with the rest of the building. It helped to uncover issues in the design and saved the client from expensive rework on-site.
The 3m cube has short-throw 3D projectors for the three vertical walls and another projector for the floor. Four high-performance Boxx computers with synchronized NVIDIA Quadro 6000 cards were needed to drive this with our geometry-heavy experiences. Originally, the system had one computer and two Quadro cards, but it was unable to display large models with a high framerate and stay synchronized. Once we quadrupled the computers and doubled the GPUs, the system worked very well with large and highly detailed models. The system uses Vicon trackers for tracking the 3D glasses and hand controllers.
Newer systems can be created with Unreal Engine’s new nDisplay technology, with similar (albeit updated) hardware as the EON system. This open us opportunities for less expensive installations than previous CAVEs, however, this is still the most expensive option.
Summary
In this blog post, I gave an overview of the various real-time “XR” technologies you may encounter, and cover some of the overlaps in terminology. This is all rapidly evolving, and at the moment I’m writing this, a new CES is on the horizon, and along with that, a host of new hardware and software to talk about.
For simplicity sake, you’ll see me refer to VR as VR in my posts, and not Mixed Reality or XR unless I mean something not strictly a VR-like experience.
In the next post, I’ll cover more of the details of what you need in GPUs for enterprise-level real-time applications.