EASY meets mixed reality HoloLens

Mixed, augmented, and virtual reality – all red hot buzzwords which are stirring up the IT industry right now. Trainees and work students at EASY SOFTWARE have been exploring in practice how mixed reality can be used with the Microsoft HoloLens in industry, for example.
What is mixed reality? The term describes a mix of a user’s natural perception with artificial perception. Very specifically, users are able to artificially depict 2D documents or 3D projections in the “normal” world. That is possible with the Microsoft HoloLens. The Microsoft HoloLens is a set of mixed-reality glasses which can be controlled by voice and gesture recognition, as well as head movements.

Practical options for using the Microsoft HoloLens

EASY SOFTWARE quickly initiated a project. Trainees were to investigate and try out options for using the mixed reality HoloLens for industry. Naturally, that appealed to the trainees, and the work students joined in. “The question was how our products EASY Content Web Services and EASY Archive Smart could be combined with new technologies such as the Microsoft HoloLens,” says Andreas Schneider, Manager of the Business Unit ECM, who initiated the project.

The future of technical service with Microsoft HoloLens?

The use case of the trainee project is an elevator service engineer whose maintenance work should be performed completely without physical documents. Specifically, technicians on site can call up virtual documents such as instructions or the construction plans of an elevator directly from an archive with the Microsoft HoloLens via voice control and move them in and out of their field of view via gesture control. Furthermore, 3D projections, blueprints for example, can be displayed as a virtual model without hindering natural sight. The project was oriented on the use case of thyssenkrupp Elevator AG, which is already developing with the HoloLens in the field of elevator service.
The project thus has two aspects: one technical, which raises the question of programming, and one ergonomic, which describes the interface between person and machine and represents the user experience.

Investigate, develop, and gain experience

The team of Jan Vogt, Manuel Fessen, Leon Tambak, Michael Mahr and Daniel Zimmermann worked on the project for a month, concluding with a five-day hackathon. There were a few hurdles to overcome. “Some functions of the HoloLens are still in development. The gesture recognition works, but very rudimentarily,” said Michael Mahr about the results. At the end, there was a prototype with which it is possible to display documents from the EASY Archive Smart by voice control and to browse through the documents. Users can zoom in and out on the documents.

The next step in the project will be the integration of content web service for reading PDF documents, the integration and display of 3D models, and a physical simulation of 3D models. The team sees the potential for EASY in the integration of an AI, in which the AI assistant recognizes and suggests actions, as well as the revision-proof archiving of 3D objects. Also possible are the saving of 3D models after inspection and the access to a history. Virtual conferences with the Microsoft HoloLens could also be useful to EASY. The goal is to create a showcase in order to be able to demonstrate – at trade fairs and customers, for example – what can be achieved with EASY products in combination with cutting-edge technologies.