The developers of the Collabora company presented the xrdesktop project, in which, with the support of Valve, a library is being developed with elements to interact with traditional desktops within three-dimensional environments trained with 3D glasses and virtual reality helmets. The library code is written in C and distributed under the MIT license. The finished builds are ready for Arch Linux and Ubuntu 19.04 and 18.04.
Currently, Linux already has the means to directly generate virtual reality headsets (Vulkan extensions VK_EXT_acquire_xlib_display for X11 and VK_EXT_acquire_wl_display for Wayland), but there is no support for correct window drawing level in 3D space and screen refresh rate synchronization.
About Xrdesktop
The objective of the xrdesktop project is to develop methods that allow the use of classic interfaces in virtual environments, focused on output on a two-dimensional screen and control using the keyboard and mouse.
The xrdesktop components extend the available window and composite managers with capabilities to use the runtime of virtual reality systems to render windows and the desktop in three-dimensional virtual environments.
In xrdesktop, the idea of integrating into existing desktop environments without the need to launch a specialized composite administrator is promoted separate and allow existing user settings used with a regular monitor to be used with 3D helmets.
The architecture of the project implies the ability to integrate with any desktop, but at the current stage of development, components are implemented to support virtual reality headsets for KDE and Gnome.
For KDE, support for 3D helmets is implemented through the Compiz-like plugin, and for GNOME through a set of packages for GNOME Shell.
These components mirror existing windows to the virtual environment of the 3D helmets in the form of a separate scene or in overlay mode, in which the desktop windows can be overlaid on other running virtual reality applications.
In addition to the representation mechanisms, xrdesktop provides components for navigation and input support using specialized spatial controllers.
xrdesktop, based on information from VR controllers generates input events ordinary, simulating the use of the keyboard and mouse.
Xrdesktop includes several libraries that create window textures for the VR runtime using OpenVR, as well as an API-based system for rendering an entire desktop in a 3D environment.
Since xrdesktop does not provide its own window manager, work is required to integrate with existing window managers (xrdesktop can be ported to any X11 or Wayland window manager).
The main components of xrdesktop:
gulkan: a glib binding for Vulkan, which provides classes for processing devices, shaders, and initializing textures from memory or DMA buffers.
gxr: is an API for abstracting software interfaces to develop virtual reality applications. Currently only OpenVR is supported, but support for the OpenXR standard will be added soon.
libinputsynth: is a library for synthesizing input events such as mouse movement, clicks, and keystrokes, implemented in the form of backends for xdo, xi2, and Clutter.
xrdesktop: a library for managing windows in a 3D environment, a set of related widgets and backends for rendering the scene.
kwin-effect-xrdesktop and kdeplasma-applets-xrdesktop: a KWin plugin for KDE integration and a plasma applet to put KWin into output mode on a 3D helmet.
gnome-shell patchset and gnome-shell-extension-xrdesktop: is a set of patches for Gnome to integrate xrdesktop support and a plugin to change the output to a 3D helmet in Gnome Shell.
The project supports various methods of organizing desktop and window interaction in a virtual environment, which can be used to capture windows, scale, move, rotate, overlay on a sphere, arrange and hide windows, use the control menu, and simultaneously control two hands using multiple controllers.