Magically Turn your Desktop Apps into VR Apps

Its 2017 and one of the popular trend in technology apart from  Internet of Things, Augmented reality is Virtual reality. There are plenty of VR Goggles available in the market starting from the VR Cardboard till Oculus Rift . Both types of goggles have apps for them, One from the android market and the other from Oculus rift themself.

Last december I bought a VR cardboard from a local electronic shop. I assembled it and played around for a while with some android apps, It was very interesting. My smartphone got bloated up with apps, and I tried most of the free vr apps in the market. Some days later after each app started updating themself, I removed all of them.

I realised that for any new platform there is always a app store and surrounding developers etc. In a way its good, there are very least effort taken to reuse the existing things. I was looking at space exploration apps and thought it would have been nice to have Celestia seen through VR.

I thought of the idea like this

Now the question is How to do it?  The idea seems to be a  bit complex at the beginning but If you break down the process then it is all about finding the right tool for the right purpose.

First we need to capture the images of an application which is running Linux.  There are many ways to do it, I ran the application inside Xephyr display server . There are many reasons for me to run the App in Xephyr because it completely covers the App inside its environment including option to export keyboard and mouse input.  Next I choose a python module which can take screen shots of the display server. The module I chose was a combination PyVirtualDisplay,   Pyscreenshot, EasyProcess. These modules are used for taking screenshot of the browser for cross browser testing purpose . I luckily got to know these somehow.

The screenshot images are handled are base64 encoding. Now we need to make these images available for a smartphone. This can be achieved using a Data Distribution Service. There are many methods to use but I choose this because this is much easier for me to achieve the goal. I chose Crossbar as my dds. The crossbar is a networking platform for realtime applucation,tt has a static web server inbuilt which can be used to distribute the javascripts to the smartphone.

Now any WAMP client can connect to the Crossbar router and get the images from something called “topics” . The Screenshot module takes the image and publishes it to a topic com.example.image . Now the we can fetch the image from the crossbar by using a smartphone chrome browser. I chose chrome because it has lot of features which I need and need now know about to run my things smoothly .

The crossbar is configured to run the webserver and wamp service on port 8080. If your machine and smartphone are on the same network then you open  chrome browser and type IPAddress:8080. But wait not yet, if you do that you will get only a single image of the screenshot and it is not capable of displaying in VR enabled devices such as VR Cardboard.

When the chrome loads the html pages from the web server, we also load a bunch of javascript for the frontend. The Autobahn is client side module for crossbar router, we need this to receive the images from the topic com.example.image. From there we get the image convert it in to a VR image using ThreeJs library.

 

oooohhoooooh I got that working finally. Now I tried implement keyinput functionality, i.e. to send the gyro activities happening in smartphone to the crossbar. That is done by using a GyroJS library, the resulting values are published to the topic com.example.topic1. If you move left, right forward backward those information will go back to the server and there it will be converted to corresponding keyboard or mouse values and fed back to the display server by using a uinput module. The uinput module is a linux kernal mechanism to create inputs from user space, there is a python module for uinput called python-uinput which has been used.

Now truth to be told, everything is just a prototype and gyro part works poorly. Both javascript, python code needs lot of things to be changed to work well. May this can open a new idea to host the linux application in cloud and view them via smartphone.

You can checkout  the code at

$ git clone https://github.com/codelectron/VRWebNativeApps
$ cd VRWebNativeApps
$sudo apt-get install python-virtualenv

$sudo apt-get install libjpeg8-dev

$sudo apt-get install libudev-dev

$sudo apt-get install xserver-xephyr

$tar -xzf python-uinput-0.10.2.tar.gz # download from the web

$cd python-uinput

$python setup.py build

$python setup.py install

$sudo apt-get install imagemagick

$sudo apt-get install scrot

$sudo apt-get install celestia # Our application
 

$virtualenv localpy

$source localpy/bin/activate

$pip install -U -r requirements.txt
 
 

$modprobe uinput

$chown username.username /dev/uinput # Here use your username

$source localpy/bin/activate # Activate it if you have not activated

(localpy)$cd VRWebNativeApps/src

(localpy)$crossbar start
 

If you like this idea then share it, if you have comments/feedbacks then feel free to comment below.

2 thoughts on “Magically Turn your Desktop Apps into VR Apps

Leave a Reply

Your email address will not be published. Required fields are marked *