If you have been developing for VR long enough you will know the pain of testing using a headset. It’s slow, uncomfortable and just physically demanding when done for 8 hours a day.
Out of necessity a simple idea is born, what if I could test my VR application without having to wear the headset. On every single project I have worked on, either I or someone else has setup a system to control the VR pawn using mouse and keyboard.
As a matter of fact I have implemented such a system in the Hand Tracking Plugin to allow developers to test their application with just mouse and keyboard.
This idea is taken a step further by an XR simulator which simulates an entire XR runtime where we can directly test our VR application.
In this tutorial we take a look at how to work with the Meta XR Simulator in Unreal Engine. We will cover how to do so in two engine versions, as in previous versions, setting up the Meta XR Simulator is not as straightforward as it should be.
What do we need?
There is one thing we need before getting started and that is the relevant Meta XR plugin for the version of the engine we are using.
In this tutorial I cover Unreal Engine 5.3 and 5.5; therefore depending on the version you will be working with you will either need the Meta XR plugin version 67 or 78 (latest at the time of writing).
Meta XR Simulator with Unreal Engine 5.3
When dealing with 5.3 and previous versions the Meta XR Simulator must be installed manually. In later versions of the engine the Meta XR Simulator will be downloaded automatically by the Meta XR plugin when enabled.
Based on the above we want to download and install the Meta XR Simulator in the same folder as the folder where the future Meta XR plugin versions will install it. This prevents us from installing several versions used by different engines.
You can download the Meta XR Simulator from this link.
Make sure to install it under the path C:\Users\USER\AppData\Local\MetaXR\MetaXRSimulator\METAXRSIMULATORVERSION. For example, if your username was MyUsername and the Meta XR Simulator version was 77, the full path would be C:\Users\MyUsername\AppData\Local\MetaXR\MetaXRSimulator\77.
Now that we have the Meta XR Simulator installed launch your VR project in 5.3 and make sure to have the Meta XR plugin enabled. Then go to the project settings and under Project Settings > Plugins > Meta XR we need to set the property Meta XR Simulator JSON File to the path of the file meta_openxr_simulator.json.
This file can be found directly under the installation folder. Therefore the whole path, based the last example, would be C:\Users\MyUsername\AppData\Local\MetaXR\MetaXRSimulator\77\meta_openxr_simualtor.json.
Now that the Meta XR plugin knows where to find the Meta XR Simulator we can enable it. From the editor viewport, click the Meta XR Simulator dropdown on the toolbar, then toggle on the Meta XR Simulator box.
When it comes to using the Meta XR Simulator itself this is all you need. You can now hit Play in VR and start simulating your XR environment.
What you should see is the following.
For context, in the above screenshot, I have switched to hand tracking and this is what my pawn looks like.
Meta XR Simulator with Unreal Engine 5.5
The grass is much greener in 5.5 land, installing and using the Meta XR Simulator is a breeze.
All you have to do is enabling the Meta XR Simulator from the viewport’s toolbar as shown below.
This will automatically cause the Meta XR plugin to start downloading an installing the Meta XR Simulator for you.
Restart the editor after installation for the Synthetic Environment Servers to be discovered
Once the process is finished simply Play in VR and you should see something like this.
For context, in the above screenshot, I have switched to hand tracking and this is what my pawn looks like.
Testing passthrough
The Meta XR Simulator offers a huge amount of features, however, one of the most impressive ones must be testing for passthrough. Using the Meta XR Simulator Synthetic Environment Server (SES) you can fully test as if you were using the headset in passthrough.
Let’s start by setting up the project for passthrough. If your project is already fully working in passthrough you can skip this part.
For passthrough to work we need to go to Project Settings > Plugins > Meta XR and tick the Passthrough Enabled box.
After this navigate to Project Settings > Rendering > Post Processing and set the property Enable alpha channel support in post processing (experiemental) to Allow through tonemapper. This change will require a restart of the editor, grab some coffee ☕ as there will be some shaders to recompile.
Finally we need to enable passthrough in our pawn. In 5.3 here is how you do it.
And in 5.5 here is how you do it.
Finally remove everything you have in your map apart from a directional light.
Launching SES in 5.3
This is where working with 5.3 goes wrong, in theory launching the SES should be as easy as going to the Meta XR Simulator drop down in the toolbar and under Synthetic Environment Server clicking on any of the options.
Unfortunately clicking any of the options does nothing. If we inspect the logs we see that there are two errors that occur any time we try to start a SES.
The errors point out that the synth_env_server.exe and local_sharing_server.exe cannot be found. And sure enough looking into those paths reveals that they don’t exist. The two folders in which the exe files should be found .synth_env_server and .local_sharing_server are instead called synthetic_env_server~ and local_sharing_server~.
To fix the errors we can simply duplicate those folders and rename them.
Now launching the SES should work and you should see the following. One window is the synthetic environment and the other is the server.
Minimise both windows and let them run in the background then Start in VR. You should see the following.
Launching SES in 5.5
As per installing the Meta XR Simulator launching SES is significantly easier and just works.
Go to the Meta XR Simulator drop down in the toolbar and under Synthetic Environment Server click on any of the options.
Minimise both windows and let them run in the background; then, Start in VR. You should see the following.
Conclusion
Now that you have setup the Meta XR Simulator have fun exploring its features, we have not covered any of them here but there is a lot you can do with it.
At the beginning of this tutorial I mentioned how I have implemented my own version of the Meta XR Simulator in the Hand Tracking Plugin. This happened before the Meta XR Simulator was even released, however, even after its release I kept the system in the plugin and developed it further.
The main question I had at the time was, is it still worth developing your own VR testing framework? Based on my personal experience the answer is yes.
While the amount of effort that needs to go into such system should not be underestimated I believe that there are still a lot of benefits in doing so. Let me list some:
- The Meta XR Simulator only works with Quest headsets, your own system may support any VR headset.
- You can implement your own system to work just the way you want. While the Meta XR Simulator is great, there is an initial learning curve to move your pawn using mouse and keyboard.
- You can implement additional features into your own testing framework. An example of this for my own version in the Hand Tracking Plugin was the addition of custom gestures to apply to the hands while testing.
- Your testing framework will most likely end up being lighter. From my own testing the Meta XR Simulator is performant, however, it’s not as light as my testing framework. Each time I start testing with the Meta XR Simulator it takes a few seconds for the whole environment to start up. My own framework has virtually 0 overhead and starts up instantaneously.
- Based on the last point, while the Meta XR Simulator allows to test in a multiplayer environment, I have not found a way to only launch a single instance on the multiplayer instance I want to test. This means that if I am testing with two clients and a server on my local machine I will have 3 Meta XR Simulators running. Performance becomes a problem at that point as your machine may struggle to run 3 unreal engine simulations plus 3 XR runtimes.
I hope you enjoyed this tutorial and see you next time!
