Multi-Threading in HoloLens: Building a Plugin

We ran into a problem recently where we had some long-running computations that needed to take place, but we did not want it slowing down the user’s experience.  Our approach was to spawn a new thread for the computation and simply alert the user when it has completed.  This, we found to be unexpectedly difficult to do with Unity and Microsoft HoloLens, but we have managed to find a solution.

Threading in Unity

Before getting to our solution, I wanted to discuss a little bit about using threads in Unity in general.

Coroutines:

Can’t we simply put any operation we want to do asynchronously into a coroutine?

A coroutine in Unity is not the same as using a different thread.  All operations done within a coroutine are still performed on the main thread, which in turn could affect your performance. Depending on the type of yield used, your coroutine will simply continue to execute after all the update functions have been called, an amount of time has passed, or specific conditions are met. This execution will still occur on the main thread—certainly not the same as creating and executing a new thread.

Limitations:

Once you’ve decided to use a new thread, what are some of the limitations that you may run into?

The Unity API is not thread safe. This means that you cannot access or modify any objects in your scene, or even API level utilities like Debug.Log. All of the Unity API work must be done on the main thread, and Unity has included checks in the framework to make sure that this is the case. Unfortunately, this also includes creating an instance of a prefab or interacting with the asset database. In order to work with this, both the main thread and the new thread can access the same memory space to modify data within the thread, or for the thread to notify the main thread when it has completed its work. You should just remember to make this data thread-safe to avoid race conditions and collisions.

Examples:

The following is an example of what cannot be done in Unity.

You can see that the threadStart method, which is executed by the new thread, is directly updating the text property of the Unity TextMesh object. When run, this will throw the exception: “set_text can only be called from the main thread.”

The next example will work in Unity, but is not considered thread safe.

Finally, we can use a simple lock to guarantee that only one thread is accessing the shared properties at a time.

Threading in HoloLens

Now that we have discussed how to use threads in Unity we can start to talk specifically about HoloLens. If you try to build the previous examples as a Windows Store App with your environment set up for HoloLens you will get the following errors when attempting to build:

‘Thread’ does not contain a constructor that takes 1 argument

‘Thread’ does not contain a definition for ‘Start’ and no extension method ‘Start’ accepting a first argument of type ‘Thread’ could be found (are you missing a using directive or an assembly reference?)

Error building the Player because scripts had compiler errors

This is due to the fact that Unity is using .NET/Mono framework version 3.5, while a Windows Store App is using .NET version 4.6. The implementation of System.Threading has changed between the two versions in a way that is not backward compatible. Our solution for including .NET version 4.6 features that the HoloLens will use through Unity is by building a custom plugin.

Creating a WSA Compatible Plugin

In Unity

Create a Plugins folder in your Assets directory

Create a WSA folder inside Plugins

In Visual Studio

Open Visual Studio and start a New Project…

Template: Visual C# -> Windows -> Universal -> Class Library (Universal Windows)

Target Framework: .NET Framework 4.6

Project Name: Should end with WSA (will make things easier for later)

Click OK

New Universal Windows Project dialog will be displayed…

Leave the defaults

Click OK

The project just created will be the one used by the HoloLens as the HoloLens supports Universal Windows Platform applications and .NET up to framework version 6.5. Unity, on the other hand, supports only .NET/Mono up to version 3.5. For this reason, we will create a new project to use as a stub by unity and then be replaced with the UWP assembly when built as a Windows Store Application.

In the Solution Explorer; Right click on Solution and select Add -> New Project… from the context menu.

This will be your stub project and will only be used by the Unity Editor.

Template: Visual C# -> Class Library

Target Framework: .NET Framework 3.5

Project Name: Should be the same as the previous project but without the WSA suffix

Click OK

Delete the Class1.cs files from both projects

In the Solution Explorer right click the Stub Project and select Properties

Click the Build tab and change the Output Path to the Plugins directory in your Unity project

Save changes

In the Solution Explorer right click the WSA Project and select Properties

On the Application tab change the Assembly name and Default namespace to match the Stub

Click the Build tab and change the Output Path to the Plugins/WSA directory in your Unity project

Save changes

In the Solution Explorer right click the WSA Project and select Add -> Class…

Name the class whatever is appropriate to your functionality

This is where you will place your plugin code

In the Solution Explorer right click on the Stub Project and select Add -> Existing Item

Browse to the class file that you had just created, but instead of clicking add, click the arrow next to the Add Button and select Add As Link

Since we don’t want to replicate all the same changes in both projects, we have created a soft link to the same file in both projects. Therefore, any changes made in one project will be in both.

If you open your class file from the stub project, or try to build at this point, you will immediately notice the following error:

“The type or namespace name 'Tasks' does not exist in the namespace 'System.Threading' (are you missing an assembly reference?)”

This is due to the fact that the System.Threading namespace is not available in .NET 3.5, but it is available in .NET 4.6; so it is time to use some compiler directives. By default, the UWP Project will have NETFX_CORE and WINDOWS_UWP defined as compilation symbols. Unity also provides a number of symbols defined that can be used to segregate your code.  A complete list can be found here. 

Using this information, we can change:

to

We can now see that if the file is opened from the Stub Project, the using declaration is greyed out meaning that it will not be compiled. But, if we open the same file from the UWP Project the declaration appears normal.

Make sure your class declaration is marked as public and Build your project

Go back into unity

In the Project window, browse to your Plugins/WSA directory

Click on the DLL

In the Inspector…

Select platforms for plugin: only WSA Player should be checked

Platform settings: set the placeholder value to the stub .DLL in the Plugins directory

Click Apply

Browse to your Plugins directory

Click on the DLL

In the Inspector…

Select platforms for plugin: uncheck Any Platform and check Editor and Standalone

Click Apply

Now, when you return to your Unity scripts, you should have access to any code included in your plugin assembly. Any updates made to your plugin will have to be made and recompiled from Visual Studio, before they will be available from your Unity scripts.

Threading Plugin for HoloLens

Now that we have created a plugin that can be used in both Unity and HoloLens, we can implement some threading functionality within the plugin. The following is a very simple start of a thread manager plugin.

Since the NETFX_CORE symbol is only loaded for the UWP project, it will not be utilized by the Unity Editor. Code running within Unity will utilize System.Threading.Thread while the HoloLens will use System.Threading.Tasks.Task. To use this plugin from our Unity scripts, you can see the changes required for our previous example below.

becomes

Conclusion

Using threads within a HoloLens app is definitely possible. There are a few extra steps involved, and as always when writing a multi-threaded application takes some careful consideration. But, as long as you remember not to use the Unity API from within your new thread and alert the main thread of changes in a thread safe way, it can be very useful.

Rob Gotschall