14 June 2017

Setting up a HoloLens project with the HoloToolkit - June 2017 edition

Preface with lame apologies

My apologies for the longest blogging hiatus in the history of this blog, I was very busy with things both HoloLens and not-HoloLens related. Chief culprit though was my HoloLens app "Walk the World", whose development completely captured and absorbed me. Passion for tech sometimes does that with me ;). Then came some work-related issues, but now I finally am up to speed again.

Intro

A consequence of the hiatus is a lot of change (the only thing that changes faster than your average JavaScript-framework-of-the-month is the HoloToolkit) so any HoloLens how-to blog post I'd start with I would have to start with 'how to set up a project'. So I make a little separate post of that subject, thus I can refer to it often. Until the HoloToolkit changes again ;)

File/new project and Getting what you need

First, start a new Unity Project. I have called mine June2017HoloLensSetup.

Then, you open a browser and go to Rafael Rivera's  HoloToolkit download site and you click the top HoloToolkit-somenumber-unitypackage. At the time of this writing the top looks like this:

image

Downlaod and double click the resulting .unitypackage file - Unity will automatically import it into your project. This will give you a window with a lot of checkboxes:

image

This looks a bit intimidating, but just collapse the HoloToolkit node in the list, that will reveal another node "HoloToolkit-test" that you don't need, and you can de-select itimage

And then hit the Import-button. Then very soon, Unity will show a popup:

image

Select "Force Text Serialization". This is very important if you want to be able to easily upgrade the HoloToolkit and be able to check it in into your source control of choice.

The HoloToolkit will now be imported. Well, that was not so hard, wasn't it? This is time to get some coffee as Unity will churn a little on this.

Setting up the camera

HoloLens apps need some settings applied to the camera, and the HoloToolkit folks have taken care of that. So first of all, you delete the Main Camera from your Hierarchy - just select it and delete it.

image

Then, find the HoloLensCamera by entering "Camera" into the search box indicated with a red ellipse, and drag the new Camera into your hierarchy

image 

This is also a good moment to save your scene. Hit CTRL+S (or click File/Save Scenes), enter an name for your scene (I usually take "Main") and you are done with the this part.

Setting a whole lot of properties - the easy way

Now you may not have noticed it, but after importing the HoloToolkit you got an extra extra menu options.

image

This was added because you have to set a whole lot of properties necessary for or beneficial to a HoloLens projects -  scattered all over Unity, and good luck to you determining what you forgot if something doesn't work and where you have to correct that. Those were the bad old days (like a year ago ;) ). Now it's easy.

Clicking "Apply HoloLens Scene Settings" gives you this popup. Make sure everything is selected, then hit Apply. Always hit apply!

image

Next up is "Apply HoloLens Project Settings"

image

Same procedure: make sure everything is selected, hit apply. Unity asks if you want to reload the project - click Yes and wait for the project to reload. Finally, "Apply HoloLens Capability Settings"

image

This is what I always select - my HoloLens apps all tend to use voice commands, all use Spatial Perception (that is, after all, HoloLens' biggest USP) and most of them download some data from the internet. I never use the camera. Make your selections according to your necessary capabilities and hit apply.

Sound should be Spatial!

For some reason this is not part of these nice HoloToolkit setup screens - but if you use spatial sound in your apps (and you should!) there is an extra step. You select "Edit/Project Settings/Audio" like this

image

"Project settings is almost all the way down in a long Edit menu. Now on the right side of the editor, all the way on the top, a new menu has appeared:

image

The Spatializer Plugin is default set to "none", which will give you the default Unity stereo sound - nice, but that's sub-optimal usage of a $3K (or $5K) HoloLens - we want real spatial sound. So set it to MS HRTF Spatializer. This is, by the way, a very important and often overlooked part of immersive experiences. Human beings are 3D aware not only by vision, but also by sound. Think for instance of how you move around in a city, you hear cars coming to you even if they are outside of your vision (and that has more than once saved my life). In your house, there is the hum of the air conditioner, the bubbling of the fish tank filter, the voice of your partner - you don't have to see any one of those, but still you can tell where they are. It all contributes to your 3D awareness - and so does spatial sound contribute heavily to an immersive holographic experience.

So far for this soapbox moment ;).

Some organizing

It tend to create two empty gameobjects - HologramCollection and Managers. The HologramCollection will actually hold Holograms that are part of the application, the Managers object is like a container for global functions, object and standard stuff.

image

Some standard stuff to always add

What (I think) you will always need is an InputManager (the thing that acts as a kind of hub for all input, be it gestures, speech or whatever), SpatialMapping (duh) and some kind of cursor. I usually take the object that is just called "Cursor", there's also "DefaultCursor" and "CursorWithFeedback".

image

The easiest way to find them is once again entering their name in the search box. Now this can sometimes give cause for some confusion. For instance, if you enter "Cursor" in the search box you will get a lot of hits.

image

Only choose the so-called prefabs, the things that get the simple sky blue cube next to it. The same goes, by the way, for InputManager and SpatialMapping. Always take the prefabs.

Finally some stubbornness

The current settings make that stuff is clipped when you came within 85cm of a Hologram. There are good technical and performance reasons for it, yet I am a stubborn *** ** * ***** so I tend to make this 15 cm. image

Simply because I think it's cool to get that close to a Hologram.

A final word on source control

Assuming you use Git, making a good .gitignore is pretty hard. Especially the HoloToolkit contains stuff you would expect not to reside in a source repository, there's all kinds of dlls in it and folders called x86 and x64, typically folders that are part of a build path and should usually be ignored. I've been bitten so many times by just that little piece of the HoloToolkit, especially when upgrading. Therefore a hard-won piece of advice: if you value your sanity, put a .gitignore file in Assets\HoloToolkit containing just these lines:

!*.*
!*

That should always add everything, I think.

You can download the (empty) project for reference, with my .gitignore files (there are no less than three) here.

12 April 2017

Using a messenger to communicate between objects in HoloLens apps

Intro

I know I am past the ‘this has to work one way or the other’ stage of a new development environment when I start spinning off reusable pieces of software. I know that I am really getting comfortable when I am start thinking about architecture and build architectural components.

Yes my friends, I am going to talk architecture today.

Unity3D-spaghetti – I need a messenger

Coming from the clean and well-fleshed out of UWP XAML development using MVVM (more specifically MVVMLight), Unity3D can be a bit overwhelming. Apart from the obvious – the 3D stuff itself - there is no such thing as data binding, there is no templating (not sure how this would translate to a 3D environment anyway) and in samples (including some of my own) components communicate by either getting references to each other by looking in parent or child objects and calling methods in those components. This is a method that breaks as soon as 3D object hierarchies change and it’s very easy to make spaghetti code of epic proportions. Plus, it hard links classes. Especially speech commands come in just ‘somewhere’ and need to go ‘somewhere else’. How lovely it would be to have a kind of messenger. Like the one in MVVMLight. There is a kind of messaging in Unity, but in involves sending messages up or down the 3D object hierarchy. No way to other branches in that big tree of objects without a lot of hoopla. And to make things worse, you need to call methods by (string) name. A very brittle arrangement.

Good artist steal…

I will be honest up front – most of the code in the Messenger class that I show here is stolen. From here, to be precisely. But although it solves one problem – it creates a generic messenger – it still uses strings for event names. So I adapted it quite heavily to use typed parameters, and now – in usage – it very much feels like the MVVMLight messenger. I also made it a HoloToolkit Singleton. I am not going to type out all the details – have a look in the code if you feel inclined to do so. This article concentrates on using it.

So basically, you simply drag this thing anywhere in your object hierarchy – I tend to have a special empty 3D object “Managers” for that in the scene – and then you have the following simple interface:

  • To subscribe to a message of MyMessageType, simply write code like this
Messenger.Instance.AddListener<MyMessage>(ProcessMyMessage);

private void ProcessMyMessage(MyMessage msg)
{
    //Do something
}
  • To be notified to a message of MyMessageType, simply call
 Messenger.Instance.Broadcast(new MyMessage());
  • To stop being notified of MyMessageType, call
Messenger.Instance.RemoveListener<MyMessage>(ProcessMyMessage);

Example setup usage

imageI have revisited my good old CubeBouncer, the very first HoloLens app I ever made and wrote about (although I never published it as such) that basically uses everything a HoloLens can do: it uses gaze, gestures, speech recognition, spatial awareness, interaction of Holograms with reality, occlusion, and spatial sound. Looking back at it now it looks a bit clumsy, which is partially because of my vastly increased experience with Unity3D and HoloLens, but also because of the progress of the HoloToolkit. But anyway, I rewrote it using the new HoloToolkit and using the Messenger class as a working demo of the Messenger.

In the Managers object that I use to group, well, manager-like scripts and objects, I have placed the a number of components that basically control the whole app. You see the messenger, a ‘Speech Command Handler’ and a standard HoloToolkit Keyword manager. This is a enormous improvement over building keyword recognizing script manually, as I did in part 4 of the original CubeBouncer series. In case you need info on how the Keyword Manager works, see this post on moving objects by gestures where it plays a supporting role.

Note, by the way, that I also assigned a keyboard key to all speech commands. This enables to test quickly within the Unity3D editor without actually speaking, thus preventing distracting (or getting funny looks and/or remarks) from your colleagues ;).

 

 

The SpeechCommandHandler class is really simple

using CubeBouncer.Messages;
using UnityEngine;
using HoloToolkitExtensions.Messaging;

namespace CubeBouncer
{
    public class SpeechCommandHandler : MonoBehaviour
    {
        public void CreateNewGrid()
        {
            Messenger.Instance.Broadcast(new CreateNewGridMessage());
        }

        public void Drop(bool all)
        {
            Messenger.Instance.Broadcast(new DropMessage { All = all });
        }

        public void Revert(bool all)
        {
            Messenger.Instance.Broadcast(new RevertMessage { All = all });
        }
    }
}

It basically forwards all speech commands as messages, for anyone who is interested. Notice now, as well, that in the Keyword Manager both “drop” and “drop all” call the same method, but if you you look at the image above you will see a checkbox that is only selected for ‘drop all’. This is pretty neat, the editor that goes with this component automatically generates UI components for target method parameters.

Indeed, very similar to how it's done in MVVMLight

Example of consuming messages

image

Now the CubeManager, the thing that creates and manages cubes (it was called “MainStarter” in the original CubeBouncer) is sitting in the HologramCollection object. This is for no other reason than to prove the point that the location of the consumer in the object hierarchy doesn’t matter. This is (now) the only consumer of messages. It's start method goes like this.

void Start()
{
    _distanceMeasured = false;
    _lastInitTime = Time.time;
    _audioSource = GetComponent<AudioSource>();
    Messenger.Instance.AddListener<CreateNewGridMessage>(p=> CreateNewGrid());
    Messenger.Instance.AddListener<DropMessage>( ProcessDropMessage);
    Messenger.Instance.AddListener<RevertMessage>(ProcessRevertMessage);
}

It subscribes to three types of messages. To process those messages, you can either used a Lambda expression or just a regular method, as shown above.

The processing of the message is like this:

public void CreateNewGrid()
{
    foreach (var c in _cubes)
    {
        Destroy(c);
    }
    _cubes.Clear();

    _distanceMeasured = false;
    _lastInitTime = Time.time;
}
	
private void ProcessDropMessage(DropMessage msg)
{
    if(msg.All)
    {
        DropAll();
    }
    else
    {
        var lookedAt = GetLookedAtObject();
        if( lookedAt != null)
        {
            lookedAt.Drop();
        }
    }
}

private void ProcessRevertMessage(RevertMessage msg)
{
    if (msg.All)
    {
        RevertAll();
    }
    else
    {
        var lookedAt = GetLookedAtObject();
        if (lookedAt != null)
        {
            lookedAt.Revert(true);
        }
    }
}

For Drop and Revert, if the “All” property of the message is set, all cubes are dropped (or reverted) and that’s it, the rest works as before. Well kind of – for the actual revert method I now used two LeanTween calls to move the Cube back to it’s original location – the actual code shrank from two methods of about 42 lines together to one 17 line method – that actually has an extra check in it. So as an aside – please use iTween, LeanTween or whatever for animation. Don’t write them yourself. Laziness is a virtue ;).

Conclusion

I will admit it’s a bit contrived example, but the speech recognition is now a thing on it’s own and it’s up to any listener to act on it – or not. My newest application “Walk the World” uses the Messenger quite a bit more extensively and components all over the app communicate via that Messenger to receive voice commands, show help screen, and detect the fact the user has moved too far from the center and the map should be reloaded. These components do not need to have hard links to each other, they just put their observations on the Messenger and other components can choose to act. This makes re-using components for application assembly a lot easier. Kind of like in the UWP world.

01 April 2017

A ‘roller blind’ animation component for HoloLens applications

Intro

This is a cool little tidbit that I wrote in the cause of a project that required 2D images to be shown in a 3D context. Not everyone has 3D models of everything, and sometimes you just have a schematic drawing, a picture, or whatever 2D thing you want to see on a ‘screen’. That does not excuse you from making a good user experience, and I made this little tidbit to give just that little extra pizzazz to a boring ole’ 2D image in a HoloLens. So you click on a 3D device, out comes a schematic drawing. So it’s 2D in a 3D context, not 2D per se.

Say what ?

It basically pulls an image ‘down’ like a roller blind is being expanded. Typically you ‘hang’ this below a ceiling or the object the image is connected to/needs to clarify. Without much further ado, let’s just show what I mean

Nice, eh? I have the feeling my study is becoming quite a household scene by now for the regular readers of this blog ;).

Setting the stage

Being the lazy b*st*rd that I am, I just made a branch of my previous post, deleted the floating 3D objects, implemented the roller blind as a Togglable, and used the 3D ‘button’ already present in the project as a Toggler. So now I have something to click on and start the animation. I also reused my DynamicTextureDownloader from the post before that to show this image of daffodils in my front garden because that what less work than actually making a texture. Did I mention already I can be pretty lazy at times?

Unity setup

imageWhat we have, initially, is just the button and a floating plane. There are some important settings to its rotation and it’s scaling. The rotation is because we want use see the image head-on. This important, as a Plane has only one side – if you look from it from behind you look right trough it.

The default position of a plane is horizontal, like a flat area. So in order so see it head-on, we first need to rotate in 90⁰ over x (that will put it upright) and then 180⁰ over z to see the ‘front’ side (that used to be the top). Don’t try to be a clever geometrist and say “Hey, I can just rotate it 270⁰ and then I will look at the front side as well”. Although you are technically right, the picture will appear upside down. So unless you are prepared to edit all your textures to compensate for that, follow the easy path I’d say. The picture left shows the result, and the picture below it how it’s done.

 

image

So to the Plane, called RollerBlind, we add two components. First the DynamicTextureDownLoader. Set it’s Image Url to http://www.schaikweb.net/dotnetbyexample/daffodils.jpg, which is a nice 1920x1080 picture of dwarf daffodils on the edge of my front garden

(yeah, I was a bit pessimistic about the ‘return rate’ and the buggers turned out to be multi headed too – so I am aware it’s a bit overdone for the space). Important is not to check the “Resize Plane” checkbox here as that will totally mess up the animation. You have to manually make sure the x and z(!!) sizes match up. So as the image is 1920*1080, horizontal size = 1.78 x vertical size. As the horizontal size is 0.15, so vertical size should be 0.15 / 1.78 = 0.084375. Be aware that a standard plane’s size – at scale 1 = is 10x10m, so this make the resulting picture appear as about 150 by 84 cm. I will never understand why standard shapes in Unity3D have different default sizes at scale 1 – for instance a cube =1x1x1m, a capsule roughly 1x2x1m, and a plane 10x10m – but I am sure there’s logic in that. Although I still fail to see it. But I digress.

I stuck the RollerBlind into a “Holder” and used that to position the whole thing around. I place it 1.5 meters from the camera (same distance as the rotating button) and 70cm below it. Go convert that to feet if you must ;)

image

The only thing missing is the RollerBlindAnimator itself

Code! Finally!

We start with the first part - basically all state date and the stuff that collects the initial data

using HoloToolkit.Unity.InputModule;
using UnityEngine;

namespace HoloToolkitExtensions
{
    public class RollerBlindAnimatior : Togglable
    {
        public float PlayTime = 1.5f;

        private AudioSource _selectSound;

        private float _foldedOutScale;
        private float _size;

        private bool _isBusy = false;

        public bool IsOpen { get; private set; }

        public void Start()
        {
            _selectSound = GetComponent<AudioSource>();
            _foldedOutScale = gameObject.transform.localScale.z;
            var startBounds = gameObject.GetComponent<Renderer>().bounds;
            _size = startBounds.size.y;
            AnimateObject(gameObject, 0, 0);
        }
    }
}

_selectSound is a placeholder for a sound to be played when this thing is being toggled, but the button is already taking care of that, so that’s not used here. Now the roller blind is going to be animated over what appears to be y, but since it’s rotated over x, that should now be the z-axis. So we collect the initial z scale. We also collect the objects apparent y–size. That we get from the bounds of the renderer, that apparently gives back it’s value in absolute values, not taking rotation into account. And then it quickly ‘closes’ the blind so it’s primed for use.

Why do we need to know this size and center stuff?

The issue, my friends, is that a Plane’s origin is at it’s center. So if you start shrinking the scale of the z-axis, the plane does not collapse to the top or bottom, but indeed - to it’s center. So rather than a roller blind going up, we get the effect of an old tube CRT TV being turned off (who is old enough to remember that? I am) – the picture collapses to a line in the middle. In order to compensate for that, for every decrease of scale by n, we need to move the whole thing 0.5*n up.

And that is exactly what AnimateObject does:

private void AnimateObject(GameObject objectModel, float targetScale, float timeSpan)
{
    _isBusy = true;

    var moveDelta = (targetScale == 0.0f ? _size : -_size) / 2f;
    LeanTween.moveLocal(objectModel,
            new Vector3(objectModel.transform.localPosition.x,
                objectModel.transform.localPosition.y + moveDelta,
                objectModel.transform.localPosition.z), timeSpan);

    LeanTween.scale(objectModel,
               new Vector3(objectModel.transform.localScale.x,
                   objectModel.transform.localScale.y,
                   targetScale), timeSpan).setOnComplete(() => _isBusy = false);
}

As you can see I have taken a liking to LeanTween over iTween, as I find it a bit easier to use – no hashes but method chaining, that supports IntelliSense so I don’t have to remember that much names (did I mention I was lazy already?).

The last thing missing is the Toggle methods that you can override in Togglable. That’s not very special and only mentioned here for the sake of completeness

public override void Toggle()
{
    if (_isBusy)
    {
        return;
    }

    AnimateObject(gameObject, !IsOpen ? _foldedOutScale : 0, PlayTime);

    if (_selectSound != null)
    {
        _selectSound.Play();
    }

    IsOpen = !IsOpen;
}

Two final things

We need to tell the toggle button that it needs to toggle the roller blind when it’s tapped.So we set it’s Togglers Size value to 1 and drag the RollerBlind object from the hierachy to the Element 0 field.

image

And the very final thing: this app accesses the internet. It downloads the daffodil image after all. Do not forget to set the ‘internet client’ capability. I did. And spent an interesting time cursing my computer before the penny dropped. Sigh.

Concluding words

I hope I have added once again a fun new tool to your toolbox to make HoloLens experiences just a bit better. I notice I get a bit of a feeling for this – past the ‘OMG how am I ever going to make this work’, now into spinning of reusable components and pieces of architecture. As I said, I was too lazy to set up a proper repo, so I’ve put this in a branch of the code belonging to previous blog post. Enjoy!

21 February 2017

A generic toggle component for HoloLens apps

Intro

The following scenario is one I have seen a lot of times – the user taps on a UI element, and then it and/or a couple of elements need to fade out, disappear, whatever. I suppose every developer has felt this itch that occurs when you basically make something the same the second time around, and you feel there will be a third and a fourth time coming up. Time for spinning up a new reusable component. Meet Toggler and it’s friend, Togglable.

The Toggler

This is a simple script that you can attach to any object that will function as a toggle – a ‘button’ if you like. It’s so simple and concise I just write the whole thing in one go:

using System;
using System.Collections.Generic;
using HoloToolkit.Unity.InputModule;
using UnityEngine;

namespace HoloToolkitExtensions
{
    public class Toggler : MonoBehaviour, IInputClickHandler
    {
        private AudioSource _selectSound;

        public List<Togglable> Toggles = new List<Togglable>();

        public virtual void Start()
        {
            _selectSound = GetComponent<AudioSource>();
        }

        public virtual void OnInputClicked(InputClickedEventData eventData)
        {
            foreach (var toggle in Toggles)
            {
                toggle.Toggle();
            }
            if (_selectSound != null)
            {
                _selectSound.Play();
            }
        }
    }
}

This thing has a list of Togglable. When it’s clicked, it calls the method “Toggle” on all Togglable objects in the list, and optionally plays a feedback sound to confirm the toggle has been clicked.

The Togglable

This is almost embarrassingly simple.

using UnityEngine;

namespace HoloToolkitExtensions
{
    public abstract class Togglable : MonoBehaviour
    {
        public abstract void Toggle();
    }
}

and in itself completely uninteresting. What is interesting though is that you can use this base class to implement behaviours that actually do something useful (which is the point of bas classes, usually. D’oh). I will give a few examples.

A toggleable that ‘just disappears’

Also not very complicated, although there’s a bit more to it than you would think

namespace HoloToolkitExtensions
{
    public class ActiveTogglable : Togglable
    {
        public bool IsActive = true;
        public virtual void Start()
        {
            gameObject.SetActive(IsActive);
        }

        public override void Toggle()
        {
            IsActive = !IsActive;
            gameObject.SetActive(IsActive);
        }

        public virtual void Update()
        {
            // This code to make sure the logic still works in someone
            // set the IsActive field directly
            if (IsActive != gameObject.activeSelf)
            {
                gameObject.SetActive(IsActive);
            }
        }
    }
}

To if Toggle is called, SetActive is called with either true or false and it will make the gameobject that it’s attached to flash in and out of existence.

A toggleable that fades in or out

This is a bit more work, but with the use of LeanTween animating opacity is pretty easy:

using UnityEngine;

namespace HoloToolkitExtensions
{
    public class FadeTogglable : Togglable
    {
        public bool IsActive = true;
        public float RunningTime = 1.5f;
        private bool _isBusy = false;
        private Material _gameObjectMaterial;

        public virtual void Start()
        {
            Animate(0.0f);
            _gameObjectMaterial = gameObject.GetComponent<Renderer>().material;
        }

        public override void Toggle()
        {
            IsActive = !IsActive;
            Animate(RunningTime);
        }

        public virtual void Update()
        {

            // This code to make sure the logic still works in someone
            // set the IsActive field directly
            if (_isBusy)
            {
                return;
            }
            if (IsActive != (_gameObjectMaterial.color.a == 1.0f))
            {
                Animate(RunningTime);
            }
        }

        private void Animate(float timeSpan)
        {
            _isBusy = true;
            LeanTween.alpha(gameObject, 
                IsActive ? 1f : 0f, timeSpan).setOnComplete(() => _isBusy = false);
        }
    }
}

Initially it animates to the initial state in 0 seconds (i.e. instantly), and when the Toggle is called it animates in the normal running time from totally opaque to transparent – or the other way around.

There is a little caveat here – the object that needs to fade out then needs to use a material that actually supports transparency. So, for instance:

image

So what is the point of all this?

I have created a little sample application to demonstrate the point. There is one ‘button’ – a rotating blue sphere with red ellipses on it, and four elements that need to be toggled when the button is clicked – two cubes that simply need to wink out, and two capsules that need to fade in and out:

image

You drag the ActiveTogglable on both cubes, and FadeTogglable on both capsules. In fact, I did it a little bit different: I made prefab of both cube and capsule and dragged two instances on the scene. Force of habit. But in the end it does not matter. What does matter is that, once you have dragged a Toggle script on top of the sphere, you can now simply connect the Toggle and the Toggleables in the Unity editor, like this:

image

Which makes it pretty darn powerful and reusable I’d say – and extendable, since nothing keeps you from implementing your own Toggleables.

The result in action looks like this:

Why not an interface in stead of a superclass?

Yeah, that’s what I thought too. But you just try – components that can me dragged on top of each other need to be just that – components. So everything you drag needs to be a component at minimum, but you want the concrete class to be behaviours. So – you have to use a base class that’s a behaviour too. Welcome to the wondrous world of Unity, where nothing is what it seems – or what you think it is supposed to be ;)

Concluding remarks and some thoughts about 3D interfaces

Remember how Apple designed skeuomorphic user interfaces, that for instance required you to take a book out of a bookshelf? For young people, who never may have held much physical books, that’s about as absurd as the floppy disk icon for save – that is still widely used. But it worked in the real world, so we took that to the digital 2D world, even when it did no longer make sense. Microsoft took the lead with what was then called ‘Metro’ for the ‘digital native’ float design. Now buttons no longer mimic 3D (radio buttons) and heaven knows what.

We are now in the 2007 of 3D UI design. No-one has any idea how to implement true 3D ‘user interfaces’, and there is no standard at all. So we tend to fall back on what worked – 2D design elements or 3D design elements that resemble 3D objects – like 3D ‘light switch buttons’ attached to some ‘wall’. Guilty as changed – my HoloLens app for Schiphol has a 2D ‘help screen’  complete with button.

With my little rotation globe I am trying to find a way to ‘3D digital native design’, although I am not a designer at all. But I am convinced the future is somewhere in that direction. We need a ‘digital design language’ for Mixed Reality. Maybe it’s rotating globes. Maybe it’s something else. But I am sure as hell about what it’s not – and that is floating 2D or 3D buttons or ‘devices’ resembling physical machinery.

Code, as per my trademark, can be found here.

11 February 2017

A behaviour for dynamically loading and applying image textures in HoloLens apps

Intro

After two nearly code-less posts it’s time for something more code-heavy, although it’s still out of my ordinary mode of operation: it’s fairly short and not much code. So rest assured, not the code equivalent of “War and Peace”, as usual ;)

For both a customer app and one of my own projects I needed to be able to download images from an external source to use as texture on a Plane (this is a flat object with essentially only width and height). Now that’s not that hard – on the Unity scripting reference there’s a clear example how to do that. But for my own project I need to make sure I could also change the image (so reload an image on a plane that already had loaded a texture before) and I must also be able to make sure the image was not distorted by width/height ratio differences between the Plane and the image. That required a radical different approach.

Enough talk: code!

The behaviour itself is rather small and simple, even if I say so myself. It starts as follows:

using UnityEngine;

public class DynamicTextureDownloader : MonoBehaviour
{
    public string ImageUrl;
    public bool ResizePlane;

    private WWW _imageLoader = null;
    private string _previousImageUrl = null;
    private bool _appliedToTexture = false;

    private Vector3 _originalScale;

    void Start()
    {
        _originalScale = transform.localScale;
    }

    void Update()
    {
        CheckLoadImage();
    }
}

The ImageUrl is property you can either set from code or the editor and points to the location of the desired image on the web, ResizePlane (default false) determines whether or not you want the Plane to resize to fit the width/height ratio of the image. You may not always want that, as the center of the Plane stays in place. For instance, if the Plane’s top is aligned with something else. If the resizing makes the Plane’s height decrease, that may ruin your experience.

The other first three privates are status variables, the last one is the original scale of the plane before we started messing with it. We need to retain that, as can’t trust that scale once we start messing with it. I have seen the Plane become smaller and smaller when I alternated between portrait and landscape pictures.

The crux is the CheckLoadImage method:

private void CheckLoadImage()
{
    // No image requested
    if (string.IsNullOrEmpty(ImageUrl))
    {
        return;
    }

    // New image set - reset status vars and start loading new image
    if (_previousImageUrl != ImageUrl)
    {
        _previousImageUrl = ImageUrl;
        _appliedToTexture = false;
        _imageLoader = new WWW(ImageUrl);
    }

    if (_imageLoader.isDone && !_appliedToTexture)
    {
        // Apparently an image was loading and is now done. Get the texture and apply
        _appliedToTexture = true;
        Destroy(GetComponent<Renderer>().material.mainTexture);
        GetComponent<Renderer>().material.mainTexture = _imageLoader.texture;
        Destroy(_imageLoader.texture);

        if (ResizePlane)
        {
            DoResizePlane();
        };
    }
}

This might seem mightily odd if you are .NET developer, but that’s because of the nature of Unity. Keep in mind this method is called from Update, so it’s called 60 times per second. The flow is simple:

  • If ImageUrl is null, just forget it
  • If an ImageUrl is set and it is a new one, reset the two status variables and make a new WWW object. You can see this as a kind of WebClient. Key to know it’s async, and it has a done property, that only gets true when it’s downloading. So while it’s downloading, the next part is skipped
  • If, however the WWW object is done, we will need to apply it to the texture, but only if we did not do so before. So then we actually apply it.

So after the image is applied, the first if clause is false, because we have an ImageUrl. The second one is false, because the last loaded url is equal to the current one. And finally, the last if clause is false because the texture is applied. So although it’s called 60 times a second, it essentially does nothing. Until you change the ImageUrl.

An important note – you see that I first destroy the existing Render’s texture, then load the WWW’s texture into the renderer, and then destroy the WWWs texture again. If you are using a lot of these objects in one project and have them change image regularly, Unity’s garbage collection process cannot keep up and on a real device (i.e. a HoloLens) you will run out of memory soon. The nasty thing is this won’t happen soon in the editor or an emulator. This is why you always need to test on a real device. And this is also why I had to update this post later ;)

Resizing in correct width/height ratio

Finally the resizing, that’s not very hard it turns out. As long a you keep in mind the Plane’s ‘natural posture’  is ‘flat on the ground’, so what you tend to think of a X is indeed X, but what you tend to think of as Y, is in fact Z in the 3D world.

private void DoResizePlane()
{
    // Keep the longest edge at the same length
    if (_imageLoader.texture.width < _imageLoader.texture.height)
    {
        transform.localScale = new Vector3(
            _originalScale.z * _imageLoader.texture.width / _imageLoader.texture.height,
            _originalScale.y, _originalScale.z);
    }
    else
    {
        transform.localScale = new Vector3(
            _originalScale.x, _originalScale.y,
            _originalScale.x * _imageLoader.texture.height / _imageLoader.texture.width);
    }
}

It also turns out a loaded texture comes handily with it’s own size attributes, which makes it pretty easy do to the resize.

Sample app

I made this really trivial HoloLens app that shows two (initially empty) Planes floating in the air. I have given them different with/height ratios on purpose (in fact they mirror each other):

image

I have dragged the behaviour on both of them. One will show my blog’s logo (that’s a landscape picture) and one comes from an Azure Blob container and shows portrait oriented picture of… well, see for yourself. If you deploy this app – or just hit the play button in Unity, the will initially show this:

image

If you air tap on one of the pictures you get this:

image

In the second picture, the pictures are a lot larger as they fit ‘better’ into the pane. If you click a picture again they will swap back. The only thing that actually changes it the value of ImageUrl.

Bonus brownie points and eternal fame, by the way, for the first one who correctly tells me who the person in the picture is, and at what occasion this picture was taken :D.

Some concluding remarks

If you value your sanity, don’t mess with the rotation of the Planes themselves. Just pack them into an empty game object and rotate that, so the local coordinate system is still ‘flat’, as far as the Planes are concerned. I have had all kinds of weird effects if you start messing with Plane orientation and location. Not sure why this is, probably things I don’t quite understand that but – you have been warned.

This is (or will be part) of something bigger, but I wanted to share the lessons learned separately, preventing them to get lost in some bigger picture. In the mean time, you can view the demo project with source here.

03 February 2017

Using a HoloLens scanned room inside your HoloLens app

HoloLens can interact with reality – that’s why it’s a mixed reality device, after all. The problem is, sometimes, the reality you need is not always available. For example, if the app needs to run in a room or facility at a (client) location you only have limited access to. And you have to locate stuff on places relative to places in the room. Now you can of course use the simulation in the device portal and capture the room.

image

You can save the room into an XEF file and upload that to (another) HoloLens. That works fine runtime, but in Unity that doesn’t help you much with getting a feeling of the space, and moreover, it messes with your HoloLens’ spatial mapping. I don’t like to use it on a real live HoloLens.

There is another option though, in the 3D view tab

image

If you click “update”, you will see a rendering of the space the HoloLens has recognized, belonging to the ‘space’ it finds itself in. Basically, the whole mesh. In this case, my ground and first floor of my house (I never felt like taking the HoloLens to 2nd floor). If you click “Save” it will offer to save a SpatialMapping.obj. That simple WaveFront Object format. And this is something you actually can use in Unity.

Only it looks rather crappy. Even if you know what you are looking at. This is the side of my house, with left bottom the living (the rectangular thing is the large cupboard), on top of that the master bedroom* with the slanted roof, and if you look carefully, you can see the stairs coming up from the hallway a little right of center, at the bottom of the house.

image

What is also pretty confusing it the fact meshes can have only one side. This has the peculiar effect that at a lot of places you can look into the house from outside, but not outside the house from within. Anyway. This mesh is way too complex (the file is over 40mb) and messy.

imageFortunately – there’s Meshlab. And it’s free too. Thank heavens, because after you have bought a HoloLens you are probably quite of money ; )

Meshlab has quite some tools to make your Mesh a bit smoother. Usually, when you look at a piece of mesh, like for instance the master bedroom, it looks kinda spikey – see left. But after choosing Filter/Remeshing, Simplication and Reconstruction/Simplification: Quadratic Edge Collapse Decimation

imageimage

My house starts to look at lot less like the lair of the Shrike – it’s more like an undiscovered Antonio Gaudi building now. Hunt down the material used (in the materials subfolder), set it to transparent and play with color and transparency. I thought this somewhat transparent brown worked pretty well. Although there’s definitely still a lot of ‘noise’, it now definitely looks like my house, good enough for me to know where things are – or ought to be.

Using this Hologram of a space you can position Kinect-scanned objects or 3d models relative to each other based upon their true relative positions without actually being in the room. Then, when you go back to the real room, all you have to to is to make sure the actual room coincides with the scanned room model – add world anchors to the models inside the room, and then get rid of the room Hologram. Thus, you can use this virtual room as a kind of ‘staging area’, which I successfully did for a client location to which physical access is very limited indeed.

You might notice a few odd things – there are two holes in the floor in the living room – that is where the black leather couch and the black swivel chair are. As I’ve noticed before, black doesn’t work well with the HoloLens spatial mapping. Fascinating I find also the rectangular area that seems to float about a meter from left side of the house. That’s actually a large mirror that hangs on the bedroom wall, but the HoloLens spatial mapping apparently sees it as a reclined area. Very interesting. So not only this gives you a view of my house, but also a bit about HoloLens quirks.

The project showed above, with both models (the full and the simplified one) in it, can be found here.

* I love using the phrase “master bedroom” in relation to our house – as it conjures up images of a very large room like found in a typical USA suburban family house. I can assure you neither our house nor our bedroom do justice to that image. This is a Dutch house. But is is made out of concrete, unlike most houses in the USA, and will probably last way beyond my lifespan

31 January 2017

Scanning physical objects with an Xbox One Kinect to use as Holograms in HoloLens

Intro

imageThere are several demos out there that show obviously scanned models of people or physical objects used in HoloLens applications. I think it was fellow MVP and Dresdener 3D genius RenĂ© Schulte who first used his bust in a demo or at least went public with it. Unfortunately I have not been able to find very much about the actual process used to do the scanning and get to this result, so I have been trying to kludge together a procedure to make a full color 3d scan of a physical object – myself – and show it in a HoloLens.

Shopping list

For this you will need the following hardware:

And the following software:

Setting up the hardware

The Kinect adapter, when you remove it from the box, seems to be quite an intricate contraption of two boxes and three wires, one of them permanently connected to one of the boxes. That box – I’ll call it box #1 - is about the size of a package of cigarettes, and is the power supply. There is a wire with a mains plug that needs to be connected to this box. The other wire – that is permanently attached to box # 1– has a round plug on the other side.That plug goes into box #2 (the one without fixed wire). It’s about the size of an overly thick Mars bar. Now we have one wire unaccounted for – one with a weird squarish plug on one side, and a USB-3 plug on the other side. The squarish part also goes into box #2, next to the round plug coming from the power supply. Now there’s only one hole unaccounted for - on the other side box #1 is another hole – that’s where you need to connect the actual Kinect 2.

Plug in the mains connected to box #1, and connect the USB plug coming from box #2 to your computer’s USB3.0 port. This will start the installation of a couple of drivers, and I seem to recall it also automatically installed the 3D Scan app that you also can find in the store here.

Downloading additional software

If the 3D scan app did not install automatically, you will need to install it from the Windows Store. Also, download and install CloudCompare. Finally download the shader. That’s a zip file – we will need that later in the process. So store that in your downloads folder and unblock it.

The actual scanning

The 3D Scanning app is actually rather straightforward. You can set a few options, and I noticed that you have to fiddle a lot with the settings to get the effect you want. A higher scan resolution give a lower depth, for instance. Also, Kinect sometimes just loses track of the object it tries to track. What you do have to make sure is that there is plenty of light and prevent shadows, because although the actual object tracking is done via Kinect’s magic sensors, the overlaying of color is using the camera and that just needs light. And move slowly. Very slowly.

I scanned myself sitting on a rotating chair and a 120 seconds settings, rotating very very slowly. For this I used the “Kinect Sensor-stationary” setting. When it’s done, it will open the 3D builder app. That usually complains about invalid geometries that need to be fixed. Click the popup, prepare to wait for quite a while, and then finally the 3D object will be ready for the next step

Converting to a format usable for Unity

The annoying thing is that although the 3D builder can save the scanned object as a WaveFront Object (obj) file – that is readable by Unity -  it will strip any color from it. So we will need to take an in-between step. To that effect, don’t save the file in OBJ but in PLY format. Then start CloudCompare, and open the PLY file in that you just created. Hit file/save as and save the file now as an FBX file. That gives you a number of options – I usually just use FBX binary.

imageImporting in Unity and showing the colors

When you import this into Unity and drag the scanned object on the canvas, you will soon notice a few things about me

  • I’m rather large – like statue-of-Roman-emperor-with-overly-inflated-ego large
  • I seem to hang at a random place at a random angle in the sky, even if position and rotation are 0,0,0
  • I am … rather pale.

Getting to let me look less like a roman statue uses requires the special shader. Using my standard folder structure, I added a folder “Shaders” and copied the contents of the UnityVC zip file into it. Net result:

image 

Then in your assets folder, go to the folder where you have imported the scanned model into. In my case that’s App/Models:

image

In that you will find a Material ColorMaterial

image

Select that Material, go into it’s properties in the Inspector over on the right and select either “Standard (Vertex Color)” or “Standard Specular (Vertex Color)”

image

And boom. There I am, in full color color and glory looking like a zombie.

image

Now to make me appear in front of the HoloLens view and not seem like some giant ancient, balding and grey god of… whatever descending from Heaven, I used the following settings on the scanned Hologram:

image

And then you get a more or less life-sized floating ghost/zombie/Borg me-like appearance

image

I had the dubious pleasure of walking around myself, noticing the back part was missing.

Lessons learned

  • First of all – already mentioned in passing, make sure there is enough light, and prevent shadows. This is harder than it sounds. You will notice scanned Holograms tend to look rather pale when created with insufficient light.
  • The handheld setting is way harder to use than the stationary setting. Consider placing objects on a rotary platform rather than moving around it with the Kinect
  • Move or rotate the object you want to scan slowly (or move the Kinect slowly)
  • You will have to fiddle a lot with settings before you get the result you want
  • Larger objects (like humans) are way easier to scan than small objects
  • Make anything you don’t want on the scan as black as and non-reflective as possible

Concluding remarks

Unity will complain regularly about an error in the shader in the editor, but it still seems to work fine. Having no clue about shaders and how to write them yet, I tend to ignore it. The resulting project, although containing no code written by me at all, can be found here.