Visualizing a Room Mesh with HoloToolkit for Unity

In today's video we leverage the HoloToolkit-Unity library (available on GitHub) to enable a visual mesh that represents a room to be rendered in the HoloLens emulator. Since the emulator doesn't actually visualize a real room around you (since you're not actually inside one), this can help you to "see" where you are in this virtual space. We also use the Cursor Manager and Gaze Manager from this toolkit to display a cursor in the device for users to select and interact with holograms.

https://www.youtube.com/watch?v=VknKYTWMomE

Visualizing a Room Mesh with HoloToolkit for Unity: Transcription

00:06.890 --> 00:11.310 Hello again, everybody this is Josh Morales from Falafel Software and I want to thank you for joining 00:11.310 --> 00:15.240 me for another developer video on Unity and HoloLens. 00:15.240 --> 00:20.190 In this quick video I wanted to take a brief look at the HoloToolkit available on GitHub. 00:20.190 --> 00:25.560 Now I had mentioned this toolkit before but only the context of the project that they gave us at 00:25.560 --> 00:30.420 Build which contained a lot of these assets. But all of the previous projects 00:30.420 --> 00:35.100 that I've built have used that particular copy of the toolkit and 00:35.100 --> 00:38.570 what I wanted to do is kinda move forward by making sure that 00:38.570 --> 00:43.010 I always have access to the latest version of that toolkit. 00:43.010 --> 00:48.700 I figured the best way to do that was just go ahead and get a copy of the repo and clone it 00:48.700 --> 00:53.450 into my system so that... I don't expect to be doing any kind of pull requests 00:53.450 --> 00:58.640 as the Unity assets are quite complex... but at 00:58.640 --> 01:00.100 the very least I'll always be able to have 01:00.100 --> 01:04.600 the latest versions and I can kinda merge those in as I need them or just always make sure 01:04.600 --> 01:09.750 that I'm up to date. So if you haven't looked at this toolkit 01:09.750 --> 01:12.070 I definitely encourage you to go and check this out; 01:12.070 --> 01:16.050 it is freely available on GitHub and it contains 01:16.050 --> 01:21.030 a massive assortment of assets... a lot more than I even realized. 01:21.030 --> 01:26.000 There's a great collection of utilities and descriptions of what they do... billboard here 01:26.000 --> 01:30.390 will show a hologram so that it's always facing towards the camera and that's definitely a 01:30.390 --> 01:33.890 useful utility to have, if you need to have something presented to the user like 01:33.890 --> 01:37.600 a heads-up display or something similar... So 01:37.600 --> 01:41.810 yeah there's a whole wide assortment of things in here. Now the input managers 01:41.810 --> 01:44.770 are really important and we'll be using these a lot more, as 01:44.770 --> 01:49.180 these are responsible for allowing the user to see... most importantly 01:49.180 --> 01:54.120 a cursor in front of them that will represent their gaze so that they can select 01:54.120 --> 01:56.020 things by, of course, looking at 01:56.020 --> 02:00.200 them through the device and having a cursor to select them. And it takes care of interacting with 02:00.200 --> 02:04.250 a holograms in the world as well as the spatially mapped 02:04.250 --> 02:07.860 room around you. So that's really handy to have 02:07.860 --> 02:11.730 so that it can recognize things like gestures and, 02:11.730 --> 02:15.610 pass those on to your objects so that your users can interact with them through the HoloLens. 02:15.610 --> 02:20.010 In addition, obviously there's a spatial mapping component which is going to be key to 02:20.010 --> 02:23.580 the HoloLens device being aware of the room... 02:23.580 --> 02:26.890 of the world around it... As well as the collider which is going to 02:26.890 --> 02:30.630 be especially important as we move forward into really 02:30.630 --> 02:35.610 creating holograms that interact with the world. Especially with the next video that I'm planning 02:35.610 --> 02:40.510 in which we adapt the existing Unity tutorials 02:40.510 --> 02:45.110 so that they run in the HoloLens environment. And rather than kind of existing in 02:45.110 --> 02:49.840 this kind of pseudo three-dimensional game that you're looking into as a player, 02:49.840 --> 02:53.590 they will be superimposed in the world around you. And of course the 02:53.590 --> 02:57.360 spatial mapping collider is going to be key to that. 02:57.360 --> 03:02.310 So there's definitely a lot of stuff going on here. Plane-finding it's also important 03:02.310 --> 03:06.450 which ... I only briefly looked at this... Still a lot of information to 03:06.450 --> 03:09.740 absorb... but it's helpful in a allowing you to identify 03:09.740 --> 03:14.180 surfaces like tables and floors and walls, so that you can actually attach things to 03:14.180 --> 03:16.840 them rather than just placing them 03:16.840 --> 03:21.950 at a specific coordinate. So we'll definitely be looking a lot deeper into all of 03:21.950 --> 03:26.350 these utilities. So as you can see there's a whole lot of them; 03:26.350 --> 03:29.120 that's definitely worth your time to investigate them. But 03:29.120 --> 03:32.650 that been said I definitely want to go ahead and make use of 03:32.650 --> 03:37.240 these very quickly just to see how easy it is to create a new project, 03:37.240 --> 03:42.160 reference these assets, and be able to kind of have that workflow 03:42.160 --> 03:48.680 going forward so that I always have a project ready to go with these assets and everything needed for the HoloLens. 03:48.680 --> 03:55.030 So the first thing I want to do is... I've already cloned the repo here, and have copy of the latest version. What I want to do now 03:55.030 --> 03:59.780 is proceed to Unity, specifically the version for 03:59.780 --> 04:10.050 HoloLens, and I'll create a really simple project, and just call it "Hello Toolkit". 04:10.050 --> 04:14.810 Since we are creating this project with the toolkit assets I'm just going to go ahead and just clear out 04:14.810 --> 04:20.060 the world from any of the preloaded assets. So rather 04:20.060 --> 04:24.260 than importing the assets here one by one and 04:24.260 --> 04:26.910 there also is not a custom package of the 04:26.910 --> 04:31.850 assets that I'm aware... I think the easiest thing to do is going to be to just go 04:31.850 --> 04:37.960 into the folder of the assets from the toolkit 04:37.960 --> 04:41.460 itself and copy the toolkit 04:42.550 --> 04:44.660 contents over to 04:44.660 --> 04:47.690 my project. So I'll just leap into my 04:48.870 --> 04:52.170 Hello Toolkit project in the assets and paste those 04:52.170 --> 05:02.410 in. So now when I flip back to Unity I should see those assets being imported. 05:02.410 --> 05:06.120 I think I may not have needed to copy the whole thing; I'm not sure 05:06.120 --> 05:09.090 if the meta files would be regenerated by themselves by the 05:09.090 --> 05:14.430 toolkit... But whatever the case is, here is our toolkit with all of the assets. So, 05:14.430 --> 05:19.060 what we want to do first is obviously place the camera and that is in the 05:19.060 --> 05:23.430 "Utilities" under "Prefabs..." Yes... So here's the main camera; 05:23.430 --> 05:28.330 we'll go ahead and add that to the world, and as we expect, we get the camera preview with the ... 05:28.330 --> 05:31.250 the inspector here... solid color clear flags and the background black. 05:31.250 --> 05:35.260 Perfect. So the next thing that I want to 05:35.260 --> 05:38.760 do is add the spatial mapping component. 05:40.880 --> 05:46.150 I think in the prefabs... yes.. So here we got the spatial mapping prefab, and I'm going 05:46.150 --> 05:51.830 to go ahead and add this to the world as well. Now what this is going to do is, 05:51.830 --> 05:56.200 on launch, map the world in which the HoloLens resides, so 05:56.200 --> 06:00.840 that information is available to the application. Now, by default 06:00.840 --> 06:05.470 it looks like the "Draw Visual Meshes" is enabled. Which we want in this case as I definitely want 06:05.470 --> 06:09.860 to be able to see them. I think in the examples and documentation that I have 06:09.860 --> 06:14.450 read so far, this kind of gets toggled on and off as 06:14.450 --> 06:19.060 a needed for placement of objects in the world around you. So if you are, 06:19.060 --> 06:22.280 especially in the emulator, where you're not actually in a real 06:22.280 --> 06:26.430 world, you can see these meshes or rather... The visual 06:26.430 --> 06:31.700 representation of those meshes is not as important, then I would imagine that would just be left off 06:31.700 --> 06:35.630 by default. But in the emulator, you definitely want to be able to see where 06:35.630 --> 06:40.020 you're placing an item. So that's why we've got this enabled by default, and it's going to 06:40.020 --> 06:44.980 render this material. Let's see, that is located here. 06:44.980 --> 06:51.060 So, you can see here that we just got a simple wireframe material that will be used to render the 06:51.060 --> 06:56.170 Spatially mapped world that the HoloLens sees. So, 06:56.170 --> 07:00.680 that being said, I think the last thing I want to do is, I also want to go ahead and try 07:00.680 --> 07:06.350 to show the cursor. So that is under the input section 07:06.350 --> 07:08.700 and what I want to do is show the cursor, here. 07:13.160 --> 07:15.530 And, so we have a cursormanager 07:16.740 --> 07:22.380 and we have obviously... adds the "cursor on" hologram which is a 07:22.380 --> 07:24.330 ring here that you can hopefully 07:24.330 --> 07:28.130 see... and then the "cursor off" hologram which uses 07:28.130 --> 07:31.160 a material to kind of show a different icon. So you 07:31.160 --> 07:35.560 can see it differentiates when you're actually on a hologram versus when 07:35.560 --> 07:40.770 you're not on a hologram. And you'll be able to see which items you can actually interact with. Now 07:40.770 --> 07:45.050 I believe in order for us to actually be able to track where this is 07:45.050 --> 07:50.490 going, we do need to have a "Gaze Manager" on the 07:50.490 --> 07:51.840 project. 07:51.840 --> 07:54.510 So, what I'm going to do is I'm just going to add a quick empty 07:54.510 --> 07:58.390 gameobject, and add it here to top. 07:58.390 --> 08:06.350 And, I think I want to add the Gaze Manager. 08:06.350 --> 08:11.720 What this does is, if we look at the scripts for the cursor... 08:11.720 --> 08:17.050 The script is a cursormanager. It actually, let's see... I thought 08:17.050 --> 08:19.840 that it referenced the Gaze Manager here... 08:21.530 --> 08:24.550 Ah yes, here we are. So you can see here, 08:24.550 --> 08:27.930 in the preview of the script, this actually does 08:29.020 --> 08:31.400 reference the Gaze Manager instance in order to determine 08:31.400 --> 08:35.940 where the user is looking. So in order for that work we do need to have the Gaze Manager present. 08:35.940 --> 08:40.400 So I'm just going to go ahead and add this to the Game Object so that it is in our world 08:40.400 --> 08:45.790 and accessible. I'm going to go ahead and save this scene we'll just 08:45.790 --> 08:49.770 call this, again under the assets, we'll just call it 08:49.770 --> 08:56.690 "Hello Toolkit Scene." And then finally, I think we're ready... Oh you know, what I think just for good measure I'll go 08:56.690 --> 09:02.450 ahead and add just a cube. So there's a little something to look at while this is in the world 09:02.450 --> 09:07.890 besides just the mesh that gets hopefully rendered. And I'll go ahead and give it a little bit of 09:07.890 --> 09:12.660 rotation just so that looks interesting, and also in preparation for the next video where we'll be 09:12.660 --> 09:16.610 using this cube a lot more. 09:16.610 --> 09:21.110 So, we'll save that again and go to the build settings. So there's one thing... 09:22.350 --> 09:26.720 Ok so I'll go ahead and select the store, switch to that platform, configure it 09:26.720 --> 09:27.880 for Universal ten 09:27.880 --> 09:32.720 and Direct3D and build the project... and of course finally add the open 09:32.720 --> 09:35.060 scene for the holotoolkit. Now, 09:35.060 --> 09:40.040 there's one thing that we have to remember to do, and this keeps getting me every time 09:40.040 --> 09:44.950 I'm tinkering with the Unity HoloLens... and that is that we need to adjust 09:44.950 --> 09:47.720 the player settings for the store 09:47.720 --> 09:53.570 to enable the virtual reality component. And that's under again, 09:53.570 --> 09:56.710 the player settings - the store icon, 09:56.710 --> 09:59.570 under "Other Settings." 09:59.570 --> 00:01.710 And, we need to make sure that we check "Virtual 00:01.710 --> 00:06.420 Reality Supported" and that windows holographic is shown. 00:06.420 --> 00:10.330 If we don't do this, as we saw the previous video, rather than being 00:10.330 --> 00:16.320 immersed in the 3D world, this will render as 2D app in this in that kind 00:16.320 --> 00:19.950 of XAML container as regular 2D XAML apps load. 00:19.950 --> 00:25.050 So, you always want to make sure that you set this up, so that it is the 00:25.050 --> 00:26.180 3D world you expect. 00:27.450 --> 00:31.060 Besides that, there is an additional setting under 00:31.060 --> 00:35.840 "Project Settings - Quality" for the Windows Store option here that 00:35.840 --> 00:40.650 the default should be fastest. And this is again from the documentation, 00:40.650 --> 00:44.690 and we looked at that in the previous video. So now we 00:44.690 --> 00:52.230 have everything ready. I believe we can go ahead and proceed to build this, so let's go and do that. 00:52.230 --> 00:55.080 We'll select the folder to contain the projects and go. 01:00.850 --> 01:06.960 Great, so our project is built; we'll go ahead and open this up in Visual Studio. 01:06.960 --> 01:09.310 And, as always we'll make sure that we 01:09.310 --> 01:12.100 flip our build settings to be x86, 01:13.430 --> 01:17.460 and select the emulator from the device list. 01:17.460 --> 01:18.770 Go ahead and build project, 01:19.880 --> 01:25.220 and, finally launch it in the emulator. 01:25.220 --> 01:29.760 All right so let's see what happens. 01:29.760 --> 01:31.020 There is our cube ... 01:32.140 --> 01:36.510 And, if we pull back from it... 01:36.510 --> 01:38.980 First of all, we can see that our cursor is now 01:38.980 --> 01:43.530 visible. And we can see as we move from being on the hologram to 01:43.530 --> 01:47.400 being off the hologram that we definitely get a different 01:47.400 --> 01:52.310 appearance. So that definitely worked. Great. So, what I don't 01:52.310 --> 01:57.280 see is the spatially-mapped world around me. So I think I may have missed a step 01:57.280 --> 02:02.780 somewhere. So, at least we got halfway through let me go back to my project and see if I can identify 02:02.780 --> 02:06.480 what I might have done wrong. 02:06.480 --> 02:08.880 And, just to be sure. 02:08.880 --> 02:12.930 Lets... you know what? Better idea would be to open the device portal... 02:12.930 --> 02:19.010 Make sure that I actually am looking at a world. So in this case I don't 02:19.010 --> 02:22.650 see actually anything around me. I have updated... I've got 02:22.650 --> 02:29.480 the "Show Mesh" ... Oh there it is; there's is world. Okay so I did move away from it. 02:29.480 --> 02:33.780 So, there's my world and if I go back to the emulator 02:33.780 --> 02:38.110 I should actually be seeing that mesh in front of me and I do not. 02:38.110 --> 02:40.060 So, as I move closer, 02:40.060 --> 02:45.060 you can see... One interesting thing I kinda noticed, that I thought was worth pointing out, 02:45.060 --> 02:49.720 is that you can see as I step further away that the cursor is no longer 02:49.720 --> 02:51.830 recognizing the hologram. And 02:51.830 --> 02:58.830 that is because in the configuration for the Cursor Manager, 02:58.830 --> 03:03.030 we can see that our "Max Gaze Distance" is fifteen, fifteen units. 03:03.030 --> 03:07.270 And, apparently at this distance we're not close enough. As we 03:07.270 --> 03:11.150 get a little closer though, you'll see that... the cursor does indeed 03:11.150 --> 03:15.260 change to match the expected behavior of being in 03:15.260 --> 03:18.570 range of that hologram. So that's an interesting thing to note. 03:18.570 --> 03:21.410 However, as I look at the 3D view I do see that 03:22.510 --> 03:26.770 although this thing is following me, I do not see the world. So I'm going to go ahead and 03:26.770 --> 03:32.550 stop the debugger and I'm going to see if I can figure out what is happening 03:32.550 --> 03:37.500 with the mesh... spatial mapping. So I'm going to go ahead and pause this, take little bit 03:37.500 --> 03:44.020 of investigation and I'll report back when I have identified the problem. So standby, thanks. 03:44.020 --> 03:44.470 Okay, 03:44.470 --> 03:49.340 so, after a little bit of investigation 03:49.340 --> 03:54.840 a bit more reading a lot more tinkering I was able to discover what I did wrong. 03:54.840 --> 03:59.550 At first, I thought there was something wrong with my spatial mapping component... either that I was missing 03:59.550 --> 04:05.770 a script, or a supporting script was not placed in the scene. But 04:05.770 --> 04:09.140 indeed the "visual meshes" was checked which is the only requirement 04:09.140 --> 04:13.530 for getting the spatial mapping to do its function. But in order for the 04:13.530 --> 04:17.280 application to actually do it, there is an additional property in the player 04:17.280 --> 04:22.300 settings that needs to be set. And before I go into it's definitely a 04:22.300 --> 04:27.360 stark reminder, for me at least, that because I'm not starting from that 04:27.360 --> 04:29.330 base project that was provided at Build, there 04:29.330 --> 04:34.610 is a lot of things that I need to be aware of, that need to be changed 04:34.610 --> 04:39.720 for a project that's opened, or created, rather, in Unity fresh. Because there's 04:39.720 --> 04:44.370 a lot configurations that are required for the HoloLens are not set by default. 04:44.370 --> 04:48.970 In this case the setting that we need is actually a capability, very similar to 04:48.970 --> 04:53.860 the app manifest here in Visual Studio for store apps, where you've got, you know, a 04:53.860 --> 04:57.250 list of capabilities that you need to enable. There is 04:57.250 --> 05:02.200 the corresponding list of the same type of capabilities in Unity. 05:02.200 --> 05:07.020 And the one that we need specifically is "Spatial Perception." Because this was not checked 05:07.020 --> 05:08.020 I was not able to 05:08.020 --> 05:13.260 use it. I didn't get any error message or any kind of indication that anything was wrong 05:13.260 --> 05:18.590 or missing or misconfigured. But doing a bit of research I did locate 05:18.590 --> 05:24.230 here, in this document "Spatial Mapping in Unity" which 05:24.230 --> 05:27.600 does specify that in the player settings that we do need to have the spatial 05:27.600 --> 05:32.580 perception available in the capabilities list. Now interestingly enough, I 05:32.580 --> 05:36.280 did not find this link... I only 05:36.280 --> 05:40.590 found this link by doing some searches 05:40.590 --> 05:45.680 about spatial mapping and mesh rendering of the room... 05:45.680 --> 05:49.180 that specifically linked to this particular page. However when I went 05:49.180 --> 05:54.420 into the documentation, especially in the sections of Unity, there really isn't anything linking 05:54.420 --> 05:59.900 to that page. And looking through the different available items in this 05:59.900 --> 06:02.980 table of contents thing, I really didn't find 06:02.980 --> 06:06.800 that specific that specific page. And this isn't the first time 06:06.800 --> 06:10.420 that happened. However, on the right I did 06:10.420 --> 06:14.670 discover the site index. So by clicking "See all pages" I did discover there's 06:14.670 --> 06:20.270 a whole lot more information here than is linked in that first 06:20.270 --> 06:25.120 table of contents. So I definitely encourage you to take a look at this, and go down the 06:25.120 --> 06:29.440 line and fill in a lot of the gaps that you probably, like 06:29.440 --> 06:32.550 I certainly was missing... And 06:32.550 --> 06:37.110 that specific page is linked here "Spatial mapping in Unity." So, 06:37.110 --> 06:40.550 by enabling that here, in the 3D view now we can see that we 06:40.550 --> 06:46.010 are looking on into our room. And actually I'm a little bit, I think, behind it. But 06:46.010 --> 06:51.290 as I switch now to the emulator, you can see here we do render the 06:51.290 --> 06:58.260 mesh outline of the room. So now as I move around with the Xbox 06:58.260 --> 07:01.990 controller I am now in and out of the room. 07:01.990 --> 07:06.760 And, we can see some of the different objects on which you can 07:06.760 --> 07:12.490 place things or interact with our holograms. Now in this case my cube is kind of outside of the room. 07:12.490 --> 07:16.060 If I leave the room and pan around to the back of it... 07:16.060 --> 07:21.769 Let's see if I can find my cube; it's around here somewhere... 07:21.769 --> 07:24.009 Where did you go cube? 07:24.009 --> 07:26.649 "Is it about my cube?" There it is. 07:26.649 --> 07:30.849 So, my cube is a little outside the world... but it's here and you can see... and 07:30.849 --> 07:34.329 then as we kind of pull away, that the room kinda takes a little bit more 07:34.329 --> 07:39.059 of a shape. And of course the longer that you stay in the room the more the HoloLens is able to 07:39.059 --> 07:42.379 map the surroundings... but yeah, 07:42.379 --> 07:45.950 I think this kind of showcases the mesh 07:45.950 --> 07:50.590 renderer, which can be really helpful for the emulation and to see how 07:50.590 --> 07:54.050 your rendered objects are going to interact with the 07:54.050 --> 07:59.190 world around you. And hopefully we'll get to that soon; I'm still trying to get to the point where I can 07:59.190 --> 08:04.160 place objects onto these meshes... and then of course make them go away, since we don't 08:04.160 --> 08:09.020 want to be seeing these meshes when we actually wear the device... But 08:09.020 --> 08:12.580 you know, one-step-at-a-time! For now, I think that this is 08:12.580 --> 08:16.550 a good place to stop. We've got the mesh rendered, we can see what 08:16.550 --> 08:20.880 the room looks like, we've got an object in there alongside the mesh... And now the 08:20.880 --> 08:24.820 next step is to actually build something interesting. So 08:24.820 --> 08:29.580 I think the "Hello World" stuff is mostly behind us. The next thing I really would want to do is dive into the 08:29.580 --> 08:34.170 tutorials of Unity 3D and try to see if I can adapt them so that they can kind of make 08:34.170 --> 08:39.850 sense this 3D device. So we'll see how that goes. In the meantime I want 08:39.850 --> 08:45.330 to thank you for joining me again here. This is again Josh Morales at Falafel Software. 08:45.330 --> 08:50.790 And you can reach me on twitter at @SelAromDotNet and 08:50.790 --> 08:54.530 share what you're working on with the HoloLens. Let me know if you 08:54.530 --> 08:59.920 missed any glaring issues other the ones that I've been able to identify... 08:59.920 --> 09:02.510 One more thing I wanted to check while I'm here... 09:03.810 --> 09:05.640 Which is... 09:05.640 --> 09:07.450 does my 09:07.450 --> 09:09.400 Gaze cursor... 09:09.400 --> 09:14.070 react to the world? I forgot to check that... And sure enough it does. 09:14.070 --> 09:18.550 I'm not sure how well this comes out on the video, but if you get a little closer you can see here that 09:18.550 --> 09:21.870 blue cursor as I move off, does change back into the 09:21.870 --> 09:27.310 non-cursor just like it did with the cube. So that's pretty cool so this cursor is 09:27.310 --> 09:32.480 I guess "world aware"... and as I kinda pull way, it 09:32.480 --> 09:37.530 has that same limitation of 05 units, I think. So if I get far enough away it's 09:37.530 --> 09:42.000 gonna kind of stay as that purple object that represents, 09:42.000 --> 09:46.120 you know, non-holograms. But as I get closer then we do kind 09:46.120 --> 09:51.480 of get that interactivity of getting that cursor to reveal 09:51.480 --> 09:56.120 it is an actual detected object; in this case 09:56.120 --> 20:00.930 a world object, represented by that mesh. So we'll definitely look at that little bit 20:00.930 --> 20:07.800 closer... sorry I got a little... I was already on my way out, so thanks again for watching. 20:07.800 --> 20:15.290 Let me know your thoughts and, as always, I hope this was helpful. Have a good day. Thanks, goodbye.