I saw a couple of tweets with people outside of the HMD interacting with the virtual world and had been playing around with mirrors using the Vive Stereo Rendering Toolkit when I had the idea of using the tracked controller to determine what was being displayed on the mirror so that it would effectively be what you have here.
Now to be clear, I know this is not a new idea. In fact, I would say if anything this project is a bit of an homage to one of the first times I really saw the potential of positional tracking.
Just a day after getting started I saw another tweet that was getting a lot of attention which showed a similar concept (demonstrated entirely within TiltBrush) so that had me motivated to see this thing through.
While doing research on it I found many many many different instances of people exploring this in various ways.
Everything actually went together pretty quickly and I had something working within the first week, but then I ran into some performance issues that ended up causing some direction changes and delays.
The SteamVR_TrackedObject problem:
First, I had experimented with having this working with both the SteamVR Player prefab and the SteamVR [CameraRig] prefab so people could use the one of their choice. This is when I noticed a problem with SteamVR_TrackedObject. It seemed that because it was only updating the position and rotation on new pose events, that it wasn't updating 1:1.
If you turn that off with the debugging box enabled you will see that the boundaries of the screen do not stay fixed to the boundaries of the box. Using the modified version (which just takes the position and rotation update code from the Hand component of the Player prefab) this no longer jostles around.
I don't know if this is because of something I am doing wrong with the [CameraRig] prefab or the way I'm using TrackedObject or if it's a general bug in the tracking performance, but if anyone has any idea I would be more than happy to hear it. I really don't like relying on my own TrackedObject here.
The Player prefab problem:
In addition to the TrackedObject problem, I also had an issue with the Player prefab not being performant for me at all. I should note that I was building this all on my (under spec) portable rig which is using a 4690k and a GTX 770.
If I create a new project with a new scene:
- Player prefab = ~110 fps if hands out of frame and not moving, ~70fps (stuttering) if in frame at all or moving out of frame
- [CameraRig] prefab = ~110 fps constant, w/ minor drop when hands first enter frame
- [CameraRig] prefab w/ modified TrackedObject = same ~110 fps constant, w/ minor drop when hands first enter frame
Because of that I opted to release this as [CameraRig] only. If anyone knows why I'm getting those frame rate drops I would be more than happy to hear that as well.
Final thoughts:
I had an idea and wanted to get some more practice trying solve a problem, so I did, but it was all worthless if no one ever uses the thing, so please, feel free to use this for whatever you want. Nothing would make me more happy than to see some interesting uses of this.
Some ideas I've had that might get you started:
- Player using PictureWindow could be either a teammate or adversary of the person in VR
- An exhibit allowing you to peer into a living diorama
- Entirely non-VR games, maybe using the other controller as something else to interact with
In the future I'd like to add some new features like being able to move the window around in the virtual space, but there's no timeline on that right now.
Anyway, if you made it this far thanks for checking it out!