Head Tracking for Desktop VR Displays using the WiiRemote

Here's some specific suggestions. I was most recently fascinated by this video on head tracking "virtual reality". ▶ this video Another cool thing if you're into DIY robots (not really educational, but I found it inspiring) is the Yellow Drum Machine. link Not sure why I got on the robot tangent but here's another good one, a 4-legged robot that's pretty good at navigating hills and obstacles. Especially the recovery from a fall sequence around 1:25. link And here is hacking the Wii Fit controller and a Roomba to do a sort of vacuum-by-surfing... thing? link

I remember back in 2008-ish Johnny Lee at CMU built a cool hack that tracked the user's head using a Wiimote as an infrared camera, and used it for this kind of effect. ▶ this video Turns out that head-tracking parallax is surprisingly effective even without stereo vision. I'd guess there's some component about the effect working best when your head motion is large relative to the distance between your eyes, and also best for objects far enough away from your eyes that you're not getting a lot of information from the stereo vision. I don't know exactly where those thresholds are, but I wouldn't be surprised if a pinball machine is in a regime where it works well.

I tried to implement Johnny Lee's amazing idea (▶ this video) using mediapipe face tracking. I could not move much far using simple webcams since it was getting difficult to determine the distance of the face from the camera when the face was turned. I had an Inel RealSense 415 depth tracking camera from a different project and it took care of the distance thing at least. But the jitter thing had me stumped for a long time and I put the project away. With your ideas, I get the strength to revisit it. Thanks!

I absolutely agree. I'm very much a whiteboard hand-wavier guy. I explain with gestures as much as I do with the actual drawing (might be an Italian thing too). That said, this "pancake" problem reminded me of the old Wii head-tracking project.[1] There might be an interesting hybrid opportunity with that idea and 3d avatars. I feel like the head-tracking hack never really took off because it only worked as a solo experience. But since remote work is mostly us all individually sitting at a computer I could see it working better. Combine this idea with a large dedicated monitor/tv and now you have something that would literally just feel like a window/portal to the person you're talking to. [1] ▶ this video

I have very little knowledge of this, but as a product would it not be much cheaper to have something like a wii bar fixed to the top and bottom of a screen, with a small camera at the extremities of each bar (one for each of the four corners)? You could then use the camera perspectives to create a 3d image of the person you're conversing with and map the colour data correctly to that 3d image (Photogrammetry) You could also likely use the information from the four cameras to map the orientation of the 3d image of the person you're speaking with to give you that sense of depth as you shift your position. ▶ this video If you had a speaker and mic in each corner you could also capture / emit subtle differences in audio to further enhance it.

Games made to take advantage of hardware innovations that were then built upon. Nintendo has had a direct hand in the implementation of new (or newly low-cost) technologies that have then gone on to serve as the basis for more widespread innovations. Ex. They placed a cheap infrared camera in their Wiimote to enable controller tracking (which means that that tech got into the hands of millions of users). An engineering student reversed this, using that camera to track LEDs placed on a pair of glasses to achieve cheap headtracking.(1) This was then used as the basis for the tech in Microsoft's Kinect(2), which served as the basis for the tracking technologies we see in modern XR setups. That's just one example of many. 1: ▶ this video 2: link

"We offer an experience of 3D on a single 2D image using the parallax effect, i.e, the user is able move his real-time tracked face to visualize the depth effect." Since there are no examples I can't be sure if this is is what I think it is, but IF it is: I want this on huge monitor for any 3D game instead of clunky headgear VR or tiny smartphone AR. Months ago I also tested this with a small 3D visualization and very crude head tracking. The effect is damn awesome! To be able to move around in real space with the rendering adapting to it, makes it so immersive, even for my very crude tests. In my opinion the resulting 3D effect is MUCH better than viewing in stereo with one picture per eye. Here is an example from someone else from 2007: ▶ this video From 2012: link Obviously this works for only one person viewing it, but does that really matter? There are a LOT of use cases where only a single person uses a single monitor for viewing, especially in these times. In fact it is the standard.