Alrighty folks. I have another application for all you computer vision fans out there. This time, it’s an application that combines C#, OpenCV, and DirectX to produce kind of a virtual 3D environment. The application uses a standard webcam to track your head movements. As you move your head, the 3D environment will respond in return. The idea is based on the head tracking for desktop VR displays using the Wiimote by Johnny Lee as seen in the following YouTube video.

Even though Johnny Lee’s application is extremely cool and is the starting point for my application, it isn’t as good as it could be. For example, Johnny’s application requires a Nintendo Wiimote and some goofy looking glasses with IR LEDs mounted on them. My application does away with all of this unnecessary equipment and replaces it all with a single webcam. Adding OpenCV into the mix allows you to do the head tracking without the need for the Wiimote. Checkout my video and be sure to download a copy of the source code along with a working version of the application from the link below the video. Also, be sure to let me know what you think of the application in the comments below.










Download Virtual 3D using OpenCV and C# here. (7MB)

Thank you for your interest in my site. If you find the information provided on this site useful, please consider making a donation to help continue development!

PayPal will open in a new tab.
$2.00
$5.00
Other

Related Posts

Tagged with:  

7 Responses to Virtual 3D with OpenCV and C#

  1. karimmahmoud says:

    PERFECT work Lucus

    • LuCuS says:

      Thanks! I’m getting ready to share an updated version of the app sometime over the weekend. The current version has 2 separate windows and defaults to the first camera it finds. The new version has both windows displayed in a single Windows Form. That way you don’t have to drag screens around. Plus, the new version includes a drop down menu that allows you to select which camera you want to use. I’m also adding in the ability to load 3D models into the environment which can be manipulated by moving your head or by voice commands. At some point I plan on adding support for an extra camera so that you can manipulate the 3D environment using your hands as well as your face and voice.

  2. paf0044 says:

    Hi LuCus ,great tutorials man, I’m currently working on AR and i believe the tools i need are unity and C# (Which i have never done before). Can you please guide me to more tutorials on opencvsharp as well as how to go about and complete my project.
    areas i need to cover for my project:
    – Detect and track face and eyes (USB camera feed)
    – Save into database
    – Perform face recognition
    – Do calibration and projection(3D)
    – Determine the X, Y, Z, coordinates along with the angle and direct your 3D model in unity

    • LuCuS says:

      I haven’t done any work with Unity. The example in this article was all done using DirectX. As for other OpenCV articles, you can go to http://www.prodigyproductionsllc.com/articles/category/software-development/opencv/ where you will find a list of other articles I have written. (Make sure you click the red “Previous Articles” link at the bottom of each list as the lists only show 5 results per page.) You can also checkout my other website at http://www.learncomputervision.com (Learn Computer Vision [dot] com). Within those lists, you will find a few articles that show how to detect and track faces as well as eyes. This article (http://www.prodigyproductionsllc.com/articles/programming/augmented-reality-using-c-and-opencv/) also includes a ZIP download that can help get you started with Augmented Reality. That project shows how to layer a video feed over the top of markers that are within another video feed coming from a live webcam. And, for one more AR example using DirectX, you should take a look at http://www.prodigyproductionsllc.com/articles/programming/augmented-reality-video/.

    • LuCuS says:

      You can also use the link http://www.prodigyproductionsllc.com/articles/tag/opencv/ to see a better list of OpenCV articles.

      • paf0044 says:

        Hi LuCus , thx for your other website, now i believe i am getting there. I’v been trying the whole day to combine both face detection and eye detection but i keep on getting very funny results. at the end i did this, which is an empirical way of thinking about the human face, it also save some computation time. I add a little on your code to have this

        using System;
        using System.Collections.Generic;
        using System.Runtime.InteropServices;
        using System.Diagnostics;
        using System.Linq;
        using System.Text;
        using System.Threading;

        using OpenCvSharp;

        namespace Virtual3D
        {
        public class HeadTracker
        {
        #region Camera Variables
        private const double Scale = 2.0;
        private const double ScaleFactor = 2.5;
        private const int MinNeighbors = 1;

        private CvCapture cap;
        private CvHaarClassifierCascade cascade;
        private CvMemStorage storage;

        private int _posX;
        private int _posY;

        private Thread _cameraThread;
        #endregion

        public HeadTracker()
        {
        cap = CvCapture.FromCamera(1);
        cascade = CvHaarClassifierCascade.FromFile(“haarcascade_frontalface_alt2.xml”);
        //eyecascade = CvHaarClassifierCascade.FromFile(“haarcascade_eye.xml”);
        storage = new CvMemStorage();

        _cameraThread = new Thread(new ThreadStart(Update));
        _cameraThread.Start();
        }

        public int PosX
        {
        get { return _posX; }
        }

        public int PosY
        {
        get { return _posY; }
        }

        public Thread CameraThread
        {
        get { return _cameraThread; }
        }

        public void Update()
        {
        CvWindow win = new CvWindow(“Virtual 3D”);
        while (CvWindow.WaitKey(10) < 0)
        {
        IplImage img = cap.QueryFrame();
        IplImage smallImg = new IplImage(new CvSize(Cv.Round(img.Width / Scale), Cv.Round(img.Height / Scale)), BitDepth.U8, 1);
        using (IplImage gray = new IplImage(img.Size, BitDepth.U8, 1))
        {
        Cv.CvtColor(img, gray, ColorConversion.BgrToGray);
        Cv.Resize(gray, smallImg, Interpolation.Linear);
        Cv.EqualizeHist(smallImg, smallImg);
        }

        storage.Clear();

        CvSeq faces = Cv.HaarDetectObjects(smallImg, cascade, storage, ScaleFactor, MinNeighbors, 0, new CvSize(30, 30));

        for (int i = 0; i < faces.Total; i++)
        {
        CvRect r = faces[i].Value.Rect;
        CvPoint center = new CvPoint
        {
        X = Cv.Round((r.X + r.Width * 0.5) * Scale),
        Y = Cv.Round((r.Y + r.Height * 0.5) * Scale)
        };

        CvPoint leftEye = new CvPoint
        {
        X = 40 + Cv.Round((r.X + r.Width * 0.1) * Scale),
        Y = 65 + Cv.Round((r.Y + r.Height * 0.1) * Scale)
        };

        CvPoint rightEye = new CvPoint
        {
        X = 160 + Cv.Round((r.X + r.Width * 0.1) * Scale),
        Y = 65 + Cv.Round((r.Y + r.Height * 0.1) * Scale)
        };

        int radius = Cv.Round((r.Width + r.Height) * 0.25 * Scale);
        int radius2 = (int)radius/4;

        img.Circle(center, radius, CvColor.Red, 3, LineType.AntiAlias, 0);
        img.Circle(leftEye, radius2, CvColor.Blue, 3, LineType.AntiAlias, 0);
        img.Circle(rightEye, radius2, CvColor.Blue, 3, LineType.AntiAlias, 0);

        _posX = center.X;
        _posY = center.Y;

        _posX = (img.Width / 2) – _posX;
        _posY = (img.Height / 2) – _posY;
        }

        // my own piece of code to track eyes

        // end of my code.

        win.Image = img;
        }
        }
        }
        }

        what do you think?
        Thx for your kind advices

        • LuCuS says:

          That’s one way you could handle it. A while back I wrote an app that guestimated the location of facial features such as the eyes, nose, and mouth using a similar technique. Here is another, simpler in my opinion approach. Unlike your code, this code takes into account head tilting.

          using System;
          using OpenCvSharp;

          namespace EyeDetect
          {
          class EyeDetect
          {
          public EyeDetect()
          {
          CvCapture cap = CvCapture.FromCamera(1);
          CvWindow w = new CvWindow("Eye Detection");
          IplImage img;

          const double Scale = 2.0;

          CvHaarClassifierCascade face_cascade = CvHaarClassifierCascade.FromFile("haar/haarcascade_frontalface_alt2.xml");
          CvHaarClassifierCascade eye_cascade = CvHaarClassifierCascade.FromFile("haar/haarcascade_eye.xml");
          CvMemStorage storage = new CvMemStorage();

          while (CvWindow.WaitKey(10) < 0) { img = cap.QueryFrame(); IplImage smallImg = new IplImage(new CvSize(Cv.Round(img.Width / Scale), Cv.Round(img.Height / Scale)), BitDepth.U8, 1); using (IplImage gray = new IplImage(img.Size, BitDepth.U8, 1)) { Cv.CvtColor(img, gray, ColorConversion.BgrToGray); Cv.Resize(gray, smallImg, Interpolation.Linear); Cv.EqualizeHist(smallImg, smallImg); } storage.Clear(); CvSeq faces = Cv.HaarDetectObjects(smallImg, face_cascade, storage, 2.5, 1, 0, new CvSize(30, 30));

          if (faces.Total > 0)
          {
          CvRect face = faces[0].Value.Rect;
          CvPoint center = new CvPoint
          {
          X = Cv.Round((face.X + face.Width * 0.5) * Scale),
          Y = Cv.Round((face.Y + face.Height * 0.5) * Scale)
          };
          int radius = Cv.Round((face.Width + face.Height) * 0.25 * Scale);

          CvRect r = faces[0].Value.Rect;
          r.X = center.X - radius;
          r.Y = center.Y - radius;
          r.Width = radius * 2;
          r.Height = radius * 2;
          img.Rectangle(r, CvColor.Red, 1);

          storage.Clear();

          // Consider swapping out img with smallImg for faster processing
          img.SetROI(r);

          CvSeq eyes = Cv.HaarDetectObjects(img, eye_cascade, storage, 1.15, 3, 0, new CvSize(25, 15));
          for (int i = 0; i < eyes.Total; i++) { CvRect eye = eyes[i].Value.Rect; if(i < 2) img.Rectangle( eye.X, eye.Y, eye.X + eye.Width, eye.Y + eye.Height, CvColor.Red, 1); } Cv.ResetImageROI(img); storage.Clear(); } w.Image = img; Cv.ReleaseImage(img); } } } }

Leave a Reply