As I’ve mentioned before, my articles about using OpenCV and C# are the most viewed articles on this site. Among those articles, I get more emails asking about using OpenCV and C# for augmented reality applications than I do anything else. It appears that AR is a pretty big topic these days and everyone looking at getting into the field needs a good place to start. So, I’ve put together a small application that uses OpenCV and C# to do augmented reality. As always, I’m using the OpenCvSharp .NET wrapper for OpenCV. But, the same principles that apply here can also be used in pretty much any other wrapper or in OpenCV itself. Usually, I’ll walk thru every line of code in my example applications and explain what I’m doing. But, this time, I’ve decided to just provide you with the code and let you figure it out for yourself. However, as always, I’m more than willing to answer any questions you may have as you go along with the example.
To get started, you will need to download my OpenCV Augmented Reality example application from http://www.prodigyproductionsllc.com/downloads/OpenCvAugmentedReality.zip. It already has everything you need to begin your augmented reality application including the OpenCvSharp and OpenCV runtimes which are located in the bin > Debug directory. In that same directory, you will see a file called “chessboard 6×5.jpg”. In order for this example to work, you will need to print a copy of that image onto a typical 8.5×11 piece of paper (scale doesn’t really matter here). Once you’ve printed the chessboard image, go ahead and launch the app by opening the .sln file in Visual C# or by running the AugmentedReality.exe file also found in the bin > Debug folder.
When you run the application, you will see a Windows form that only includes a button with the word “Start” on it and a combo box next to it. The combo box includes options for 1, 2, & 3. Picking number 1 means that the application will look for the checkerboard image in a video feed and will overlay a typical image over it. The image used in this example is a standard JPG file of the OpenCV logo. If you choose number 2 from the combo box, the application will look for the checkerboard in a video feed and will overlay a video over it. For this example, I’ve included a video file called “trailer.avi” which is a trailer for the “Big Buck Bunny” movie created by the Peach Open Movie Project (http://www.bigbuckbunny.org/). If you choose number 3 from the combo box, the application will look for the checkerboard image in a video feed and will draw a box around the portion of the image that is used to overlay the image or video. Here is an example of the application using number 2 to display the “Big Buck Bunny” trailer video over the checkerboard.
I’m using a cheapo USB webcam to capture the video and even running on low end laptop, I’m still getting pretty good performance out of the application. The possibilities of technology like this are endless. For example, since OpenCV can be ran on an iPhone, one could easily write an augmented reality iPhone app that can overlay advertisements on top of images when the iPhone is pointed at things like store fronts or billboards. Imagine pointing your phone at a still image in a magazine and having a video commercial play in your screen. You could do that with this kind of technology. Anyways, whatever you decide to use this for, be sure to come back here and share your story with the rest of us in the comments below. I’m extremely curious as to what all kinds of cool stuff you guys can come up with. Until next time, HAPPY CODING!!!
Related Posts
9 Responses to Augmented Reality Using C# and OpenCV
Leave a Reply
You must be logged in to post a comment.
hello Lucus..I found this tutorial very useful..
Is it possible for the detection to start only when the object is still for few seconds(i.e object in same position in consecutive frames say 10 frames)…?
Yes. You could put a timer that starts counting down when the object is first detected. When the timer runs out, you could begin playing the animation or video.
But how can i find the repeated frames..I want to eliminate unnecessary frames and detect only those which are repeated for few seconds..
I’m not sure I understand your question? Are you meaning the “repeated frames” in your camera feed or in the video overlay?
sorry for being unclear….My idea is to detect the sign made using hands..
I’m capturing frames containing human hand using IplImage* frame = cvQueryFrame( capture );
I want to detect only those frames in which the hand is showing the same sign..
Hope i’m better now…
Ah. Ok. I understand now. Sign language recognition has been discussed several times on almost every article found in this list: http://www.prodigyproductionsllc.com/articles/category/software-development/opencv/. On each of those articles, scroll down to the comments and you’ll see those discussions. I also have a demo app that shows how to use OpenCvSharp to detect hands using the blob method. (http://www.prodigyproductionsllc.com/downloads/HandDetect.zip). For a while now, I’ve been planning on writing an article showing how to do sign-language / hand recognition with OpenCV, but I’ve been backed up with several other projects and haven’t had the time. Try looking thru some of those other pages, reading thru the comments, and see if that doesn’t get you to where you want to be. I know that’s a lot of stuff to dig thru, but you’ll find a lot of good information there. One of my readers, UtopiaDreamer, has been working on this very thing for quite a while now. So, be sure to pay special attentions to those comments. Also, make sure you checkout my Template Matching article (http://www.prodigyproductionsllc.com/articles/programming/template-matching-with-opencv-and-c/). That might get you moving in the right direction.
For a quick explanation to answer your last question, you would need to store a counter that gets incremented each time a particular hand-gesture is detected. Once that counter reaches a certain number, you could do whatever work comes next. If you use something like I show in my Template Matching article in the link above, you could move your hand around as much as you want and the work wouldn’t begin until the hand is displayed in the same pose as in your template image. Word of warning though, doing hand detection is a fairly complex thing to do with OpenCV due to different hand sizes, rotations, lighting, etc… To help with a lot of those issues, I would recommend using some filters such as the Canny filter and ROI (region of interest) to help eliminate unwanted pieces of your video feed and to help with performance. Unlike the Microsoft Kinect, OpenCV relies on only 1 camera whereas the Kinect uses 2 along with some IR (infrared) trickery which allows it to detect depths and help determine the ROI a lot more accurately. But, with that said, it’s still possible to do using 1 camera and OpenCV.
Let me know if any of this does / does not make sense and we’ll take it from there.
I’m highly satisfied by your explanation…Thanks for the time….I will follow what you said and report to u …
In the meantime, when i work with blobs,i’m getting the following error
“1>testing.obj : error LNK2019: unresolved external symbol _cvFilterByArea referenced in function _main
1>testing.obj : error LNK2019: unresolved external symbol _cvRenderBlobs referenced in function _main
1>testing.obj : error LNK2019: unresolved external symbol _cvLabel referenced in function _main”
But i have referenced the blob library in both project folder and also in the project in vc++…
What could be t reason ?
It looks like you’re missing a header include.
ya but i have added them all