MouseOne of my readers recently asked if I would write an article explaining how to control a mouse pointer using speech recognition and C#. Unfortunately, I have been devoting every minute to my new startup and haven’t had time to write the article as the reader requested. I did however manage to squeeze in a few minutes to throw together a simple Windows Form Application that should be a great starting point for a project like this. As soon as I get a few extra minutes, I will come back and write a complete article explaining how everything works and how to improve on it. Until then, I’m going to go ahead and provide you with the same source code I provided my other reader as I’m sure there are others out there interested in doing something similar. But, as always, if you have any questions or comments about the code below, feel free to leave me a message in the comments below and I’ll answer you as soon as possible.

In order to make this code work, you will need to right-click on References in your Solution Explorer and choose “Add Reference”. Then, you will need to select “System.speech” and add it as a reference. Next, you will need to add a text control (I used a rich text box) to your form and name it “txtOutput”. You can rename this control to whatever you want as long as you also change the code below. Once you have your reference and text control added, you are ready to use some code. When you run the code, you can say the commands “up”, “down”, “left”, “right” and your mouse pointer will jump respectively in increments of 50. But, you can change that to whatever you want.

So, here is the code to make it all happen. Until next time, happy coding!

using System;
using System.Drawing;
using System.Text.RegularExpressions;
using System.Windows.Forms;

using System.Speech.Recognition;

namespace MouseController
{
    public partial class Form1 : Form
    {
        SpeechRecognitionEngine recognitionEngine; 

        public Form1()
        {
            InitializeComponent();
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            Initialize();
        }

        private void Form1_FormClosing(object sender, FormClosingEventArgs e)
        {
            recognitionEngine.RecognizeAsyncStop();
        }

        private void Initialize()
        {
            recognitionEngine = new SpeechRecognitionEngine();
            recognitionEngine.SetInputToDefaultAudioDevice();
            recognitionEngine.SpeechRecognized += (s, args) =>
            {
                string line = "";
                foreach (RecognizedWordUnit word in args.Result.Words)
                {
                    if (word.Confidence > 0.5f)
                        line += word.Text + " ";
                }

                string command = line.Trim();

                switch (command)
                {
                    case "left":
                        MoveMouse(Cursor.Position.X - 50, Cursor.Position.Y);
                        break;
                    case "right":
                        MoveMouse(Cursor.Position.X + 50, Cursor.Position.Y);
                        break;
                    case "up":
                        MoveMouse(Cursor.Position.X, Cursor.Position.Y - 50);
                        break;
                    case "down":
                        MoveMouse(Cursor.Position.X, Cursor.Position.Y + 50);
                        break;
                }

                txtOutput.Text += line;
                txtOutput.Text += Environment.NewLine;
            };

            recognitionEngine.UnloadAllGrammars();
            recognitionEngine.LoadGrammar(CreateGrammars());
            recognitionEngine.RecognizeAsync(RecognizeMode.Multiple);
        }

        private Grammar CreateGrammars()
        {
            Choices commandChoices = new Choices("left", "right", "up", "down");
            GrammarBuilder grammarBuilder = new GrammarBuilder();
            grammarBuilder.Append(commandChoices);
            return new Grammar(grammarBuilder);
        }

        private void MoveMouse(int x, int y)
        {
            this.Cursor = new Cursor(Cursor.Current.Handle);
            Cursor.Position = new Point(x, y);
            Cursor.Clip = new Rectangle(this.Location, this.Size);
        }
    }
}

Thank you for your interest in my site. If you find the information provided on this site useful, please consider making a donation to help continue development!

PayPal will open in a new tab.
$2.00
$5.00
Other

Related Posts

Tagged with:  

20 Responses to Control Your Mouse with Speech Recognition and C#

  1. Visitha says:

    Hi Lucus,
    First of all I really appreciate about your posts and they are really helpful to any one who are not familiar with voice recognition. I red your this article and “Simple voice recognition with C#” and those articles were very interesting and helpful to increase my knowledge..Thanks again for you about this great articles and your service for other people.

    These days I’m developing a voice recognition application for fire fox which can “close” ,”Go back”,”scroll” firefox using my language “Sinhala” as my university project. I’m very beginner for this subject and have no idea about how to implement it. I kindly request from you to help me to build this application. If you please give me some resources about this application..

    Thanks for consideration
    Good Luck..!!

    • LuCuS says:

      I’m glad you’ve found my site helpful. In order for you to control Firefox (or any other Windows application) from C#, you’re going to need to take a look at Windows Automation. I have written a simple example of this that automates the Windows calculator. You can find that article at http://www.prodigyproductionsllc.com/articles/automation/windows-automation-with-c/. You could essentially use the code from this article and combine it with the code from the Windows Automation article to achieve what you’re looking for. I’ll try to find some time to put together an example app for you, but I’m not sure when I’ll be able to get around to it as I’m neck deep in coding for my new startup. I’ll try to squeeze in a bit of code here and there for you and hopefully have something working by the end of the weekend. Until then, be sure to take a look at the Windows Automation article as it’ll help you successfully achieve your goal.

    • LuCuS says:

      Actually, I just went ahead and took a couple of minutes to put together an example application for you. Before you run it, make sure you have Firefox already opened. Once you have Firefox running, run the application and click the “Start Automation” button. The only commands I’ve implemented are “home” and “back”. But, you can see how I did that and can easily implement other buttons such as “forward”, “refresh”, etc… You can download the example app from http://www.prodigyproductionsllc.com/downloads/FirefoxAutomation.zip. Let me know if it helps point you in the right direction.

      • Visitha says:

        Hi lucus,
        Thank you very very for quick response for my issue..=)..I try with your Firefox automation.zip application. But when it runs, after searching for firefox window, it gives an error saying “Error locatng firefox window”. I have already run firefox when run the application. But I couldn’t fix this error..What is the reason to that error..If you have time, please consider about that.

        Thank you.

        • LuCuS says:

          What version of Firefox are you running? I just tried it with v10.0.2 and it worked fine. Go to the Windows Automation article I posted a link to and grab the UISpy tool. While Firefox is running, run UISpy. Then, scroll down until you see a “window” that says something like “[PAGE TITLE] – Mozilla Firefox” where “[PAGE TITLE]” will be the title of the page you are currently on. Click on that window and on the right side of UISpy you will see a “Properties” panel. Under “Identification”, you will see a class name. This class name should be “MozillaWindowClass”. If you see something different, copy that class name and paste it into line 67 of my app and re-run it. Here is what line 67 looks like:

          Automation.Condition condition = new PropertyCondition(AutomationElement.ClassNameProperty, "MozillaWindowClass");

          Paste the class name from UISpy into the last section of that line, replacing “MozillawindowClass”.

          • Visitha says:

            Haai lucus,
            Your application works perfectly wel..=) Now i’m trying to implement it in my language and add more functionality. You are such a great and talented person I have ever seen. That application helps me very very very much..=)Thnkz again for your consideration about my application by consuming your valuable time.

          • LuCuS says:

            Thank you. I’m glad I could help. In order to for your application to support another language, you will need to look into globalization. To begin with, you will need to add a reference to System.Globalization at the top of your code like this:

            using System.Globalization;

            Next, you will need to create a new CultureInfo object (System.Gloablization.CultureInfo) for your culture like this:

            CultureInfo culture = new CultureInfo("si-LK");

            After that, you will want to pass your culture object to your SpeechRecognitionEngine constructor like this:

            SpeechRecognitionEngine sre = new SpeechRecognitionEngine(culture);

            Now you have everything you need to support your language. The only thing you have left to do is change the words in your Grammar Builder from “back” and “home” to whatever words those translate to in your language. Be sure to change them both in the CreateGrammars function as well as in the switch-case at line 110.

  2. Visitha says:

    Haai lucus, it was great to see that you have made another wonderful article for Firefox..=) it will be very useful for every one who are going to learn advance voice recognition whole over the world..If you can please send me your e mail address..I am going to add several functions for your application..I think I will need your help for make this functions..Please..

  3. shuvro says:

    Thanks for this code LuCus.This is the starting for what i was looking.I understand your code and found very helpful.

  4. Sniderman22 says:

    Thanks for so many awesome articles! Can i ask you something however. Iv read all of your guides and even tried(with proper crediting) copying the code 100% word for word into a document of my own. But i honestly just cannot get it to work! Visual studio 2010 windows form app in C# .NET 4.0, we are doing the same thing and yet i get errors in some of your lines of code like the “=>” on line 33. Am i missing something i just dont get why nothing makes it work. Iv been all over the web looking for any examples that i can use to get me started and yours is the closest. Im not the most experiences programmer but i just NEED to get this to work lol

    • LuCuS says:

      I’m not sure why you would be getting an error about the “=>” on line 33. But, we’ll take this one step at a time. Did you add a reference to “System.Speech”? In your Solution Explorer, right click on “References” and select “Add Reference”. Click the .NET tab and select “System.Speech”, then click “OK”. Another thing you might want to try is click on “Project” on the menu bar and select Properties (last item in the menu). Click the “Application” tab and make sure Target Framework is set to “.NET Framework 4”, not “.NET Framework 4 Client Profile”. If you change this, when you save it, it will close the pages you have open. When that happens, just reopen the .cs file. Also, when you created the app, did you select “Windows Form Application” and did you add a RichTextBox to your form called “txtOutput”?

      If these things don’t work for you, can you post the errors you’re getting?

      • Sniderman22 says:

        Okay so in working with your advice i do manage to get all of the code accepted. No errors but for some reason there is no response to the voice commands :/.
        Iv been doing a lot of research on this topic and of the manyyy sites if visited yours is by far the best in terms of post quality. Im looking for just a very SIMPLE as it gets example of like one instance of it working. Im trying to learn the basics of it before i jump into apply it.

        If i can just ask can you help me along those lines? Like im talking a txt box that prints “hello” when you say the word hello. Nothing to generally complicate just the essentialsI just need to get a working example of the basic structure so i can study it and apply it for myself, ya know?
        -So after adding the proper references and properties, include a “using System.Speech.Recognition;”
        -After that intialize the SpeechRecognitionEngine”
        I have that much memoriszed, im just not confident in what should follow. I see what should but dont more or less know how to work with it yet. If you could help that would be truly amazing, thanks

        • LuCuS says:

          In this article “http://www.prodigyproductionsllc.com/articles/programming/simple-speech-recognition-using-c/”, I show a very simple example of listening for speech input and printing it to a text box. I took the code from that article and made you a ZIP file for download. This has everything you need to get started with doing speech recognition. You can download the example ZIP file from http://www.prodigyproductionsllc.com/downloads/SpeechExample.zip. Hopefully that will get you moving in the right direction. One thing to keep in mind though is if you have an accent or plan on speaking in a language other than English (French for instance), you will need to start looking into cultures objects as shown here: http://www.prodigyproductionsllc.com/articles/programming/speech-recognition-with-c-and-xml-grammars/#comment-821.

          • Sniderman22 says:

            Its truly amazing, but it finally all works! I can finally see the wods i say being posted into the text field. Thank you so much for ziping it all up for me!
            If i may ask, can you possibly explain this part:

            foreach (RecognizedWordUnit word in args.Result.Words)
            {
            I understand it creates a for each to continually display the words, but the arguments within the () i dont understand. Im looking to apply this in a more command and execute oriented fashion. Does that entail creating separate recognition for each different command, or can all the different commands be loaded in one grammer? Eg starting programs, performing random other computer events. Thanks

          • LuCuS says:

            The recognition engine stores recognized words in a list. The line you mentioned simply iterates that list and does something with them. The foreach block is inside a callback function “recognitionEngine.SpeechRecognized += (s, args) =>”. That means that the engine is listening on a separate thread and will execute the foreach every time it has words in its list. As for your second question, yes. All of your commands can be stored in the same grammar. You can do it inside the code as shown here: http://www.prodigyproductionsllc.com/articles/programming/simple-speech-recognition-using-c-part-2/. Or, if you want the ability to add / remove commands without the need to recompile your app, you can add your commands to a grammar XML file as shown here: http://www.prodigyproductionsllc.com/articles/programming/speech-recognition-with-c-and-xml-grammars/

          • Sniderman22 says:

            That first link i think what i was perfectly looking for! Im just gonna problem solve and do my best, as it seems from the introduction to the article that its kinda showing us how to do more/less exactly what im looking for! Thank you seriously for this mentoring you’ve shown me 🙂 I just might be msging agn 😛

          • LuCuS says:

            No problem and good luck with your project.

  5. mathieutan says:

    Hi LuCus,

    I tried to build your example, and things went well..except when I start to debug(F5), it highlight Line 65, InvalidOperationException was unhandled. The language for the grammer does not match the language of the speech recognizer.

    Why is this so?

    Thanks for your help.

    • LuCuS says:

      You need to make sure that the language in Windows (Control Panel) is the same as the one that .NET is using. To get .NET to use the same language as the one set in Control Panel, you will need to incorporate a CultureInfo object like this:

      using System;
      using System.Globalization;
      ...
      public Form1()
      {
      InitializeComponent();
      CultureInfo culture = new CultureInfo("en-GB");
      recognitionEngine = new SpeechRecognitionEngine(culture);

      You will need to replace “en-GB” with the culture code for the language you have set in Control Panel. However, .NET does not support all languages. You will need to do a Google search for the language you want to use and make sure .NET supports it. If it does, you’ll also need to do a Google search for the matching culture code. If you plan on using English for your app, you can leave out the CultureInfo object and just change the system language in Control Panel as .NET defaults to the English language.

Leave a Reply