Prodigy Productions, LLC

Looking for a way to scrape Google Trends using C#? Well, look no further. Here’s a quick and dirty way to scrape Google Trends for Hot Searches using C#.

To begin, fire up Visual C# Express and create a new Console Application. You can also make this a WinForm Application if you want. But, for demo purposes, we’ll just make it a simple Console App for now. After you’ve created the new project, create a new Class called “Utilities.cs”. This will be a place holder for a simple utility we’ll be using to get text from between 2 strings. Inside the Utilities.cs class, copy the code from below and paste it inside the “class Utilities { }” curly brackets.

        public static string[] GetStringInBetween(string strBegin, string strEnd, string strSource, bool includeBegin, bool includeEnd)
        {
            string[] result = { "", "" };
            int iIndexOfBegin = strSource.IndexOf(strBegin);

            if (iIndexOfBegin != -1)
            {
                // include the Begin string if desired
                if (includeBegin)
                    iIndexOfBegin -= strBegin.Length;

                strSource = strSource.Substring(iIndexOfBegin + strBegin.Length);

                int iEnd = strSource.IndexOf(strEnd);

                if (iEnd != -1)
                {
                    // include the End string if desired
                    if (includeEnd) iEnd += strEnd.Length;

                    result[0] = strSource.Substring(0, iEnd);

                    // advance beyond this segment
                    if (iEnd + strEnd.Length < strSource.Length)
                        result[1] = strSource.Substring(iEnd + strEnd.Length);
                }
            }
            else
                // stay where we are
                result[1] = strSource;

            return result;
        }

Now, back in your Program.cs file, copy the code from below and paste it inside your “Main” method. This is the piece that will be doing all of the work. It starts out by creating a new WebClient object that downloads the page contents from http://google.com/trends and stores the HTML content in a string variable called “html”. It then splits the HTML into lines and iterates the string array of lines looking for any line that contains the text “<td><a href=/trends/hottrends?q=“. If it finds that particular string, it then uses our Utilities.GetStringInBetween method we created earlier to get the text between the “<td>” and “</td>” tags and then again between the “>” and “</” strings and prints out what it finds.

using System;
using System.Text.RegularExpressions;
using System.Net;

namespace GoogleTrends
{
    class Program
    {
        static void Main(string[] args)
        {
            WebClient wc = new WebClient();
            String html = wc.DownloadString("http://google.com/trends");

            string[] lines = html.Split('n');
            foreach (string line in lines)
            {
                if (line.Contains("<td><a href=/trends/hottrends?q="))
                {
                    string hotTermLine = Utilities.GetStringInBetween("<td>", "</td>", line, false, false)[0];
                    string hotTerm = Utilities.GetStringInBetween(">", "</", hotTermLine, false, false)[0];
                    Console.WriteLine(hotTerm);
                }
            }

            Console.ReadLine(); // This is here so we can view the hot search terms before the app closes
        }
    }
}

This could be useful for many things including picking popular search terms & optimizing your site to match those keywords. You could even save those searches into a database for statistical purposes later on. Be creative and find some nice things to do with those “Hot Searches”. Later on I’ll write another article explaining in detail how to use those search terms to generate traffic to your site.
 

Related Posts

• Create an RSS Reader Using C#
• Similarities and Differences between PHP & Ruby
• Create Your Own Programming Language
• Getting Started with OpenCV and Python
• Multiprocessing Bug with Windows and Python Idle