Friday, December 05, 2014

Scraping Google Trends with R

These R functions will allow you to programmatically download Google Trends data and importing it to R.

Step 1: Install the Google Trends functions from my Github account.
Step 2: Sign in to Google Trends in your main browser
Step 3: define the keywords you need

keywords=c("Samsung", "Apple", "Xiaomi")

Step 4: create list of URL:s (in this example, we'll have only one URL)

url=URL_GT(keywords)

Step 5: specify your browser download directory and set it as your working directory

downloadDir="C:/downloads"
setwd(downloadDir)

Step 6: download the csv:s. The function outputs the file name.

filePath=downloadGT(url, downloadDir)

Step 7: import the csv to R

googletrends_data=readGT(filePath)

In this post, I write about how to merge daily data from Google Trends into longer time series using R.

10 comments:

Unknown said...

Dear Erik,
there seems to be a problem with multiple queries: while your commands with only 1 keyword works, with 2 or more unfortunately not. I inserted setwd(downloadDir) as you told me, but when there are 2 or more queries I get the following error message:

Error in name[1] = substr(raw_name, 1, separators[1] - 1) :
object 'name' not found

Best regards, Dean

ErikJJ said...

Hi Dean,

I've updated the code the resolve that issue. Could you please try to run the code again and see if it works now?

Erik

Unknown said...

Now it works thanks. I have only one last question: I tried to use your new functions (which now work) with your old code for creating a long time series of daily data that you discussed here

https://github.com/321k/Google-Trends/blob/master/Daily%20in%20percent

but unfortunately it stops after the first download. Thanks for your continued help. Regards, Dean

ErikJJ said...

Thanks Dean for pointing that out!

Follow the steps in this post to make it work with the updated functions:
http://erikjohansson.blogspot.fi/2014/12/creating-daily-search-volume-data-from_8.html

Eleonor said...

What language is R written?

Anonymous said...

Hi Erik,

Have you noticed that when you run this scrape for a 4 hour window the downloaded file has 30 data points (one every eight minutes), however when you manually download data for the past 4 hours directly off Google Trends, the file has one data point per minute?

Do you know why? Is there a way to fix this?

Thanks!

Anonymous said...

This is the URL I created for the scrape using superbowl as the example keyword: http://www.google.com/trends/trendsReport?hl=en-US&q=superbowl&date=now%204-H&cmpt=q&content=1&export=1

Anonymous said...

Hi,

Super useful, thank you! I am really new to programming, could you explain where to change the code so I can search for a specific country? I tried to change it in the URL_GT function at the beginning, changing country=NA to country=Canada but it does not work.

I would really appreciate your help!

Thank you

Aaron Esteban said...

There seems to be a token that Google Trends requires to scrape their JSON file. Do you know of any other way to access the file without needing the token? Which URL should I use to get to the keywords on a given search?

Rebecca Michaelson said...

HELLO VIEWERS

TESTIMONY ON HOW I GOT MY LOAN €300,000.00EURO FROM A FINANCE COMPANY LAST WEEK Email for immediate response: drbenjaminfinance@gmail.com

Do you need a loan to start a business or pay your debts {Dr.Benjamin Scarlett Owen} can also help you with a legit loan offer. He Has also helped some other colleagues of mine with a loan finance. Get your Blank ATM card or CREDIT CARD deliver to your doorstep that works in all ATM machines all over the world with the help of BENJAMIN LOAN FINANCE the ATM cards can be used to withdraw at the ATM Machines or swipe, at stores and POS. they give out this cards to all interested clients worldwide, If you need a loan without cost/stress he his the right loan lender to wipe away your financial problems and crisis today. BENJAMIN LOAN FINANCE holds all of the information about how to obtain money quickly and painlessly via Email drbenjaminfinance@gmail.com

Entertaining Blogs - BlogCatalog Blog Directory
Bloggtoppen.se