Tuesday, September 02, 2014

Automate Google Trends download using R

Updated 13.10.2014

If you're using Google Trends for research or analysis, downloading the data can be time consuming. This tool uses R to automate the downloading process. The function URL_GT creates the URL for the Google Trends file export. It takes the following parameters:

keyword
A single query or a vector of queries. Google Trends does not accept more than five queries at a time.

country
If you are interested in the search volume for a specific country, add the country code here. I.e. country="US" for searches in the US.

region
If you have specified a country, you may also add a particular region in that country. Region="AL" gives you the searches in Alaska, if you have specified "US" as country.

year
If you want to get the searches for a specific year, enter year=2012 here.

month
Select the starting month by adding the months number here.

length
The parameter length specifies how many months of data to retrieve. The default is three months, as this is the maximum length for daily data. Queries for over three months will result in weekly or monthly data.

The function returns the URL for the csv download.

In order to automate the file download, you can use the downloadGT function. Add the URL to the file (as returned by the URL_GT function) and your browser's download path. The function uses your default web browser to open the link and returns the file path for the csv file.

#This script automates the downloading of Google Trends.
#It works best with firefox in combination with the Tab Mix Plus add-on that is used to automate tab closing.
#Ask firefox not to prompt for new downloads and this script should run automatically.
#Google Trends restricts the number of download to roughly 400 at a time.

URL_GT=function(keyword="", country=NA, region=NA, year=NA, month=1, length=3){


  start="http://www.google.com/trends/trendsReport?hl=en-US&q="

  end="&cmpt=q&content=1&export=1"
  geo=""
  date=""

#Geographic restrictions

  if(!is.na(country)) {
    geo="&geo="
    geo=paste(geo, country, sep="")
    if(!is.na(region)) geo=paste(geo, "-", region, sep="")
  }

  queries=keyword[1]

  if(length(keyword)>1) {
    for(i in 2:length(keyword)){
    queries=paste(queries, "%2C ", keyword[i], sep="")
    }
  }

#Dates
  if(!is.na(year)){
    date="&date="
    date=paste(date, month, "%2F", year, " ", month+length-1, "m", sep="")
  }

  URL=paste(start, queries, geo, date, end, sep="")

  return(URL)
}


downloadGT=function(URL, downloadDir){


#Determine if download has been completed by comparing the number of files in the download directory to the starting number

  startingFiles=list.files(downloadDir)
  browseURL(URL)
  endingFiles=list.files(downloadDir)

  while(length(setdiff(endingFiles,startingFiles))==0) {

    Sys.sleep(3)
    endingFiles=list.files(downloadDir)
  }
  filePath=setdiff(endingFiles,startingFiles)
  return(filePath)
}



4 comments:

Anonymous said...

How would you modify this to get all metropolitan data in each state?
Thanks!

rg said...

Hi Erik, I have some doubts here. What is the best way to reach you?

Elizabeth said...

I was so anxiuos to know what my husband was always doing late outside the house so i started contacting hackers and was scamed severly until i almost gave up then i contacted this one hacker and he delivered a good job showing evidences i needed from the apps on his phone like whatsapp,facebook,instagram and others and i went ahead to file my divorce papers with the evidences i got,He also went ahead to get me back some of my lost money i sent to those other fake hackers,every dollar i spent on these jobs was worth it.Contact him so he also help you.
mail: premiumhackservices@gmail.com
text or call +1 4016006790

patialalegitimate said...

Are you in need of Loan? Here all problem regarding Loans is solve between a short period of time what are you waiting for apply now and solve your problem or start a business with funds Contact us now. many more 2% interest rate.(Whats App) number +919394133968 patialalegitimate515@gmail.com
Mr Sorina

Entertaining Blogs - BlogCatalog Blog Directory
Bloggtoppen.se