Paper review: Can We Use Daily Internet Search Query Data to improve Predicting Power of EGARCH Models for Financial Time Series Volatility?

Seeing as Risteski & Davcev (2014) has honored me with a reference in their paper, I will share my thoughts on their research.

Their paper is available here.

Risteski & Davcev uses daily Internet Search Data from Google Trends to improve the predictive power of an EGARCH volatility model for the CAC40 index. They find that the predictive ability of their model, called EGARCH-SVI in the paper, is greater than the normal EGARCH model. The results are in line with my own research.

Data

Ristesku & Davcev uses daily and weekly index returns of the CAC40 index. They extend the EGARCH model with search volume for the term "CAC40". They point out that there is a two day delay in the data from Google Trends, something that is important to take into consideration when contemplating any practical implementation of a predictive model. In accordance with this practical limitation of the data, Ristesku & Davcev uses a two day lag for the daily model, and a one week lag for the weekly data.

Method

To compare the strength of the EGARCH-SVI model, they compare it to a normal EGARCH model. They kindly provide a reference to this blog for the method I have developed to create daily time series from Google Trends on time periods longer than the 90 days provided by Google. The strength of the Search Volume Index variable is measured by its significance level in the EGARCH regression. The forecasting performance is measured by difference in the mean square error, mean absolute error, the information criteria, and Diebold-Mariano between the normal EGARCH and the EGARCH-SVI model.

Results

The results show that the EGARCH-SVI model has better forecasting power than the basic EGARCH model. The in sample test results are stronger than the out of sample results, but both point to an improvement in the predictive ability of the EGARCH-SVI model as compared to the basic EGARCH model.

Comments on the paper

Seeing as the EGARCH model has been shown in previous research to be powerful in accounting for heteroskedasticity in financial time series, it is an ideal method to append with search volume data. The results of Ristesku & Davcev are in line with my own results.

Scraping Google Trends with R

These R functions will allow you to programmatically download Google Trends data and importing it to R.

Step 1: Install the Google Trends functions from my Github account.
Step 2: Sign in to Google Trends in your main browser
Step 3: define the keywords you need

keywords=c("Samsung", "Apple", "Xiaomi")

Step 4: create list of URL:s (in this example, we'll have only one URL)

url=URL_GT(keywords)

Step 5: specify your browser download directory and set it as your working directory

downloadDir="C:/downloads"
setwd(downloadDir)

Step 6: download the csv:s. The function outputs the file name.

filePath=downloadGT(url, downloadDir)

Step 7: import the csv to R

googletrends_data=readGT(filePath)

In this post, I write about how to merge daily data from Google Trends into longer time series using R.

Converting Google Trends weekly data into a regular date in R

Getting dates in the right format is always a big headache when getting your data ready for analysis. Google Trends provides data on three levels, monthly, weekly or daily. Here, I explain how to convert Google Trend's weekly dates into R's date class.

The weekly date format used by Google Trends looks like this:

2004-01-04 - 2004-01-10

This might pose a problem for instance if we want to plot the data. We will need to convert the date interval provided by Google Trends into a single date. We can do so using the following code:

#First, import the data 

data=read.csv(filePath, header=F, blank.lines.skip=F)

#Then select the ending date of the date interval

data[,1]=sapply(data[,1], substr, start=14, stop=30)

#And convert it into a date 

data[,1]=as.Date(data[,1], "%Y-%m-%d")

And then you have a date format that you can use for plots or data analysis.

Creating your own Google Trends plots in R

With this post, I want to demonstrate how you can use the R functions I've built to create your own Google Trends graphs.



First, install the functions by pasting this code into R.

You now have the following functions at your disposal:

• URL_GT(keywords)
• downoloadGT(url, downloadDir)
• readGT(file path)

By running the following code, you will get chart above.

#downloadDir=Where you save the csv from Google Trends
downloadDir="C:/downloads"
setwd(downloadDir)
keywords=c("Samsung", "Apple", "Nokia")
url=URL_GT(keywords)
filePath=downloadGT(url, downloadDir)
smartphones=readGT(filePath)

library(ggplot2)

ggplot(smartphones, aes(x=Date, y=SVI, color=Keyword))+geom_line()

Two methods for combining daily Google Trends data into longer time series

There are two methods for combinging daily Google Trends data into time series longer than the 90 days provided. We can either start from the weekly data provided for 2004-present and add the daily data in between the gaps, or we can combine the daily time series based on their percentage change. The results will be markedly different. The percentage change is complicated by Google Trends going to zero at times. To correct for this, we must assign some value to the zeros, which skews the data- In order to adjust for this the natural logarithm is applied.

Another option is to adjust the daily data based on the weekly time series. This creates a graph that looks more like the weekly data. This is done by dividing the daily search volume by the previous available weekly search volume.

The scatterplot between the two variables shows that they are correlated, but that there are significant differences.

R functions
R code to execute functions

Daily data

 Log(daily data)

 Percentage change

Log(percentage change)

 Weekly + daily data

Scatterplot of log(variables)

Intercept:-0.17***
Log(weekly adjusted):0.18***

Scatterplot without outliers (no logarithm applied)

Intercept: 0.93***
Weekly adjusted: 0.04***

*** Significant at 0.1% level

Conclusion

It seems that starting from the weekly time series creates a time series that is better behaved, with a distribution closer to the normal distribution. Since we cannot say what the true distribution of the underlying sample is, we must choose the method based on what phenomena we think the search data represent. It could for instance be argued that search data represents Internet users attention towards a particular search term. The method for calculating the daily search volume should then be chosen based on which method corresponds most closely to what we believe is the actual attention of Internet users.

It could for instance be relevant to identify a particular topic where we have good data on attention already, and then measure how it corresponds to the search data.