I recently wrote up, and submitted to the Social Science Research Network, a study of the Central England Temperature series that I did a couple of years ago and recently updated. You can access the paper at the SSRN here: A Seasonal Autoregressive Model for the Central England Temperature Series.

What I do in this analysis is build a seasonal autoregressive model for what is the longest time series of directly measured temperatures available for analysis. This model includes a linear trend component. I fit the model for both the pre-Industrial Revolution Period (1659 to 1849) and the Post-Industrial Revolution period (1949 to 1999). In the former period no trend is apparent, in the latter period I find a warming trend of 1 °C/century. This result has a borderline statistical significance (less than 3σ). I use both models to forecast the temperatures for the 21st. Century and find that the older, trend free, model does a slightly better job out-of-sample than the newer model with the trend.

 

I put the most up-to-date version in Google docs. If you follow this link you should be able to get to it. I will update this from time-to-time.  

In the prior post we noted the outperformance of Support Vector Regression over OLS models out-of-sample. This is referred to in the Machine Learning community as their superior ability to generalize. I think that the enhanced statistical reliability coupled with the fact that the univariate response model found by the SVM departs highly from our prior prejudices regarding smooth and low order responses is quite a striking result.

In this post we seek to replicate the response function of the SVM with a high-order polynomial model. This is to investigate whether the superior forecasting skill out-of-sample arises from the lower-order “wriggles” in the SVM response function or from the higher-order “kinks.” This is interesting because we can certainly replicate the lower-order features via classical linear methods, but it is unlikely that we can do such a thing for the higher-order features of the response. Thus we define our linear polynomial model as

.

Here the P_n(x) are Legendre Polynomials of order n. These are orthogonal on [-1,1], so are a useful basis to express our response function. Because the functions are orthogonal, the estimators should be independent in expectation. In addition, following Vapnik, we reject the Occam's Razor driven methodology of standard classical statistical analysis to find a parsimonious model (what my ex-boss, Peter Muller, used to refer to as “Keep It Simple Stupid”) and find the N large enough to match the testing set R² of the SVM.

The above chart illustrates a 30th order Legendre polynomial model replicating the response of the Support Vector Machine and exhibiting an equivalent out-of-sample forecasting skill. From the point-of-view of classical inference, there is no way an analyst would ever suggest using such a high order model on this data, and the t-statistics for the β_n coefficients are all small, yet this is the type of functional response picked out by the Support Vector Machine!

 

Continuing the recent theme on the application of Machine Learning and Interior Analysis, here we investigate the utility of Support Vector Regression methods versus Ordinary Least Squares. The job is to predict the daily range of the top ranked stock in the Compact Model Portfolio from the prior value of that metric. By Daily Range we mean the ratio of the difference between the Closing Price and Opening Price to the difference between the Highest Price and Lowest Price. I chose this particular metric because it is an interior metric but it doesn't use any microstructure information. It is also frequency discussed in non-academic literature.

The data used is the daily range computed for the top ranked stock in the Compact Model Portfolio, with the period 01/03/2001 – 12/31/2009 use as the training data and the period 01/03/2010 – 08/24/2010 used as the testing data. This division is a simple binary sample cross-validation technique. The analysis was performed in R, and the code is appended to this post.

Before discussing the chart above, which exhibits many interesting features, let's talk about the methods. On the training set, the ksvm procedure was used to execute an ε-insensitive regression and allowed to use it's default methods and tolerances. The OLS procedure lm was similarly run without user tuning. I then used both models to predict responses in the testing set and used an OLS regression of the response onto the forecasts as a simple methodology for evaluating the quality of the systems. Notable differences were found. The out-of-sample β was established to be 2.01417 ± 1.30812 for the OLS model and 1.04366 ± 0.43193 for the SVR model. The R²'s were 0.01526 and 0.03676, respectively. Thus the SVR model is a much more accurate performer out-of-sample, as it is advertised to be.

The chart exhibits the out-of-sample predictor and response data as well as the OLS regression line and the SVR model. We see that the SVR model contains numerous wriggles and kinks, yet my instinct is to reject the information content of these features — making the assumption that they indicate a need to tune the kernel used by the system. However, intuition is not necessarily truth, so we are in need of a procedure to establish where the superior predictive power of this model comes from. Does it come from some simple non-linearity in response that the algorithm has picked up — or is it actually due to the more funky nature of the model. One way to establish this would be to see if we can create some kind of piecewise linear model that does as well as the SVR.

---

require(kernlab)
training<-read.table("CMP_Training.txt",header=TRUE)
testing<-read.table("CMP_Testing.txt",header=TRUE)
training$DailyRange<-(training$ClosingPrice-training$OpeningPrice)/(training$HighestPrice-training$LowestPrice)
training$PriorDailyRange<-(training$PriorClosingPrice-training$PriorOpeningPrice)
    /(training$PriorHighestPrice-training$PriorLowestPrice)
testing$DailyRange<-(testing$ClosingPrice-testing$OpeningPrice)/(testing$HighestPrice-testing$LowestPrice)
testing$PriorDailyRange<-(testing$PriorClosingPrice-testing$PriorOpeningPrice)
    /(testing$PriorHighestPrice-testing$PriorLowestPrice)
names(training)
training$sample<-!(is.na(training$DailyRange)|is.na(training$PriorDailyRange))
testing$sample<-!(is.na(testing$DailyRange)|is.na(testing$PriorDailyRange))
summary(linmod<-lm(DailyRange~PriorDailyRange,data=training,subset=training$sample))
summary(lm(testing$DailyRange~predict(linmod,newdata=testing)))
print(svrmod<-ksvm(DailyRange~PriorDailyRange,type='eps-svr',data=training))
summary(lm(rest(testing$DailyRange)~predict(svrmod,newdata=testing)))
isort<-order(rest(testing$PriorDailyRange))
plot(testing$PriorDailyRange[testing$sample],testing$DailyRange[testing$sample],axes=TRUE,
    xlab='Prior Daily Range',ylab='Daily Range',main='Comparison of SVR and OLS Models for Daily Range',
    sub=paste('Data: Compact Model Portfolio, Top Ranked Stock; Resolution: Daily; Training:',
    training$MarkDate[1],'--',training$MarkDate[length(training$MarkDate)],'Testing:',testing$MarkDate[1],
    '--',testing$MarkDate[length(testing$MarkDate)]))
lines(c(-1.1,1.1),c(0,0),col='gray')
lines(c(0,0),c(-1.1,1.1),col='gray')
abline(linmod,col='red')
lines(rest(testing$PriorDailyRange)[isort],predict(svrmod,newdata=testing)[isort],col='blue')

 

To learn more about the asymmetric relationship between empirical volatility and market returns, I thought we should study more data and more asset classes. To augment this analysis, however, I thought it worthwhile to introduce a more suitable parameterization than that available with the standard GJR AGARCH model. Let's call this Piecewise-Quadratic GARCH, or PQGARCH.

I fitted this model to daily interest rate changes of U.S. Three Month Treasury Bills. The data I used is available from the Federal Reserve Bank of St. Louis. The model was fitted independently to each year's daily data, from 1954 to date. This yields an annual time series of parameter estimates D_y and E_y. Our null hypothesis is for regular (i.e. symmetric) GARCH, which implies that the D's and E's should agree within sampling errors. The chart below illustrates the empirical distribution functions for both data series and the results of applying the Kolmogorov-Smirnov test to that data; which implies that we cannot reject the null hypothesis with a confidence greated than 82% — which is insufficient. (Years in which the regression did not converge without “tuning” are omitted.)

This demonstrates that asymmetric response does not appear to be a characteristic of U.S. interest rate changes over more than half a century of data. i.e. It is an attribute of stock markets not of all markets.

 

In earlier posts we studied the relationship between the VIX Index and GARCH Models of the daily volatility of the S&P 500 Index. We found a quadratic relationship between the log of the VIX and the log of the GARCH model.

However, we also found considerable support for completely asymmetric response between empirical volatility and market returns. When we properly incorporate this structure, we find the curvature in the relationship between the VIX Index and the GARCH model is removed, as is illustrated in the above chart.