DCI 2015 Predictions Using Machine Learning

[FTNK]

maths

[dalyea]

A couple of folks have tried to do this in the past (I very simplistically tried neural nets a decade ago), and this is easily the most comprehensive modeling job in terms of the choice of variables and the ensemble approach.

I couldn't tell from the output - were the estimated coefficients for regression and GBM interesting (I don't know anything about GBM, but I assume the output is similar to GLM)? Did they give any insights on what "show" is the most influential on the finals week outcome? I guess at 21 days out, that'd be a difficult thing to nail down, though I suspect regional performance matters.

There is an H20 library function called deeplearning now, which is basically neural networks at hyperspeed! H20 has implemented deeplearning to use dropout, which is akin to regularization in some ways, but is more aptly described as multi-modeling to avoid overfitting. I'm keeping that in my hip pocket for future modeling. :-)

GBM in a nutshell is tree-based learning, but with a twist. For classification, at each step of defining what the eventual tree will look like, the decision rule is a stump. That is, for one feature only, the model finds a split value for it. The first stump is chosen to classify the data correctly as best as possible. Then, for the next stump, the split is determined by, as before, trying to classify as many items as possible correctly, but the loss function this time weights the misclassified items by the first stump more heavily. The second stump boosts the treatment for misclassified items. This system of boosting to account for misclassifications continues with subsequent stump decision boundaries.

One advantage of GBM is that it tends to not overfit the data. If you look at RMSE for tighter and tighter training data fits, the test data RMSE tends to decrease in lockstep and eventually flatten out - without rising as would be seen with overfitting by other modeling techniques. For those reasons, I used two GBM models in my approach for predicting finals scores this year.

The GBM library in R then has a build in varimp function to tell us, which of the features was most important when defining all those stumps? As I'd expect, S5, the most recent score, was most important. Surprisingly, S1 or S2 was often 2nd most important. The reason I think this is the case is that S1 and S5 values implicitly define the scoring slopes that are directed toward the target final scores. I'm not sure that S4 and S5 would always provide as much pair-indicated directional guidance toward the finals scores. Presumably, variable selection or variable importance for linear models may also favor S5 and S1/S2 as most relevant.

[dalyea]

I've heard it my entire career:

"Past performance is not indicative of future returns"

It is interesting that your predicted results are close, but I wonder if your placement results could have been as accurately predicted by "random walk" analysis.

Fun exercise. Nice work, but I think you lost most readers after the first paragraph.

And, as much as I love PR, I'll be disappointed if they are the "surge" corps that knocks BK down a notch.

I'd have to think about how a random walk approach might lead to predicted scores. I get the thought - I was once working on price predictions and, at some point, I realized that using random walks as a form of simulation would get me the results I needed. Though now, I'd likely use bootstrapping instead - I didn't know about that technique back then! Thanks for reading my predictions and for the idea.

[dalyea]

I noticed this the first time I saw it, but aren't the corps Predicted placement v. Rank flip flopped, for 2015?

In a judged activity where cleanliness is not the only factor for scoring, I'm not sure about the results.

Done quite a bit of math in my day, kind of skeptical on some of these fronts but cool analysis on stuff I had always wished I had tried to put together.

If I reversed some numbers, let me know and I'll make the correction. I'll doublecheck my posted 2015 ranks, but I don't recall them being reversed.

[dalyea]

One thing I noticed while doing analysis, but didn't have the time to look into until now, was that all my 2015 scoring predictions based on 2005-2014 training data were unusually low. For other years where I back-tested, my final model scores predicting winners' scores between 97 and 99, which is in line with usual scoring at Saturday finals. The highest predicted scores I was getting this year, from any of the 4 sub-models, was in the 94 range. Why is that the case?

The plot below shows that scores this year are -4 points behind scores at the same point in the season, from -30 to -21 days removed from finals, than ALL other years 2003 to 2014. Each set of colored dots are the 5 scores for the champion drum corps from 2003 to 2014 - I've noted that the peak score was an S4 score at T-22 days from finals for the Blue Devils in 2010. The 3 lines below the dots are the 5 scores for the top 3 drum corps for 2015, as of T-21 days from the 8/8/2015 finals.

For whatever reason, this is a low scoring year. Were there rules or judging changes? I haven't followed drum corps news all summer, so perhaps I'm not aware of some change that would result in scores being, relative to the past 12 years, so low.

What I would find troubling at this point is to see that finals scores rocket to 98+ as semis and finals next week. Certainly, every corps is working hard to perfect their show, and performance levels and scores will rise. However, there is no reason to think that corps will work harder or improve more in the final 21 days before finals this year than in any of the previous 12 years. From what we see in the scores so far this season, if the winning drum corps finals score is much over 96.0, it appears that would be out of line (regression humor?!) with clearly delineated scoring trajectories from the past 12 years. If we assume that the judging criteria and the drum corps' final polishing up their shows remain the same as in all previous years, the scoring trends this year clearly point to a champion score of about 96.0.

[BRASSO]

There is an H20 library function called deeplearning now, which is basically neural networks at hyperspeed! H20 has implemented deeplearning to use dropout, which is akin to regularization in some ways, but is more aptly described as multi-modeling to avoid overfitting. I'm keeping that in my hip pocket for future modeling. :-)

GBM in a nutshell is tree-based learning, but with a twist. For classification, at each step of defining what the eventual tree will look like, the decision rule is a stump. That is, for one feature only, the model finds a split value for it. The first stump is chosen to classify the data correctly as best as possible. Then, for the next stump, the split is determined by, as before, trying to classify as many items as possible correctly, but the loss function this time weights the misclassified items by the first stump more heavily. The second stump boosts the treatment for misclassified items. This system of boosting to account for misclassifications continues with subsequent stump decision boundaries.

One advantage of GBM is that it tends to not overfit the data. If you look at RMSE for tighter and tighter training data fits, the test data RMSE tends to decrease in lockstep and eventually flatten out - without rising as would be seen with overfitting by other modeling techniques. For those reasons, I used two GBM models in my approach for predicting finals scores this year.

The GBM library in R then has a build in varimp function to tell us, which of the features was most important when defining all those stumps? As I'd expect, S5, the most recent score, was most important. Surprisingly, S1 or S2 was often 2nd most important. The reason I think this is the case is that S1 and S5 values implicitly define the scoring slopes that are directed toward the target final scores. I'm not sure that S4 and S5 would always provide as much pair-indicated directional guidance toward the finals scores. Presumably, variable selection or variable importance for linear models may also favor S5 and S1/S2 as most relevant.

I was curious... In your methodology utilization how much consideration in your modeling have you given to the floccinaucinihilpilification factor. What is the Standard Deviation Probability computer modeling of this factor look like at the moment, dalyea ?

[Clutchtow]

I was curious... In your methodology utilization how much consideration in your modeling have you given to the floccinaucinihilpilification factor. What is the Standard Deviation Probability computer modeling of this factor look like at the moment, dalyea ?

Okay now I'm like 90% certain y'all are just making up words

[Northern Thunder]

how much consideration in your modeling have you given to the floccinaucinihilpilification factor.

Isn't this a come-on line that guys use to make porn videos?

[BRASSO]

Okay now I'm like 90% certain y'all are just making up words

Actually, no.

floccinaucinihilpilification is a real word. I utilized the word purposefully as it seemingly provides both the required gravitas befitting this thread, while simultaneously providing an apt description perhaps of where the poster's methodology here might lead us to what may be derived from this scientific exercise.

For a better understanding perhaps.. posters here could look up the definition of " floccinaucinihilpilification " and see how it just might apply here, imo.

Edited July 31, 2015 by BRASSO

[FanLib]

CADETS are done , Was just a matter of time till other corps matched their cleanliness.

Middle of that show is terrible and slow.

Colts will not beat BOSTON , no brass , Bad guard , Bad show .

w/r/t Colts: Seriously? Bad guard? Bad show? No brass? I think many would disagree with you, including me. I'm hoping they surge and beat BAC. Like the Cinderella team in the final 4 kind of thing. Doesn't DCI need a story like that?