Rep. Vs. Ach. scores

[crfrey71]

This has bugged me for years and years.

The main problem isn't the two things that are being judged. I'd generally agree that rep is a judgment of the staff quality, and that's a dollar-for-points debate. I'll leave that alone. The main problem with this system is that it is designed and presented as two independent scores. I can give you an 8.8 in rep and a 6.2 in ex. Right? No. It's not right. That's because well over 90% of all ex scores are within -0.4 -> +0.1 of the rep score.

Why is this bad? Aren't they related? Well, they are by definition, but they aren't generally presented that way. If I said you got an 8.8 in rep, and 8.6 in ex, you wouldn't notice. If I told you that you got a 17.6 with a -0.2 in execution, you'd also be saying the same thing. This is why execution isn't half the score. It accounts for only about a single point out of 20. Yes, 5% of the score is execution.

The "rep" is the 95%. This is why drum corps is so hard to get used to. It's a system where we all kid ourselves into thinking it's sort of half content, half ticks. It's not. It's a constant evaluation of what is possible in the potential of the show, and then the minor tweaking of the show based upon the performance at hand. DCI is won and lost on Pyware and Sibelius.

I'm not suggesting that the criteria is wrong on its own. What I can tell you is that it's my personal belief that the scoring of each of the two categories by the same judge each evening, and the associated consistent pattern in which it is applied, along with the definitions, application, anecdotal evidence, statistical analysis, and simple observation will tell you is that the problem is the system itself. (Sorry; couldn't break that sentence up.) You're asking people to evaluate one half, and then based on it, evaluate the other half. Plain and simple, if the deviation from the first half is small, the ACTUAL effect on scores of the second number is very, very minimal.

I'm sure someone will take issue with how I've described the process of arriving at the two sub-scores. That's fine. As long as one is dependent upon the other, the dependent number is exceptionally less powerful.

Now, let me put on my helmet before my judge friends see this...

Well said.

[drumcat]

Well said.

Thanks Frey.

Just so I don't sound way off base... numerically, it's a 5% score. In reality, my perception is that execution drives between 15-25% of split scores... specifically, the old adage that "if I can't read it, I can't credit the design". I figured I better put that in there for posterity. :)

[crfrey71]

Thanks Frey.

Just so I don't sound way off base... numerically, it's a 5% score. In reality, my perception is that execution drives between 15-25% of split scores... specifically, the old adage that "if I can't read it, I can't credit the design". I figured I better put that in there for posterity. :)

Then they need to revamp how they score big time. Content, Achievement, Repertoire are easy terms to understand. They are clear-cut in meaning. No ambiguity. I don't buy into that, ""if I can't read it, I can't credit the design" nonsense as far as Content/Rep. Does that not apply to GE. Isn't that where reading the effectiveness of the show design comes into play?

I'm a brass guy. So let's take that for example. Between content/rep. and achievement, it should be, "What's the degree of difficulty as far as the content of what they are playing?" "How well are they playing those notes?" "How musically are they playing those notes?" "How does the hornline sound as a whole?" "What is the overall quality of sound from the hornline?" "Do I hear all the voices"...

[superOOk]

Dude the entire judging system is bonkers. And you are trying to make sense of a senseless scoring system. Just enjoy the show!

[Jeff Ream]

That might make sense from a visual standpoint, but a music book is a music book.

nuances and details will become more readable as performance cleans up. just putting notes on the paper doesnt make it hard. playing well on the move makes it hard

[brichtimp]

nuances and details will become more readable as performance cleans up. just putting notes on the paper doesnt make it hard. playing well on the move makes it hard

Thus, you have The Cadets :-)

[Cavfan930]

I'm studying math/stats at college right now, and what's always bugged me about the system in DCI is the immense crossover between captions that are weighted differently. Last season, I kept a personal series of spreadsheets for my five favorite corps, with each competition they were in on the left hand column, and each caption/sub-caption across the top (like an expanded recap, but for only one corps across a full season). This is back with the GE Music and GE Visual (When I say it like that, it sounds so long ago). I figured, like many posting here have said, that as the shows cleaned musically and visually, the GE would rise accordingly. Because one of the GE captions was all about the effect of the music, how well the corps played their instruments ought to have a direct causal relationship with that caption. The same for visual. What I discovered was that, besides a general positive trend of y=x for the corps overall score, the plots relating the music and GE music looked totally different! About half the time you'd see decreases (small, but in the negative direction nonetheless) in the majority of the music sub-captions, but an increase in the ge music score.

(On a similar note, it bugs me that you pretty much never see the achv surpass the cont/rep.)

It was maddening to see, so I chose not to do that this year. Of course, they also changed the sheets this season anyways, so I can't really assert either way that the same would continue to exist. (Are GE 1 and GE 2 the same, or is one more considerate of music and one more for visual?)

From a statistical analysis point of view, because there is an inherent relationship between all the captions but the GE is weighed more heavily than the scores it should be dependant on, absolutely no value can be placed in saying that X corps beat Y corps by less than 4 points (arbitrary number, it's probably not 4). Statistically, I would look at those scores, consider things like variance, standard deviation, etc. and be forced to conclude that the difference between them is not meaningful. In short, you end up with 5 champions instead of one when the normal conventions of stats and math are observed in considering the scores.

Something else that bugs me is that at the regionals and throughout finals week, it is never the same panel every night. If you want to be able to say, as objectively as possible, that the scores from these events are meaningful representations of the corps ability to perform a show better than everyone else, then you've got to have the same panel in the same positions at each of the big contests. Although, if you really want to rigorously decide who's the best, then you've got to force each corps to perform exactly the same show, in the same uniform. That's really the only way to isolate a variable in this activity and then evaluate it.

In conclusion, from an academic point of view, we would be better off if ALL the judges cast a simple vote for the best corps, and then do likewise for things like best percussion and brass. No more recaps filled with weird decimal points. Because honestly, how many times have you seen a corps clearly outperform someone else's brass, but then the recap says they only beat them by a tenth? It's ridiculous.

Edited August 4, 2014 by Cavfan930

[MikeD]

That might make sense from a visual standpoint, but a music book is a music book.

Yes, but as a show gets cleaner, a corps with a higher level of Rep will be able to score higher in that area than a corps with a lower level. The top ends of the Rep between the corps are not the same as the Ach level increases.

[MikeD]

(On a similar note, it bugs me that you pretty much never see the achv surpass the cont/rep.)

On the Sat Allentown recap, of 60 sheets where that might have happened, it happened nine times (visual and music captions)...only twice on Friday. Less than 10% of the time, in other words.

[Cavfan930]

On the Sat Allentown recap, of 60 sheets where that might have happened, it happened nine times (visual and music captions)...only twice on Friday. Less than 10% of the time, in other words.

Across the whole season and across all corps, I'd say it happens less than 3% of the time. But I'm not going to comb through all the recaps to find out.