Rank and Rate...managing numbers...Kind of?

[ShortAndFast]

I have to say, even though there are obviously differing opinions on what actually happens, what should happen, and some pretty intriguing ideas on possible changes, this has been one of the more cool and interesting discussions on the DCI forums in a long while.

The one thing I think we're all in agreement on is that there needs to be more transparency in the judging process in DCI (and DCA for that matter). DCI has made the sheets publicly available, which is a great start, but I would LOVE to see video from the judging conclave and more insight into the process. There would be some push-back, but I think DCI should have short interviews with members of the panel immediately following regional. Heck, make them part of the live stream before or after scores are announced. Then drop them on the FN as a wrap-up.

Another interesting thing would be to get a camera crew involved in post-show critiques. I think fans would be very interested in the process of how that works.

I would really like to see this. DCI should make the judging as transparent as possible: post some tapes, do video interviews with judges. It took them an eternity to get the sheets posted where anyone could read them.

I think their fear is that some of the judging won't stand up to that kind of scrutiny.

[OrlandoContraAlum]

Didn't they win field brass with a 9.9 with six corps left? Now that was a ballsy move from the judge, saying only one group has a remote chance. Wow!

Yep.

[Pete Freedman]

So I'm getting the idea here...

It's really hard to rate drum corps. If the judges just rate them (as I proposed) then the accuracy drops, because it's just too hard to do. By ranking first you at least get the order right, and also have a range to base the scoring on. (While one corps may perform radically differently than expected, overall they will do slightly better than they did last time out, so you've got a baseline.)

Take a look at this picture:

Nice picture, eh?

There's a point. If I were to assign half of you to rank order these trees by height, and time yourselves doing it, it would be a straightforward project. Yes, there would be some tough calls, but you could do it.

If I were to assign the rest of you to ignore their relative heights and just indicate their actual heights, (and time it) it would take much longer. Even if you knew the fence is exactly six feet tall, it would still be very hard. Worse, the results would be less accurate in the sense that you might put trees higher than others that obviously are higher.

In other words, pure rating leads to ranking errors.

Thus the method "Rank then rate," is a way of ensuring that the rankings are correct even if the ratings are not. But the phrase may have different meanings to different people.

I would argue that it's still not justification to put an inaccurate rating just to ensure there are no ties. If, for example, you are limited to 1 foot increments, then the correct thing to do is put tree 1 and tree 2 the same. Maybe tree 3 is one foot lower, and tree 4 is one foot higher. Maybe all four are the same within one foot. Whatever. Leave it that way! Because someone else is judging the width of the trees, and someone else is judging how green they are. By exaggerating the spread to avoid ties, you're messing up the overall score.

[Jeff Ream]

And my immediate reaction to that is to ask: Why? What does the number itself provide that is useful to an instructor, designer, or performer? Do the numbers themselves provide any kind of actionable insight? How does the number itself inform decisions? What are the practical ramifications of removing the score component?

I can think of an answer to that myself, although I'm not convinced it's enough: Numbers provide a sense of progress. With a scoring system, the members see the numbers keep going up, and even though their ranking may not change, the fact that the numbers get bigger gives the impression that their work is being rewarded. If an ordinal system is used, a corps that stays second place all season gets the false impression of stagnation. Now, we all know you need to work hard and continually improve just to hold your place in the rankings, so that's a false impression. But it does exist.

But, the scoring system can also generate false impressions. We've all been at that show where the numbers seemed to inexplicably drop, and members take that as a sign that something went wrong, when really it was just a read from a different panel, or some other factor that has nothing to do with the performance itself.

why? Well look at Crown this year as a perfect example.

If you win almost every sheet, but have one low ordinal, you could probably still win.

But in this system now, if you win almost every sheet, but have one caption that's killing you, you could lose the show.

So, do you reward people for having a good ranking, and ignore the rating?

[Jeff Ream]

I would really like to see this. DCI should make the judging as transparent as possible: post some tapes, do video interviews with judges. It took them an eternity to get the sheets posted where anyone could read them.

I think their fear is that some of the judging won't stand up to that kind of scrutiny.

I think it may expose those with perceived biases. But I trruly feel, especially for GE, it would help people really understand that GE is more than fans throwing burning babies on the field

[ShortAndFast]

So I'm getting the idea here...

It's really hard to rate drum corps. If the judges just rate them (as I proposed) then the accuracy drops, because it's just too hard to do. By ranking first you at least get the order right, and also have a range to base the scoring on. (While one corps may perform radically differently than expected, overall they will do slightly better than they did last time out, so you've got a baseline.)

(snip)

In other words, pure rating leads to ranking errors.

Thus the method "Rank then rate," is a way of ensuring that the rankings are correct even if the ratings are not. But the phrase may have different meanings to different people.

I would argue that it's still not justification to put an inaccurate rating just to ensure there are no ties. If, for example, you are limited to 1 foot increments, then the correct thing to do is put tree 1 and tree 2 the same. Maybe tree 3 is one foot lower, and tree 4 is one foot higher. Maybe all four are the same within one foot. Whatever. Leave it that way! Because someone else is judging the width of the trees, and someone else is judging how green they are. By exaggerating the spread to avoid ties, you're messing up the overall score.

I see your point here - small differences are inflated by ranking with no ties. It's exactly the same problem as the ordinal system discussed above. Suppose you have 4 brass lines of very similar ability, and the judge assesses all of them as high box 4. That's likely to give you something like

A 17.9

B 17.8

C 17.6

D 17.5

But remember, they were all very similar. So by ranking, you've effectively rated D as 0.4 below A and that's not correct.

You are obviously very comfortable with the idea that the judge could tie the 4 lines if they differ by less than a tenth. But I think most in the activity are not comfortable with it - DCI hires Jeff Prosperie to judge finals because the stakeholders trust him to find and weigh the minuscule differences between Cadets, BD and SCV and declare a winner. The bottom line is that the competition drives excellence, and to have meaningful competition you must have a winner - not just several groups that are "close enough".

After reading this thread, I think an extra decimal point would do wonders to minimize the distortion from ranking close groups. I'd be happiest if the brass judge above could say

A 17.90

B 17.89

C 17.85

D 17.80

All 4 corps are now rated correctly, their spreads indicate how close they were, and they are still ranked correctly from best to worst.

[cixelsyd]

There's a point. If I were to assign half of you to rank order these trees by height, and time yourselves doing it, it would be a straightforward project. Yes, there would be some tough calls, but you could do it.

If I were to assign the rest of you to ignore their relative heights and just indicate their actual heights, (and time it) it would take much longer. Even if you knew the fence is exactly six feet tall, it would still be very hard. Worse, the results would be less accurate in the sense that you might put trees higher than others that obviously are higher.

First, can I see a recap with the height numbers the last judge came up with?

[ScoutMello]

why? Well look at Crown this year as a perfect example.

If you win almost every sheet, but have one low ordinal, you could probably still win.

But in this system now, if you win almost every sheet, but have one caption that's killing you, you could lose the show.

So, do you reward people for having a good ranking, and ignore the rating?

If one caption is killing you, it'll show in the ordinals as well. The only way that's not the case is if that caption is killing everyone else as well. Assuming you're referring to Crown's percussion score, that did affect the ordinals as they were 6th in percussion and still won.

[Pete Freedman]

...

You are obviously very comfortable with the idea that the judge could tie the 4 lines if they differ by less than a tenth. But I think most in the activity are not comfortable with it - DCI hires Jeff Prosperie to judge finals because the stakeholders trust him to find and weigh the minuscule differences between Cadets, BD and SCV and declare a winner. The bottom line is that the competition drives excellence, and to have meaningful competition you must have a winner - not just several groups that are "close enough".

After reading this thread, I think an extra decimal point would do wonders to minimize the distortion from ranking close groups. I'd be happiest if the brass judge above could say

A 17.90

B 17.89

C 17.85

D 17.80

All 4 corps are now rated correctly, their spreads indicate how close they were, and they are still ranked correctly from best to worst.

I like this solution, but I have to assume DCI doesn't do this for a reason, presumably that they feel it's not accurate at that granularity.

Still, if it's accurate for ranking purposes, then it could be allowed for ties only. That is, the judge can add the extra digit only to break a tie. That decimal still gets added to the aggregate score on the theory that the closeness of the two performances allowed the judge greater accuracy. Or, that decimal might not count on the theory that the additional accuracy it provides only applies to those two units, not in comparison to other units. Confusing, though.

Another way might be to allow judges to break ties with a +/- system. Thus, up to three tied units can be resolved that way (one gets neither + nor -). It doesn't affect the score (Aha!) EXCEPT to break an overall score tie.

If a fourth corps is involved in the tie, break out the ++. Or actually tie them, if they're really too close to call.

This would allow judges to say to instructors, "A was better than B by a skosh, even though I can't pinpoint their actual score at that granularity." Which is exactly what they should be saying, right?

[Kamarag]

While I cannot stand bottom line ties, and think judges should avoid sub-caption ties unless there's a really good reason, I have absolutely no problem with overall ties. If that's how the math adds up, so be it. To me there's nothing worse than a forced tie-breaker.

Just considering the three times we've had ties at the top, who was harmed by having co-champions?