Rank and Rate...managing numbers...Kind of?

[Shadowtron]

I like the idea of adding more numbers to work with, but in practice you'd still see BD 19.95, Cadets 19.90, Crown 19.85 etc. A bigger factor in slotting is the aversion to one person "making a call" and putting a decent gap between 2 competitive teams.

[ScoutMello]

It seems to me the constraining factor is the language defining the "boxes"

100 points is enough to use for judging.

I think this is getting to the heart of it. There's no reason to have a categorization system with 5 boxes if everyone ends up in the top 2. Expectations need to shift to spread things out more. Unfortunately, I think the real effect of doing something like that is that instead of opening up the spread between corps, it would just open up the spread between clusters of corps. Perhaps we just need to accept that the top groups perform at a very close competitive level, which means they will be scored closely with a lot of tie potential.

[GUARDLING]

I think this is getting to the heart of it. There's no reason to have a categorization system with 5 boxes if everyone ends up in the top 2. Expectations need to shift to spread things out more. Unfortunately, I think the real effect of doing something like that is that instead of opening up the spread between corps, it would just open up the spread between clusters of corps. Perhaps we just need to accept that the top groups perform at a very close competitive level, which means they will be scored closely with a lot of tie potential.

actually,your last line is very true BUT it's like that from top to bottom.......called neighborhooding...and there's at least 5 or 6 of them

[soccerguy315]

pretty sure I've seen some captions out of 200 in band circuits (US Bands?) but it really doesn't matter b/c instead of a 15.6/20 you just get a 156/200, and there is still no difference that you can articulate between a 165/200 and a 167/200.

maybe it would be better if the top 3 corps step on the field the first show and score a 40 instead of a 70+?

Edited August 13, 2013 by soccerguy315

[ScoutMello]

actually,your last line is very true BUT it's like that from top to bottom.......called neighborhooding...and there's at least 5 or 6 of them

Yup. That's what I was referring to with 'clusters of corps'. So how do we break up the neighborhoods? More boxes? I don't think that would actually fix the problem, just make smaller clusters.

Brainstorm time. What if we start reevaluating basic assumptions? There's a lot of effort going into the numbers, but aren't they kinda arbitrary anyway? What if we threw out the numbers completely, and just used rankings? Instead of a judge deciding if a given performance was a 97 or a 98 out of 100, they just have to decide "Was group A better than group B?" and rank accordingly. Ties could still be possible, if the judge can't put one corps ahead of another.

Upsides:

• No effective ceiling on scores. Everything is relative to the other corps performing at the show and there's always room for the next group on the top. No performance is perfect, simply the best of the night.
  
• Judge decisions are very easily readable. Instead of "Why did they give Crown a 92 and BD a 98?" It's simply "The judge thought these 5 groups were better than Crown"
  
• Harder for a single judge to tank a groups overall score.
  
• Pretty much what's already happening anyway. Just less obfuscated by numbers.
  

Downsides:

• No granularity. No real difference between "slightly better" and "a lot better".
  
• No scores to really carry from show to show. Just rankings. (Some may argue thats a good thing)
  

And just for giggles, I worked up an example using the effective rankings from Finals:

https://docs.google.com/spreadsheet/ccc?key=0Alp8hKUs7sdSdFo0eE1XOGlpdnBxa0tQZWQzanJaZEE&usp=sharing

Details:

• Took the ranking from each judge in each sub-caption.
  
• Judge totals are averages of their two numbers
  
• Caption totals are averages of the judge totals.
  
• Overall score is an average of the 8 captions. (this ensures the same weighting as the current system: 20/30/30 GE/Visual/Music)
  
• Final number is a calculated rank. Lower numbers are better. A corps that sweeps all sub-captions would end up with a 1 here.
  

The results are effectively identical to the actual outcome of finals, with the exception of Boston and Cavies who switch under this system. (Notably, they were the closest scoring corps with only a tenth between them)

Tear my idea apart!

[Kamarag]

I have to say, even though there are obviously differing opinions on what actually happens, what should happen, and some pretty intriguing ideas on possible changes, this has been one of the more cool and interesting discussions on the DCI forums in a long while.

The one thing I think we're all in agreement on is that there needs to be more transparency in the judging process in DCI (and DCA for that matter). DCI has made the sheets publicly available, which is a great start, but I would LOVE to see video from the judging conclave and more insight into the process. There would be some push-back, but I think DCI should have short interviews with members of the panel immediately following regional. Heck, make them part of the live stream before or after scores are announced. Then drop them on the FN as a wrap-up.

Another interesting thing would be to get a camera crew involved in post-show critiques. I think fans would be very interested in the process of how that works.

Edited August 14, 2013 by Kamarag

[skywhopper]

Brainstorm time. What if we start reevaluating basic assumptions? There's a lot of effort going into the numbers, but aren't they kinda arbitrary anyway? What if we threw out the numbers completely, and just used rankings? Instead of a judge deciding if a given performance was a 97 or a 98 out of 100, they just have to decide "Was group A better than group B?" and rank accordingly. Ties could still be possible, if the judge can't put one corps ahead of another.

Jeff Fiedler proposed this method in 2007. It failed with a unanimous negative vote in the instructors' caucus.

[ScoutMello]

Jeff Fiedler proposed this method in 2007. It failed with a unanimous negative vote in the instructors' caucus.

Interesting, that totally flew under my radar. Was there ever any discussion of the concerns surrounding the proposal? Searching here isn't turning up much.

[Jeff Ream]

I like the idea of adding more numbers to work with, but in practice you'd still see BD 19.95, Cadets 19.90, Crown 19.85 etc. A bigger factor in slotting is the aversion to one person "making a call" and putting a decent gap between 2 competitive teams.

actually adding the .05 option can actually help keep that one person from calling the show

[Jeff Ream]

Interesting, that totally flew under my radar. Was there ever any discussion of the concerns surrounding the proposal? Searching here isn't turning up much.

they wanted more than a 1,2,3,4 etc in placement