Judging bias

[ordsw24]

Sorry if this has been asked/ clarified on here before. We all know "bias" exists to a point....each judge inevitably brings his or her own set of expectations, knowledge, perspective, interpretation of the sheets, etc. to the field. Totally get that.

I'm just wondering if DCI has any kind of tracking system and remediation in place for considerable fluctuations in scores from one judge to another, and in particular, if one judge differs from the general consensus of the rest of the judging panel in a particular caption on subsequent/ previous contests, placing them 1 or 2 spots lower.

Corps can have an off night, but the "content" caption should not fluctuate very much in one day, right? And I don't mean fluctuations in raw numbers, but in placement/ relational terms.

It's pretty easy to track these types of variations once you notice them (I'm a bit of a recap breakdown geek). There is one judge who is scoring one corps a lot lower than other judges in whichever caption he is judging -- and I'm not just talking a few tenths -- it's sometimes literally a point and half or more difference in just that one caption from one night to the next compared to another corps right next door in placement (scoring said corps lower and the other corps higher = 1.5 or more net difference). This judge may be perfectly justified in scoring the corps in question so low compared to other judges, but 1.5 in one caption is a huge difference from one night to the next, (again, comparatively speaking in relation to the corps it has been placing next to all season) so this discrepancy matters and possibly warrants a closer look imo.

It would seem a computer software program could track anomalies such as this. Any ideas if such discrepancies are tracked, and how DCI handles them?

Edit: I'm obviously not going to name the judge or corps in question (there may be and probably is more than just this one example), but one judge/ one corps across all captions he's judged stood out in particular to me as being lower than his judging peers.

Edited July 29, 2013 by ordsw24

[3rd Glasgow BB]

Theydo have the "CHSLC" tracking system (Caption Heads Shouting Loudly in Critiques) which although a little low tech is very effective at highlighting the discrepancies you speak of.

Edited July 29, 2013 by 3rd Glasgow BB

[Jeff Ream]

Content can change sure. Does every judge see the show the exact same way as the last guy? Look at exactly the same places or be on the field at the sane place at the same time?

No. So their view can be different

[barigirl78]

Sorry if this has been asked/ clarified on here before. We all know "bias" exists to a point....each judge inevitably brings his or her own set of expectations, knowledge, perspective, interpretation of the sheets, etc. to the field. Totally get that.

I'm just wondering if DCI has any kind of tracking system and remediation in place for considerable fluctuations in scores from one judge to another, and in particular, if one judge differs from the general consensus of the rest of the judging panel in a particular caption on subsequent/ previous contests, placing them 1 or 2 spots lower.

Corps can have an off night, but the "content" caption should not fluctuate very much in one day, right? And I don't mean fluctuations in raw numbers, but in placement/ relational terms.

It's pretty easy to track these types of variations once you notice them (I'm a bit of a recap breakdown geek). There is one judge who is scoring one corps a lot lower than other judges in whichever caption he is judging -- and I'm not just talking a few tenths -- it's sometimes literally a point and half or more difference in just that one caption from one night to the next compared to another corps right next door in placement (scoring said corps lower and the other corps higher = 1.5 or more net difference). This judge may be perfectly justified in scoring the corps in question so low compared to other judges, but 1.5 in one caption is a huge difference from one night to the next, (again, comparatively speaking in relation to the corps it has been placing next to all season) so this discrepancy matters and possibly warrants a closer look imo.

It would seem a computer software program could track anomalies such as this. Any ideas if such discrepancies are tracked, and how DCI handles them?

Edit: I'm obviously not going to name the judge or corps in question (there may be and probably is more than just this one example), but one judge/ one corps across all captions he's judged stood out in particular to me as being lower than his judging peers.

I don't know if DCI does it, but I've been tracking it by building my own database. It just takes time for me to enter the numbers every day. A pivot table allows me to cut the database various ways. I'm not doing it to the degree of including the content scores, though. Maybe next year.

Biases do exist, but one challenge to detecting them on an ongoing basis is that you have to get pretty far into a season to have individual judges with more than one instance of judging the same corps in the same caption. The judges list is large and DCI tends to rotate them among different captions.

If I can do it, it wouldn't be hard for someone at DCI to do it if they want to.

But, do they want to? I often wondered how judges get selected for the big competitions, but it seems to be done far ahead of time.

[drilltech1]

Just floating an idea, not fully formulated, for considerations.

Should content scores be removed from the sheet after the first regionals and corps be assigned a content rating for the rest of the season as content will not change dramatically unless they change songs or full arrangements?

Achievement would definitely fluctuate but maybe content would be given a fixed rating, similar to diving difficulty, figure skating difficulty, or State music evaluations.

[seepoo]

I have often wondered who is looking in on this. The question has been asked many times. What's wrong with DCI (drum corps)?

Many of us contend it's inept judging. Judges try to say they are not influenced by who is teaching a particular group or caption, OR, they are not looking at previous scores.

And we all know they do. It's pathetic.

[fsthnds]

And unfortunately, the judges who have been around the longest(15 to 20 years or more) are usually the most biased. They reward groups for things that were done in prior years instead of rewarding what was just presented to them on the field. It is time to replace these judges with newer judges who do not have old opinions or biases of the corps.

Maybe it is time to put term limits on DCI judges just like we have for the president of the US.

[Cleveland1]

Its a subjective sport with a subjective scoring system. There is no real right or wrong for the judges or for judging, at least for corps grouped together. Putting pioneer ahead of crossmen would be wrong. But lets focus on the top 4 corps, and some captions are currently within .4 from 1-4.

Who is better? Yes a judge can put them in order A and a different judge may put them in order B. But which is "right"? None. The reason we disagree with B is because we had A and it changed. Flip B and A and we would still be arguing.

Only way to really eliminate the fluctuations is to have the same judging panel all the time. Creating a system that adjusts a score if a caption is deemed to be scored differently than we expected is kinda silly to me.

Im sorry ______ but your score was too high because a judge rated you significantly higher in _____ and we at dci disagree with that judge.

That just wouldnt work, haha. There really is nothing to do. And apart from the corps at the cutoff for semis and finals, the only scores that really matter are finals. They could go the whole season without giving scores, but having judges give their insights at shows. And at prelims bring out a full judging panel. Same panel for 3 nights.

What we have is an imperfect system but i dknt see it being improved really.

We leave content to be judged once and we fall back to the same problem of "what if this judge was judging that night"

[crest99]

Its a subjective sport with a subjective scoring system.

This.

As a marching member, I remember my corps being on both ends when a judge went off the reservation. And I distinctly remember it altering placement for better and worse in some rather important shows (quarterfinals, for instance).

As a spectator, I have to admit I kind of enjoy seeing marks go all over the place every now and then. It preserves at least some semblance of the idea that drum corps is a live, spontaneous art-form about which people may form differing opinions.

I also have a feeling that any system enforcing judging consistency would only serve to reinforce the status quo in corps placements, keeping the haves at the top and making sure the have nots continue having naught. It would also work against some of the story lines we enjoy most each season: the little guy having an exceptional year, the surging dark-horse, and the half-mad half-genius mid-season rewrites.

[skywhopper]

An example that I think is interesting from the past three consecutive nights (ie, likely exactly the same book in each performance) in the brass caption top 3 (BD = Blue Devils, CC = Crown, CD = Cadets):

7/26 Murfreesboro:

BD: 94/90

CC: 93/92

CD: 92/91

7/27 Charlotte:

BD: 92/90

CC: 94/92

CD: 93/91

7/28 Atlanta:

BD: 91/92

CC: 93/92

CD: 93/91

So in Murfreesboro, the judge says BD has the best book, but the worst performance (and below book by quite a bit). Charlotte they get worst book/worst performance (but only 0.2 apart, which all three had that gap). And in Atlanta, they get worst book (below SCV even)/best performance (over book).

Now, my own feelings about BD's book is that they are challenging themselves to an extreme degree and while they're improving rapidly, they aren't yet quite playing it up to snuff, particularly tuning and range wise, so I would probably most agree with the judge for the Murfreesboro show. So what's the explanation for the flip-flop to Atlanta? How can their book-to-execution gap flip from under by 0.4 to over by 0.1 in two days while their book drops 0.3 real points and from 1st to 4th in relative terms?

Of course the real answer is sampling differences plus the judge's own tastes, variations in training and understanding of the sheets, the number of corps and the performance order, and of course the fact that the difference between 94 and 91 is not actually explainable. They all play a role and the final number has no quantifiable value. Rank and spread are the only remotely useful numbers, and even spread has some issues.