Judging bias

[Pete Freedman]

I've similarly track individual judges and the scores they give. There's no way to show bias just from the recap numbers. Sure Judge X may have had a group up or down from where judge Y had them the day before, and Judge Z the day after. But there's no way to show that X was wrong and Y/Z were right - maybe they had a bad show that night, maybe they just put in some changes and struggled with them, maybe the judge sampled different parts of the show, maybe X is an expert in the caption and Y/Z are less experienced, and on and on and on.

...

Sometimes it's possible. Take the following hypothetical situation: Judges, X, Y, and Z judge caption A during the season.

1. Judge X scores everybody high, Y scores them in the middle, Z scores low. So far no problem if they are all within certain limits that DCI hopefully can monitor somewhat.

2. Judge Y gives a somewhat larger spread between corps than the other two. Still no problem (again within reason).

3. Judge Z fairly consistently scores Cadets lower relative to other corps than X or Y. This is a problem, but the solution is still unclear because the cause is unclear. Maybe both X and Y are biasing in favor of Cadets. But now at least we can say that some kind of error appears to be occurring (could be a bias, or a training issue, or a problem with the judging rubric in place).

4. Judge Z sometimes judges in a different caption, caption B. Judge Z also tends to judge the Cadets lower than other judges in caption B (but not always). The evidence is now better against judge Z, but still may not be enough to take action. For example, if judges X and Y are the other judges in that caption as well, they could be expressing their bias in caption B. Judge Z may be the hero.

This illustrates how difficult it is (agreeing in part with you ShortAndFast), but I believe it's still important for DCI to analyze this data because at the end of the day it should be helpful, combined with other information, for DCI to make their future decisions about who's going to judge what. Also, knowing that DCI does analyze the data should motivate them to reduce bias.

Edited July 30, 2013 by Pete Freedman

[skywhopper]

IMO, especially the recent discontent of who's winning etc, DCI has a potential goldmine here that is untapped:

educational articles and videos explaining the sheets and talking to judges, caption heads etc to explain the ins and outs. Maybe even do a numbers management presentation. of course then you'd have to see better numbers management, but I digress.

Start in May. Roll one out a week, every 2 weeks. Think of it as Field Pass with less fluff. As the stuff gets launched out onto social media, people will see and share. Not everyone will see it mind you, but I bet a lot of people would.

Fantastic idea. The Visual Proficiency summary/judge profile they did was pretty good. So go deeper. Do profiles on the judges and have them talk about what they look for and reward, and how they compare corps. Talk about order of appearance and yes numbers management. Be frank about the weaknesses of the system as well as the strengths, and the fact that any system of measurement involves tradeoffs. Better still, get specific. Have the judges from finals talk about exactly why they rated particular corps in particular ways. Publish the finals tapes on fan network, like some of them used to make it to DVD.

Edited July 30, 2013 by skywhopper

[MadScout80]

Maybe DCI should start tossing the highest and lowest score for each corps?

What if those are the "right" ones? And we just tossed them? Possibly the best judges aren't afraid to go out on the limbs.

[Pete Freedman]

IMO, especially the recent discontent of who's winning etc, DCI has a potential goldmine here that is untapped:

educational articles and videos explaining the sheets and talking to judges, caption heads etc to explain the ins and outs. Maybe even do a numbers management presentation. of course then you'd have to see better numbers management, but I digress.

Start in May. Roll one out a week, every 2 weeks. Think of it as Field Pass with less fluff. As the stuff gets launched out onto social media, people will see and share. Not everyone will see it mind you, but I bet a lot of people would.

I love this idea. The first part seems to be about the system itself, but the second part ("Start in May") seems to be about current scores. I see these as separate ideas, both of which are great.

DCI seems to communicate primarily via a news model. Things like the makeup of the board of directors are found by searching articles. The site should contain a board member list (and photos), a staff list and a variety of FAQs including a judging FAQ (including a discussion of various judging biases and how DCI controls them), an "Attend A Show" FAQ, a prospective corps member FAQ, a drum corps financing FAQ, a corps makeup FAQ (with the demographic stats that they've collected), etc. Also, the judging rules should be available without searching news articles. And the by-laws, and ideally a list of changes (proposals that passed - the ones that failed would be good too). I'd also like to see a history of DCI article that goes into some depth, and focuses on debates and big decisions that were made, and how they affected the activity (not promo stuff, in other words).

[Pete Freedman]

What if those are the "right" ones? And we just tossed them? Possibly the best judges aren't afraid to go out on the limbs.

I read somewhere that studies on this have been done and have found that the most accurate judges tend to judge high, and their scores are removed in those systems. That method creates the appearance of correcting for bias without actually doing so.

[ShortAndFast]

Sometimes it's possible. Take the following hypothetical situation: Judges, X, Y, and Z judge caption A during the season.

1. Judge X scores everybody high, Y scores them in the middle, Z scores low. So far no problem if they are all within certain limits that DCI hopefully can monitor somewhat.

2. Judge Y gives a somewhat larger spread between corps than the other two. Still no problem (again within reason).

3. Judge Z fairly consistently scores Cadets lower relative to other corps than X or Y. This is a problem, but the solution is still unclear because the cause is unclear. Maybe both X and Y are biasing in favor of Cadets. But now at least we can say that some kind of error appears to be occurring (could be a bias, or a training issue, or a problem with the judging rubric in place).

4. Judge Z sometimes judges in a different caption, caption B. Judge Z also tends to judge the Cadets lower than other judges in caption B (but not always). The evidence is now better against judge Z, but still may not be enough to take action. For example, if judges X and Y are the other judges in that caption as well, they could be expressing their bias in caption B. Judge Z may be the hero.

This illustrates how difficult it is (agreeing in part with you ShortAndFast), but I believe it's still important for DCI to analyze this data because at the end of the day it should be helpful, combined with other information, for DCI to make their future decisions about who's going to judge what. Also, knowing that DCI does analyze the data should motivate them to reduce bias.

The place where we disagree is #3. Even if you observe this, you don't know that you have a judging error. The performances that X, Y and Z saw on different nights were different, and X/Y/Z can legitimately have sampled different material from each performance. So they might all be accurately scoring what they saw, and still have discrepancies. Such is life in subjective arts adjudication.

I also think that if a corps wants to complain to DCI about Judge Z, this kind of argument (Z always has us down by a bigger spread than X & Y) isn't very strong. Complaints that Z makes unsatisfactory tapes, or Z says inappropriate/clueless things in critique, or Z displays personal animosity towards staff members seem like they'd be more likely to result in action, especially if multiple corps make them.

[Pete Freedman]

The place where we disagree is #3. Even if you observe this, you don't know that you have a judging error. The performances that X, Y and Z saw on different nights were different, and X/Y/Z can legitimately have sampled different material from each performance. So they might all be accurately scoring what they saw, and still have discrepancies. Such is life in subjective arts adjudication.

...

I did say "consistently", so I'm assuming it's not based on a single show for judge Z. In my mind these judges are trading off shows back and forth over a season. In such a case it's unlikely that X and Y would just happen to see the corps on much better days than Z does. Still, I imagine it's more a more realistic model that judge X follows that branch of the tour for several shows, say, then Y takes over for awhile, etc. In which case you are correct that the Cadets might have gotten worse in that area (or failed to improve in that area as much as other corps) just at the time Z took over.

Still, my point is only that it might be possible to detect bias, not that it always will be.

[c mor]

Fantastic idea. :thumbup:/>/> The Visual Proficiency summary/judge profile they did was pretty good. So go deeper. Do profiles on the judges and have them talk about what they look for and reward, and how they compare corps. Talk about order of appearance and yes numbers management. Be frank about the weaknesses of the system as well as the strengths, and the fact that any system of measurement involves tradeoffs. Better still, get specific. Have the judges from finals talk about exactly why they rated particular corps in particular ways. Publish the finals tapes on fan network, like some of them used to make it to DVD.

Jeff, Sky - you both rock!

Pull back the curtain. Make the info available. Huge p.r. (public relations, not phantom regiment) win.

Hey, Boo - send this up the chain will ya?

Edited July 30, 2013 by c mor

[perc2100]

IMO, especially the recent discontent of who's winning etc, DCI has a potential goldmine here that is untapped:

educational articles and videos explaining the sheets and talking to judges, caption heads etc to explain the ins and outs. Maybe even do a numbers management presentation. of course then you'd have to see better numbers management, but I digress.

Start in May. Roll one out a week, every 2 weeks. Think of it as Field Pass with less fluff. As the stuff gets launched out onto social media, people will see and share. Not everyone will see it mind you, but I bet a lot of people would.

I TOTALLY agree with you on that, and I've always thought it was kind of odd how DCI seems non-responsive publicly when it comes to design explanations (as far as some of the more subtle stuff designers program into their show), and adjudicating. The best thing DCI ever did media-wise was (besides going to high-def/Blu-ray) when they had judge tapes and design commentaries on the DVD's. This was such a wonderful look deeper inside the "process" of design + ranking/rating. DCI stopped that because it took up too much room on the DVD's, but I really think that's the type of stuff that would work well with videos like you say, or Fan Network mp3 streams, or even individual corps could offer stuff like that. I kind of think that the REAL untapped potential of DCI is that the focus seems to be design show, compete, repeat annually. There's not a lot of public offering to be had from mid-August to May, and while some corps do a better job than others delivering content many do next to nothing. I know resources are spread thin, but I think a LOT of animosity could be quelled if more people had a better understanding of the corps side ("here's why we designed this") and the adjudicator's side ("here's why this is getting rewarded on the sheets").

[Jeff Ream]

I love this idea. The first part seems to be about the system itself, but the second part ("Start in May") seems to be about current scores. I see these as separate ideas, both of which are great.

DCI seems to communicate primarily via a news model. Things like the makeup of the board of directors are found by searching articles. The site should contain a board member list (and photos), a staff list and a variety of FAQs including a judging FAQ (including a discussion of various judging biases and how DCI controls them), an "Attend A Show" FAQ, a prospective corps member FAQ, a drum corps financing FAQ, a corps makeup FAQ (with the demographic stats that they've collected), etc. Also, the judging rules should be available without searching news articles. And the by-laws, and ideally a list of changes (proposals that passed - the ones that failed would be good too). I'd also like to see a history of DCI article that goes into some depth, and focuses on debates and big decisions that were made, and how they affected the activity (not promo stuff, in other words).

as scores dont start rolling out until late June, my May suggestion was basically timing...get the info rolling as the season is about to start.

when you judge some place, you don't have your clinic 4 months before the season starts. youhave it a month or so before it starts. Same thing here. I don't at this time see the need to have scores broken down mid season, that can actually make things worse

:shutup:/>