Rank and Rate...managing numbers...Kind of?

[cixelsyd]

Every year, we hear the judges job is to rank then rate.

Exactly. And why is that?

A judge should be able to simply rate the corps, and in doing so, the ranking takes care of itself. The only reason a judge should ever be forced to choose between ranking and rating is where they do not have enough numbers to rate properly. And that is the problem.

Since i judge on a local band and indoor circuit level, I understand that message and have sat through some amazing numbers management sessions. as Kamaraq can tell you, I'm a stickler about this, and can dissect a recap in seconds.

Then I look at the recaps from the last 3 World Class shows for DCI, and I shake my head.

I have been shaking my head for years. I know what is coming long before the recaps are posted.

I understand that bottom line ties are a no-no. Especially on a show with a number of units, you'll have some sub box ties. There's only so many numbers.

Precisely. And thus, the solution has to be one that gives the judge more numbers to work with.

Technically, a judge has 201 possible numbers they can give out. But the "box" system (aptly named) boxes them into far fewer choices. By the end of the season, the top 12 are crammed into a 12-point spread (typical semifinal scores are 97-point-something for 1st place and 85-point-something for 12th). To conform to that scoring expectation, the individual judge is left with about 30 different numbers he can assign to those corps. As you can already gather from examples earlier in this thread, it is barely even possible for the judge to assign numbers to corps as they perform, and still permit themselves to rank the corps correctly. Forget about rating.

At a contest with all WC corps present, the judge has enough of a number-management challenge just allowing enough space between corps to permit subsequent performing units to rank in between. The challenge is far greater if extended to each of their two subcaptions. Accomplishing all that, and staying within the box scoring expectations too, might actually require the type of "homework" (slotting) alluded to in this thread.

This is all easily fixed by giving the judges more numbers to work with. Either of these methods will work:

- Allow the judge to use 0.05 increments.

- Give each judge 40 instead of 20, and divide the total by 2.

Remember this phrase I learned from an old school guy...."if you give out a 10 before the last team goes on, you're telling anyone that follows to #### in their hat and go home"

In a perfect world, there would be no "perfect" scores.

Even if the 20 went to the last corps performing, you are left wondering if that reflects the full margin of victory, or whether the judge was boxed in by previous scores. Obviously, giving a 20 to anyone other than the last performing corps is telling all subsequent performers that you have slotted your winner ahead of time. Never mind what or how you perform tonight - you lose.

Quarterfinals, you had 36 corps. Yeah I expected some sub box ties.

What we got was 63...including two judges having 3 way sub box ties. 3 judges were between 10-12, and 2 more had 9. Yet the VA judge had one, and percussion had 0!

But wait there's more....you had some 9.9's and a 10 handed out before the last corps went on. So if you give a 10 and there's a corps to go, what do you do? Give, in this case BD, a 10.1? Even with the 9.9's you're basically telling BD you need 10's to win the sheet.

Ok, so semifinals....only 25 corps, not so many right?

Well, yes. 51. One judge had 12, two others had 9. One judge had 1, two had zero.

Again, though dig deeper.....the amount of 9.9's and again a 10 handed out before the last corps goes on.

Finals.

Only 12 corps....and as such, only 16! commendable. I mean this is the best of the best, so yeah, there should be a lot fewer, but the quality is tighter.

Until you see 2 judges had 4 each, one of them with a 3 way tie! and wait...11 9.9's and 2 10's before Crown went on...some to the 3rd place corps?

Really?

I'll admit, I've never judged at that high of a level.I'm sure there's some tough calls to be made. I heard the Cadets drum tape, and ###### if I wouldn't have gone there myself. But....is that many ties really acceptable? if we did that on a local level, we'd be crucified.

I'd like to hear thoughts. I'm really open minded to see if I am overanalyzing this

No, you are not overanalyzing. It is actually disturbing to see how some judges are so clearly okay with sub-box ties, while others avoid them like the plague. No middle ground. Must be making the number management nightmare even a bit worse than it needs to be.

[oldtimefan]

This is a very good discussion that should be easily avoided, but the answer is way too simple and will be ignored. Score a corps for their perfomance NO MATTER WHAT. If there are 700000000 corps and the first one on has a drum line that is perfect, the drum line should get a perfect score. PERIOD. Seems pretty simple to me. If every corps ties because they all EARN their score, then so be it. It is what they earned and not where they are to be SLOTTED. I am very simple. Give the EARNED score.

Edited August 13, 2013 by oldtimefan

[OrlandoContraAlum]

Is there a recording (audio or video) of Suncoast Sound's 1989 semifinals performance? Or have I perhaps heard it? I know that the CDs issued in 1989 --which I'm grateful to have-- featured BD from semis, since the frack isn't there; is it possible they took the best recording of Suncoast as well? I guess I should closely compare the sound to the DVD.

Not sure... there was a pretty low quality video out there at one point (obviously taken from a VHS copy). I will see what I can find. I do know that the CDs, DVDs and APD are certainly Finals.

[Henson]

What would be required to forestall this "problem" of subcaption ties is keeping not two, but 4 totes. One to list the corps' numbers, one to rank the caption, and two to rank the subcaptions.

Really? Is that worth it? All to avoid a tie in a number that no one actually cares about?

I may be talking out the wrong end here, but how likely is it that a judge might look at his ranking sheet, put the corps in the right spot, check the caption scores + spread above and below, and then think up a number that he feels is correct - and then allocate that total into the subcaptions in a way that makes sense without giving a #### what he gave the corps 2 hours ago? All in the couple of minutes that he has to wrap up his comments, assign a score, submit it, grab a drink, and maybe head to the restroom?

Is this actually a problem??? What number do we care about? The score. The end product. Process matters, yes, but not to this degree in m opinion. These aren't engineering tolerances here - we're talking about judging the artistic product of a group. If it requires a slight fudge (or 3 subcaption ties) to get the correct brass score, well, make it so.

I could be more artful when I say that no one cares about the sub-boxes. The reality is that this is the judge's way of telling the staff his read on the potential of both the design and the performers. It's a tool for the corps, not for the fans to analyze. Fans just want the right score. And I could argue that Finals week is exactly when those numbers are the least useful...

Edited August 13, 2013 by Henson

[MikeD]

I could be more artful when I say that no one cares about the sub-boxes. The reality is that this is the judge's way of telling the staff his read on the potential of both the design and the performers. It's a tool for the corps, not for the fans to analyze. Fans just want the right score. And I could argue that Finals week is exactly when those numbers are the least useful...

That is why some here have mentioned a difference between judging a championship vs the rest of the season. The main focus is on the total caption score at champs. This can lead to sub-caption ties, unless there is a decision to abandon the boxes and use more of the total number of points on the sheets. If that were to happen, watch lots of scores plunge as ratings get stretched out...downward in most cases, as there is already nowhere to go on the up side.

.

[Pete Freedman]

This is a very good discussion that should be easily avoided, but the answer is way too simple and will be ignored. Score a corps for their perfomance NO MATTER WHAT. If there are 700000000 corps and the first one on has a drum line that is perfect, the drum line should get a perfect score. PERIOD. Seems pretty simple to me. If every corps ties because they all EARN their score, then so be it. It is what they earned and not where they are to be SLOTTED. I am very simple. Give the EARNED score.

This. I've just realized that this thread is a bit like getting writing advice from the local writer's workshop. They all seem very knowledgeable and experienced. And they are. They can pass on their vast knowledge of how not to be a successful writer.

You have to learn from the best. And the judges on here all seem to be band judges or DCA judges. I certainly hope the methods they are discussing here are explicitly banned from DCI practice.

DCI (at least) should use the following rules:

1. Before the season begins, decide exactly what a 100 would look like.

2. Know exactly what a 90, 80, 70, etc. look like. Find reference audio clips for brass or percussion (or video for visual) for each of these values (or for values near enough, then use those values as your reference.) Listen to these clips again periodically during the season to remind yourself of these performance levels.

3. Be absolute and unwavering in your judgement. Consider each unit as if nothing else exists in the universe at that time. You have seen no other units that day, and you will see no more later. You will never judge anything but this unit's performance right now. There is exactly one correct score for this unit, and your only job is to find it. That score is based on all the work you did before the season in creating the reference points, and reminding yourself of them.

4. You may see other numbers on the paper that look to be in your handwriting. Ignore them.

[Kamarag]

What?

Sorry, but explain to us all how any judge can "assume" that the corps who placed 4th in a caption on Friday cannot possibly advance one spot in that caption Saturday... and how that assumption would not constitute slotting (or something worse).

I'm not telling you how it should or shouldn't be, I'm simply telling you how it *is*.

If we "knew" such things in advance, we would not need judges. Your phrase "done their homework" sounds like the definition of slotting to me. Glad you are not judging in DCI, and hope your local band circuit gets better judge training soon.

Way to miss the point. I don't judge DCI precisely because of this. I hate it. It drives me nuts.

I know not to assume anything when judging bands. A band strong in one area can walk into championships and completely implode. The very next band, who may have been a train wreck all season, could give you the performance of a lifetime. Heck, I've seen it many, many times over the years. You have to consider that possibility before the first band hits the field.

Sadly, they do not do this in DCI or DCA. I can understand why, but I don't...and you don't...have to like it.

[MikeD]

DCI (at least) should use the following rules:

1. Before the season begins, decide exactly what a 100 would look like.

2. Know exactly what a 90, 80, 70, etc. look like. Find reference audio clips for brass or percussion (or video for visual) for each of these values (or for values near enough, then use those values as your reference.) Listen to these clips again periodically during the season to remind yourself of these performance levels.

3. Be absolute and unwavering in your judgement. Consider each unit as if nothing else exists in the universe at that time. You have seen no other units that day, and you will see no more later. You will never judge anything but this unit's performance right now. There is exactly one correct score for this unit, and your only job is to find it. That score is based on all the work you did before the season in creating the reference points, and reminding yourself of them.

The problem is not knowing what scores as disparate as 100, 90, 80, 70, etc look like. It is knowing what a 100, 99.9, 99.8, etc look like.

Reference clips do not really help all that much, because there are various criteria within a caption and sub-caption that have to be considered, and how each corps stacks up in the sub-captions can vary corps-to-corps. The stated criteria is not really a checklist; it is givng the judge items to consider in creating an overall evaluation. One corps might be heavily weighted in criteria 'A' while another is strong in 'B' while a third is not up to either in 'A' or 'B', but has a good mix of both.

[ShortAndFast]

3. Be absolute and unwavering in your judgement. Consider each unit as if nothing else exists in the universe at that time. You have seen no other units that day, and you will see no more later. You will never judge anything but this unit's performance right now. There is exactly one correct score for this unit, and your only job is to find it. That score is based on all the work you did before the season in creating the reference points, and reminding yourself of them.

No. The judges's first job is to rank - she must make a decision which of the competing units is better. If she assesses two units as both being an 18.0 in the reference criteria in her head, she still needs to give one of them a 17.9 or an 18.1. This is explicitly in the text of every sheet DCI uses. The judge is failing her job if she ties the units.

[corpsband]