Wed 22 Feb 2006
Comments Off on Judging – Part 2 of 2
Last article introduced the topic of judging. This is the final installment about judging. Iâ€™d like to thank those who helped me out pulling this one together.
Q. Suppose a judge likes an entry and gives it the highest score only to find the next entry is even better. Can the judge change the first entry’s score?
A. NO, NO, NO. The only allowable score change is one which is directed by the Representative. The judges are responsible for giving careful thought to the scores they hand out. Since there are 30 minutes between each turn-in, judges have plenty of time to carefully evaluate each of the entries that he/she samples. This would be an example of comparing entries, something that is not part of the KCBS judging process. Judges are instructed to evaluate each entry on its own individual qualities. Once an entry has been scored, the judge should try to forget that entry and move on to the next.
Q. How can a judge avoid comparing entries?
A. The easiest way is to simply cover the top of the score card with a napkin. “Out of sight, out of mind,” applies when it comes to judging. With the previous scores covered, the judge will not be tempted to look at the scores already given to other entries and see how those compare against the current entry. This may sound odd but once the cards are turned in, most of the entries blur into indistinct memories in the judges’ minds. Only the great or truly bad entries stand out and the memories of them quickly fade. One of the table captainsâ€™ tasks is to keep an eye on the judges as they complete their score cards. If a captain spots a judge filling out the card incorrectly or referring back to previous scores, the captain will give the judge a gentle reminder to not compare scores.
Q. What is done to prevent a dishonest team and a few corrupt judges from being able to identify that teamâ€™s entries and increasing its chances of placing well?
A. Keeping the identities of the teamsâ€™ entries secret is of primary importance and the KCBS employs two primary means of protecting that secrecy and the integrity of the results.
First, the KCBS uses a â€œdouble blindâ€ system for protecting the identities of the teams from the judges. As discussed in the introduction, each container a team receives is marked with a number unique to that team. How a team was assigned their particular number is determined by the contest reps. The number might simply be based alphabetically by team name or the captainâ€™s last name. It could also be selected by site number, the teamâ€™s registration date, the captainâ€™s ZIP Code, or by any number of other criteria. Reps have their own methods of assigning team numbers and many alternate them between contests. As the boxes are turned in for judging, labels with new numbers are placed over the number on the boxes. The correlation between the original number and the new or â€˜blindâ€™ number is known only to the contest reps. Most reps change their blind numbers between contests. With this safeguard in place, even if a judge knew a team he was pulling for was team #11, for example, no box with the # 11 on it will be shown to the judges. Instead, after the blind number is applied, that teamâ€™s boxes might be presented to the judges as #71, #132, #151, #226, or some other seemingly random number, depending on the blind number being used at that contest. The teams have no way of knowing the repsâ€™ blind numbers until after the contest. The judges should also not know the various teamsâ€™ numbers. Without knowing both of these vital pieces of information, trying to identify a particular teamâ€™s entries with any degree of accuracy is essentially guesswork.
Second, another step is taken to further reduce the possibility of an â€œagreementâ€ between a cook and a judge affecting the outcome of a contest. In this step, a table of judges will receive an entry from any particular team no more than once over the four main meat categories. After the boxes have been turned in, they are renumbered with the blind number and are staged on a large tray or bread flat for the captains. The first table captain in line compares the box numbers on that tray against the log of boxes already judged by his or her table in previous categories. If that table has already judged one or more of the numbers in any of the previous categories, the captain will remove those boxes from the tray and exchange them for numbers that table has not yet seen.
As a final word on the subject, while the possibility of cheating the system must be recognized, the likelihood of any team or judges attempting to influence a contestâ€™s outcome by cheating is extremely remote. The competition barbecue community is extremely close-knit and no responsible competitors and judges would be willing to sacrifice their standing within that community by circumventing the rules for a relatively small and briefly enjoyed gain.
Q. After the score cards have been turned in and reviewed, can a table captain or a contest rep tell a judge to change any of the scores because they were not in line with the rest of the judges at the table?
A. No. Neither the captains nor the reps have the authority to tell judges what their scores should have been after the fact. The captain and the reps will not have tasted any of the entries in question and, in all likelihood, the reps would not have seen the entries.
If a judgeâ€™s scores are significantly different (usually much lower) from the others on a couple of entries in a category, a rep will ask the judge why the scores in question were given. By doing this, the rep is not implying or suggesting the judge should be giving higher scores. The rep is simply looking for an explanation of why the judge felt that score was appropriate for that particular entry. In the vast majority of cases, the reason was justified (for example, that particular sample was fatty, burnt, had an off-flavor, etc.) and no additional consultations will be required. However, if a judgeâ€™s score is consistently well above or below that of the rest of the table, the rep can take the judge aside and first ask whether they completely understand the scoring system and offer a quick explanation of the system. Many times, this mini-consultation will bring the wayward judgeâ€™s next scores back into line with the remainder of the table. These conversations are always performed in private since the object is not to single out and embarrass a judge in front of their peers.
Q. Are certified judges required to compete from time to time to keep their certification status active?
A. Although many judges could benefit from the experience of competing, the KCBS has no requirement mandating a judge to compete at all, except in the case of judges seeking to achieve a Master Judge badge. While the intent of such a proposal to create more well-rounded and experienced judges is desirable, implementing such a requirement for all certified judges would decimate the pool of available judges. Judging is a voluntary position and many judges rationalize spending a few hours of travel and a tank of gas in order to serve as a judge a number of times throughout the season. However, if judges were required to compete in order to keep their certifications active, only a very small fraction would be in a position to justify the expense of acquiring the equipment to compete. Assuming a judge has none of the competition equipment, he or she would have to spend around $1,000 for low-priced (and unreliable) smokers, canopies, coolers, work tables, and the various other equipment needed to compete at the most basic level. While having more judges with first-hand competition experience is a desirable goal, it should be easy to conclude that any requirement stipulating judges must bear such a level of expense simply to continue to volunteer their time and travel expenses in the future would have a disastrous effect on the number of available certified judges.
Q. Barbecue cooks tend to be hardware junkies. Why canâ€™t some teams donate their old, unused equipment and spare the judges who need to compete at least a significant part of the costs to compete?
A. On the surface, this sounds like a reasonable suggestion. However, once considerations such as short- and long-term storage, maintenance, transportation, responsibility and liability for damages, the assignment and availability of the hardware, and countless other sticking points are factored in, the likelihood of such a program working is extremely remote.
Q. If pooling used equipment for educating judges is an unworkable solution, why canâ€™t the KCBS simply assign judges to cook with teams?
A. Itâ€™s fair to assume many judges would gain experience by rolling up their sleeves and getting dirty by competing. However, having the sanctioning body, organizer, or any other third-party arbitrarily assign judges to teams to fulfill this requirement would likely do more harm than good to all participants. Competition cooks, by nature, are a secretive lot. Most have their own recipes, techniques, timings, and methods that they would probably resent having to divulge in the presence of a stranger.
A cook who is resistant to the thought of revealing even a few of his secrets is more likely to have the judge instead perform superfluous jobs such as watching thermometers, washing dishes, keeping work surfaces clean, and gathering trash. These tasks need performed during the course of a contest but, they border on the menial and punitive when compared to the intended goal of such an assignment program and are not likely to give a judge much insight to the contestantsâ€™ plight. In a similar vein, the members of established teams already have their own tasks they perform throughout a contest. Introducing even a well-intentioned interloper into the mix would likely upset the teamsâ€™ workflow.
Also, the assignment of judges to teams leads to the very real possibility of personalities clashing. If the two parties are not able to get along with each other for any number of reasons, the intended beneficial contest cooking experience would become a miserable experience for everyone involved. With a turn-in deadline looming large, head cooks snap at their closest family members and friends. The discomfort of having a complete stranger in the mix at that time is discerned for all.
Finally, just because a certified judge has not yet cooked – or has chosen not to compete – simply does not translate to that judge being unqualified or incapable of judging satisfactorily. In practice, many judges do interact with the teams on a personal level. If the right chemistry exists between a judge and the team, the judge or the cook may broach the subject of joining the team to get a chance to view the contest experience from the cookâ€™s perspective.
Q. What is a Master Judge?
A. A Master title indicates a judge has served as a judge in at least 30 contests in a 5 year period, and has competed at a sanctioned contest at least once during that time. That participation may have been as a member of an existing team or by competing as his or her own team. Truth told, the title is largely an honorary one. Achieving the rank does not necessarily mean a Master is any better or more qualified than a certified judge serving at his or her fifth contest. More than anything else, the title is granted by the KCBS to recognize a judge’s dedication to supporting competition barbecue over the course of several years.
Q. Are there any regional differences in what the judges like?
A. This may have been true at one time but most judges and competitors would probably agree the existence of a vast difference in taste according to regions is largely a myth, yet one that can be debated to death. With KCBS contests being held year-round now and many cooks and judges traveling great distances to participate, any significant regional differences that may have exited at one time have largely been eliminated. Right or wrong, most successful cooks are achieving their success by appealing to middle-of-the-road tastes.
Q. Where can I find out more?
A. Go to the KCBS website at http://www.kcbs.us. On it you will find more information about judging, the location and contact information for upcoming judging certification classes, as well as a schedule of upcoming contests.