Assessment of Other Judges, one of the 12 Qualities of a Regional Judge, is most often associated with Judge Center reviews, but it goes far beyond that. When done correctly, it can be a fully-integrated part of any normal judging routine. As many of the other articles in this series have done, let’s begin with the definition of the Quality from the Judge Wiki article:

 

“Regional Judges evaluate and formally review other judges on a regular basis. The reviews they write are professional, constructive, detailed, and helpful to the judge reviewed. Regional Judges can observe and assess a wide variety of skills, attitudes, and abilities in their fellow judges, including those of higher level. A deficient judge is one who reviews infrequently or assesses other judges very superficially. His or her reviews may lack substance, depth, and detail. The reviews may show evidence that the judge only ever assesses a very small selection of skills in other judges. An exemplary judge is a prolific reviewer, is consistently detailed in their reviews and feedback, and merges evaluation of other judges with mentorship of them seamlessly. The judge is also insightful and constructive when reviewing or critiquing judges of higher level.”

Given this definition, how does one go about writing good reviews and showing strength in Assessment of Other Judges? It all starts with good observations on the floor of the tournament, notes on those observations, a solid debrief with the subject judge, and finally the review.

 

Observation

If you want to assess whether someone is doing a good job or not, what is the first thing you need to do? Watch them do their job! This isn’t a riddle, yet many people treat it as such. They will try to assess another judge with very few live observations, or they won’t follow through on their observations.

Shadowing is the most time-honored method of observation. This is the act of following another judge and watching them answer judge calls on the floor. It is an especially useful activity when there is an inexperienced judge on staff who may frequently need to call upon a more experienced judge for consultation. This article by Chris Richter is five years old as of this writing, but still highly applicable. In particular, I like the statement “The primary goal is that both the judge doing the shadowing and the “shadowee” can learn from each other.”

When you are a shadow, you have the opportunity to watch a ruling without being involved and you can take in a lot more of what’s going on. Observations about the judge’s body language or the players’ reactions are useful tidbits that the subject will naturally have difficulty gauging on his or her own.

Observation does not need to be restricted to naturally-occurring judge calls. You can use rules and policy questions or discussion to generate observations about a judge’s knowledge of the rules or policy philosophy. This is a fairly standard practice when probing a judge’s knowledge in advance of them taking an advancement exam. But this type of assessment and observation can be appropriate for any time.

 

Note-taking

Unless you have a photographic memory, you are going to want to take notes on your observations. These will be useful during the latter stages of assessment, namely the debrief and review. My note-taking usually goes through two distinct steps. When I make an observation that I want to record, I will write it down in my notes making it as short as possible. It’s important to not try to write your entire review out there on the floor; there isn’t enough time for that. On the flip side, your snap notes need to be legible and understandable by you later. I’ve had unfortunate notes that I couldn’t figure out what I had written later.

Later on, when there is more time to write, I will sit down and expand on my snap notes. This will often be during or after I have a face-to-face debrief with the subject (see below) because that conversation will inform me of the direction I want to take the eventual review.

 

Face-to-face Debrief

The debrief is an important step in the process where you take your observations and use them to present your feedback to the subject judge. This most often takes place towards the end of the event; you should have plenty of observations and notes at that point, and you’re unlikely to add to them without taking with the subject. As the player-to-judge ratio thins out, check in with your Head Judge or Team Lead to see if you can set some time aside to talk the subject without interruption by taking a “talking break.” Obviously, not everyone in staff can do this at the same time. The needs of the event come first.

If the event doesn’t allow for you to do a debrief onsite, you’ll want to find some other way to accomplish this before you enter your review because the debrief is an important time to clarify details and gain insight into what the subject was thinking when they did the things you observed. Without the debrief, you only have half the story. Post-event debriefs can be done via phone, e-mail, chat, or in person if you happen to live close by or meet at another event.

There are several goals to the debrief. First and foremost is to deliver your feedback to your subject. Assessing other judges is useless if you don’t use that assessment to help other judges improve and grow. Doing so face-to-face is important because receiving feedback is never easy; receiving feedback indirectly and impersonally is worse. You need to look your subject in the eyes and deliver your feedback in a direct manner without hedging, but also with compassion and understanding. How you deliver the feedback matters as much if not more than the feedback itself because it affects whether the subject will listen and take the feedback to heart.

Another goal is to take your observations about the subject and turn them into a conclusion, not unlike a scientist testing a hypothesis to reach a conclusion. Whatever observations you make don’t tell the whole story as they are naturally colored, perhaps even biased, by your thoughts and opinions. Feedback, by its very definition, is a two-way street, and bouncing what you’ve observed off of your subject is an important way to round out your understanding of what happened and why it happened.

You will need to keep taking notes as you debrief with the subject judge. In fact, the lion’s share of my notes come after a debrief, based on what the subject judge said and my expanded thoughts off of the debrief.

 

Review

Writing reviews is a topic that I could go on for many more articles about, but for the purposes of this one I want to focus on what it means to write reviews as a Regional Judge. I’ve often heard Head Judges and Team Leads say “my goal is to write a review for each of you [meaning judges on staff or on the team].” It’s a noble goal, but misses the mark as to the point of writing reviews. Notice how the definition of this quality uses the phrase “helpful to the judge reviewed.” We write reviews to be helpful to the judge reviewed. Let that sink in for a moment.

At its core this activity is at its most useful when you care about your subject and want to point out ways for them to improve at judging. This is why it is so important to have a face-to-face debrief to set up a personal connection with the subject judge before writing a review. A review out of the blue might be ignored or dismissed. If a review is a summary of a debrief, you’ve already done the work beforehand.

Don’t hedge. This is a message that I’ve recently given a Conference Workshop on with the help of Joe Wiesenberg and Scott Neiwert. It comes from the phrase “to hedge your bets,” meaning to play both sides of a wager. When you do this, you will obviously win at least a portion of the wager… but you will also lose a portion of it. I describe hedging in feedback as delivering critical feedback, but littering it with words and phrases that undermine that message. Phrases like “this is just a minor thing” and “other than that, you were fantastic.” While these things may very well be true, by writing them you are distracting the subject from the feedback you are giving.

 

Putting it all together

One strange thing that I’ve told people about my skills in this quality is that I learned a lot of them from reality TV shows like “Kitchen Nightmares” and “Restaurant Impossible.” These shows do a great job of replicating the assessment and feedback process in judging. The goal of the show is to help a struggling restaurant improve upon their less than stellar history and thrive. That has a lot of resonance with feedback in judging. When we make an assessment and provide feedback, we want the judge to improve in the future.

Just like a good assessment, the restaurant shows start with observation. The hosts, Gordon Ramsay and Robert Irvine respectively, will sample some of the restaurant’s food and participate in preparing one evening’s dinner service to determine what the restaurant’s issues are. Is the food bad? (Often, yes.) Is it because the chef is bad or are the kitchen conditions leading to the problem? Is the owner meddling too much and not letting his or her employees do their jobs properly?

After making some important observations, the host will have a face-to-face debrief with the owner, chef, and staff to give them his feedback. Then they show a dinner service where the restaurant has taken the host’s feedback and made improvements. Customers are happy and the restaurant goes on to better and brighter things.

 

Conclusion

I often hear various rules of thumb about writing quality reviews. The most common thing that I hear is that a review should be balanced with both positives and negatives. If framing things in this way works for you, that’s great, but I personally feel that this imposes a false dichotomy. Feedback isn’t positive or negative in nature, at least not in the sense that “you did these things well” is positive and “you need to work on these things” is negative. Those are merely observations and opinions. If I spot an area for improvement for another judge, provide feedback that identifies why this area should be improved, and give a road map for how to make said improvement, that’s all positive in my book.

To me, the true benchmark for a good review, an L3 quality review, is “will this help the subject?” It’s okay to admit that you write reviews “because you have to” or under a sense of peer pressure or social obligation, but the fundamental shift that you need to make in writing good reviews is an internal one: You have to care. Judging is a hobby. The minute you start to look at any part of it, like writing reviews, as a job, your enthusiasm will falter and your finished product will suffer.

In order to write good reviews, give quality feedback, and ultimately be an adequate L3 in the Quality “Assessment of Other Judges” it takes a dedication to the entire process outlined above. You simply can’t write a good review from just a few brief interactions on the floor and no face-to-face debrief. This is why Chef Gordon Ramsey participates in the dinner service with the kitchen staff. In order to provide the best feedback, he needs to know more than just that the food is bad; he needs to know why the food is bad by watching it being made.