The following Document contains information originally provided by Kaja Federowicz.

Addendum: This seminar was presented by Kaja Federowicz during Judge Conference at GP Milan 2011. Its main focus was a practical approach to writing evaluations of other judges – short theoretical introduction was followed by working with sample reviews, which represented both good and bad practices. Attendees were asked to identify examples of these practices and in case of bad ones, suggest better solutions.

Introduction: The Good Practices

During this part of her seminar, the presenter described six basic characteristics of a good review. They are:

• Relevance, or “be useful”. A review should always be useful for the reviewed judge. If it’s not going to help, don’t write it; there’s a wide range of other means of communication if you’d like to say e.g. “thank you” or “I enjoyed working with you”.
• Balance, or “don’t criticise nor praise too much”. A good review is balanced in both amount of listed Strengths and Areas for Improvement and the tone of describing them. It shouldn’t consist of mostly Strengths or mostly Areas for Improvement (nobody is perfect, but nobody is completely terrible as well!). Be careful with emphasizing one part of the other – balance in amount of observations is pointless, if you really, really stress only “positive” or only “negative” things.
• Accuracy, or “don’t exaggerate”. Be careful with words that express amount or intensity and with definite statements: don’t say “John IS…” if you only observed one example of certain behaviour; don’t say “it looked a bit like…” if you observed many situations which support your point of view. Be careful with serious accusations such as “John’s knowledge of rules is really, really terrible” – even if it appears to be true, try to put it in a more diplomatic way.
• Foundation, or “refer to actual examples of observed behaviour”. Every time you point out a strength or an area for improvement, tell what situations made you draw such conclusions. If you mention diplomacy, tell about that specific ruling you liked so much. If you mention bad organization, tell what exactly went wrong.
• Explanation, or “justify your point of view”. When you claim something is a strength or something is an area for improvement, explain why do you think so/why it’s important. If you say “as a team leader, John should delegate more”, explain why exactly it’s important to delegate tasks – John probably didn’t know it and now that he does, it will be easier for him to improve. Understanding strengths better also helps – it’s easier to maintain them and develop them further.
• Assistance, or “give advice”. Pointing out what should be done is good, showing how is better. With good advice, it will be much easier for the reviewee to improve.

The attendees also suggested the following rules:

• Write reviews you’d like to receive yourself. Be polite, be fair, be helpful.
• Mind the culture gap. Some things are perceived differently by people from other parts of the world – of many potential ways of expressing your thoughts, try to use the most neutral one. Avoid jokes – these are particularly troublesome (and not really needed in a review).
• Discuss the feedback. Review is better when you discuss it with the reviewee at least partially before submitting it. This way you can get broader context, improve accuracy, get new examples. Also, sometimes you’ll discover your observations were wrong, your conclusions incorrect.

Practice

At the beginning of this part of the seminar, the attendees received two sample reviews (half of them one, half – the other one), as well as a sheet of paper with empty field corresponding to the six qualities of a review. The pdf file can be found here. Their task was to evaluate the reviews, focusing on these qualities (or lack thereof). If certain characteristic was present, they were supposed to point out how exactly they were achieved. If not, they were asked to point out what was wrong or missing and to propose a change to make the review better. Then, observations were exchanged during a discussion. Since the two reviews were printed on two sides of the same sheet of paper, everybody could’ve looked at the commented fragments, even if originally they were assigned the other one.

Some of the points made by participants were as follows:

Review 1: Ash

• Relevance – the review is not very useful, mostly because of the Areas for Improvement section: observations seem exaggerated and the assistance provided is not very helpful.
• Balance – the review is not balanced at all. 1 Strength vs 3 Areas for Improvement, and the latter use a more offensive tone which dominates over the more subtle first part.
• Accuracy
  • Line 4 (“lot of awesome ideas”), which sounds like a typical example of inaccurate statements, is nicely defended in lines 5 and 6, by providing specific examples. Other than that, the review uses a lot of exaggerated statements, even in spite of some (low-quality) foundation provided. Check:
  • Line 11: one situation, especially with correct, but a bit delayed answer, definitely doesn’t mean somebody is “bad at rules”. This is not enough for such a conclusion – if there was one situation, generalization should be avoided.
  • Line 17: “lazy” is a serious accusation and it hardly can be justified just by what happened during one events, unless the judge showed blatant disregard for his work, not just “took many breaks”.
  • Line 22: again, “not being reliable” is quite serious and again seems like it’s not properly justified. Before one writes something like that, they should understand the problem first. What if the 5 minutes of delay were due to car breaking down, but the HJ knew about it?
• Foundation – the review is full of examples. Sometimes really good ones (lines 5-6), and sometimes ones that don’t justify the conclusions drawn from them (11-12, 18, 22) – but at least it is clear what led reviewer to such conclusions.
• Explanation – lines 13-14 contain the only phrase that could be considered something remotely similar to “explanation” – reviewer is not happy with Ash’s rules knowledge, because we expect L2s to be better. Other than that, reviewer says mostly things that are obviously positive or negative and don’t need explanation (e.g. everybody knows “lazy” is something bad).
• Assistance – there is some, but not very valuable. Re-reading CR and reading the Rules-L (line 15) seems reasonable, though probably obvious for a level 2 judge, but stopping by the table to watch match when one is tired (line 20) is not the most clever advice to give.

Review 2: Willow

• Relevance – this review has some potential, but its relevance depends on what has really happened during the event. Observations are not supported by example, so they might be either true or fake.
• Balance – 2 Strengths and 2 Areas for Improvement is ok. The quality of the two parts is not similar – see below.
• Accuracy
  • Line 6: reviewer’s opinion (“very good rules & policy knowledge”) seems justified. Even though we don’t know how many situations contributed to this opinion, “a good deal of Willow’s rulings” sounds like a reasonable amount of material.
  • Line 10: very generic statement; without examples, it’s hard to tell whether it’s accurate or not.
  • Line 16, line 18: both “really bad” and “never” are strong expressions that almost always turn out to be inaccurate. Everybody has their weaknesses, but hardly ever one is so terrible at something. Unless properly justified, such statements should not be used (and if they’re true, it’s still better to be a bit more diplomatic).
  • Line 24: again, such “very” without proper reasons/examples sounds like an obvious exaggeration.
• Foundation
  • Line 6: examples are in general better, but like it was mentioned above, this at least looks like a fair justification.
  • Line 10: some examples would be great, especially if Willow does this all unconsciously (thanks to a natural talent) – if she realises this, she can develop her skills even further and it will be easier for her to teach others.
  • Line 17: when, what tasks? It’s hard to improve when one doesn’t know what exactly is wrong. Also, examples let reviewee remember better her performance at given event and not repeat these situations in the future.
  • Line 24: again: when?

Without examples, these Areas for Improvement look a bit as if the reviewer didn’t really observe these problems, but needed to put something in this box to make the review “balanced”, so came up with these ideas after the event, without remembering any particular situation like this.

• Explanation
  • Lines 7-10: everybody knows rules are more or less important, but this shows an interesting point of view that a judge with little experience (even L2) may have not realised.
  • Lines 13-14: it seems obvious that diplomacy is something good, but again, not everybody realises how important is our “public image” – not just the individual judge-player relationship, but the message we send as a group, to a group.
  • Lines 21-22: and again, this is something not everybody realises, but once they do, they will really pay more attention this issue. Ideally, this fragment will make Willow think more about the consequences, not just focus on actions.
  • Line 26: the explanation is not very elaborate, but at least hints a valuable detail (“improvement is important”).
• Assistance – nothing! How is Willow supposed to work on her planning skills, if she has absolutely no idea how to do it? Any piece of advice is better than nothing.

Presenter’s Tips

• The first part may be run discussion-style rather than lecture-style, but discussions tend to take more time than expected, to wander away from the main topic, etc. Of course there still must be room for questions or suggestions, but the focus should be on the most important and interesting part, i.e. the practical one.
• Preparing two reviews makes it easy to show the good and the bad execution of certain characteristic, e.g. a balanced and an unbalanced review, or one with good foundation and one without. Half the attendees work with one review, half with the other one, then you discuss them one by one. Make sure they’re not very long, so that people from one group can read quickly the other review when it is discussed – you want everybody to participate, even if “active group” surely makes more comments than “non-active group”.
• Use line numbering in the reviews – it allows to identify the discussed fragment more easily (check this handout).
• Adjust interline spacing and make one broad margin so the attendees can take side notes and add their modifications to the text (check this handout).
• For the purposes of taking notes, prepare an extra sheet with the list of characteristics: this way people will focus on the right things and will learn more; also, discussion will be easier (point by point, without trying to find valuable observation on a sheet with some random scribbling). Blank paper is scary – produces the uneasy feeling “what should I do now?!”.
• When preparing your sample reviews, use the “backward engineering” strategy: first make a list of features you’d like your reviews to have, then write them accordingly. For example: “review 1: balanced, not very accurate, no examples, good explanations, assistance partially present; review 2: not balanced, accurate, good examples, no explanation, no assistance”. This way your reviews will illustrate your points the best; you won’t end up without a proper example of assistance, etc.
• Make your attendees work in pairs, not individually. Preferably, make less experienced judges work with more experienced ones. The former usually aren’t very active during such seminars if left on their own.