Exemplar Community Survey #1: Results

Written by Daniel Sole Garcia
Level 3, Barcelona, Spain

Hello fellow judges!

Almost 300 judges from all around the world answered the survey the Exemplar Team presented some weeks ago. Thank you very much!

Today we want to present you the results from this survey, and analyse them from the Exemplar guidelines. Before we start doing it, let’s remember the purpose of this survey. This activity had two major goals:

  • Let you (all judges) evaluate nominations as we (the Exemplar Team) do.
  • See if our expectations line up with the communities when applying the same criteria.

When facing a nomination, you were presented two options: ACCEPT, that means it meets all the criteria we use when evaluating nominations, and it would be included in the wave. And DEFER, which means it does not meet all the criteria.

So, after that short introduction, here are the results!!

64.1% of the judges would accept this nomination, while 35.9% would defer it. This is similar to what the Exemplar Team thinks. We would accept this nomination, as it contains attitudes that we want to see in judges. It’s a little light on specifics, but the overall package is something we would like to see more of.

A majority of judges (83.4%) would defer this nomination. And we would also defer this nomination. This nomination doesn’t pass the ‘Specific’ criteria we have for nominations, and we need more information in order to accept it. Reviewers are left having to infer what the nature of the participation is. The nominated judge has probably done a very good job in some aspect of online communication, but we don’t exactly know what the judge has actually done.

This is the first nomination the Exemplar team and the community differ on. 66.2% of the judges would accept this nomination. However, the Exemplar Team would defer it. The main reason is that this nomination is basically for working more hours than scheduled. In spite of the fact that helping a tournament is something positive, nominating judges for working more hours than scheduled sends the wrong message to Tournament Organizers and whole community, and that message is that we want them to do that everytime.
[Bryan’s Thoughts: This is a contentious one because this judge is helping out other judges and players.  That’s a good thing! And at the same time, it’s a nomination for working for a TO for free, even if its for a couple of hours.  Exemplar nominations are meant, in part, to encourage behavior in other judges. However, there is already enough pressure on Judges to “do it for the players” and work for a TO outside their schedule; Exemplar doesn’t need to add to that pressure.]

Here we have a small majority (52.8%) that would accept the nomination. However, the Exemplar Team would not accept it. The reason is because the nomination is mainly for judging events. The quantity of events a judge works is between the judge and the Tournament Organizer.  Having to travel far, should be a consideration when negotiating compensation for the event. If a fair compensation is reached, there is nothing particularly significant about travelling.

This nomination talks about a great action, so we would accept it without any doubts. And the same thinks the community. 98.3% of the answers would accept it, so you also think the nominated judge did something actually exemplar!
[Bryan’s Thoughts]  This is exactly the sort of thing that we want to see in judges.  The 1.7% that deferred this might have done so because they thought “This is what I expect judges to do, there is nothing special here”.  Yes, that true, we do expect judges to try.  However, trying is less frequent than we would like, and succeeding is even less frequent. We want to encourage it when we see it.

Mentoring is sometimes something difficult, and by mentoring other judges we help them growing as a judge. So a good mentor is something we would like to see at all events. So we would accept this nomination. 83.4% of the judges would also accept it, so we agree again!
[Bryan’s Thoughts]  This is exactly the sort of thing that we want to see in judges.  The 16% that deferred this might have done so because they thought “This is what I expect judges to do, there is nothing special here”.  That true is somewhat true, especially at the higher judge levels, but quality mentorship, where the mentee truly gets something out of it, is rarer than we would like, so we want to encourage it when we see it.

What else can we ask for a nomination?  We have a person that proactively learned a task, rather than waiting to be taught it. They gave meaningful feedback on process improvement, and helped mentor a peer.  We would accept it, same as judges who answered the survey (86.6% would accept it).

Again, a very specific nomination, giving specific examples and situations that justify it. The nominated judge did a very good job both as Judge Manager and Area Captain, and trained his successor so that he could continue the task. Mentoring and working for the community is something we like to see in judges, so we would accept this nomination. Same as the community, with an 85.5% of the answers positive.

A vast majority of the judges who answered the survey would defer this nomination (84.1%). We would also defer it, but the team was a bit divided on this nomination. Some of us think that we need more information to accept it. The nomination seems very vague, and requires the reader to guess to fill in the gaps on what was actually done. Some within the team think that, even though it’s short, it contains enough information, since it seems that the Head Judge wrote a very good report from the tournament so that it was very useful for other judges. As you can see, sometimes we also have different opinions on one same nomination.

This nomination is very vague. We need more information to accept it. It seems that the nominated judge actually did a very good job at that GP, but we can’t actually identify it. By just reading the nomination, we can see things that almost all judges do while judging at GP’s if they have been around for a while. In this kind of nomination, we ask the judge that wrote it to write it again during the following wave with the additional information (the nominated judge would have an additional spot to do that). This nomination is what can be thought of as a trap.  There are lots of positive words in it, ‘solid support’, ‘knowledge’, ‘experience’, ‘efficient’, and you finish the nomination with a positive impression. However, when you go back and re-read the nomination, you struggle to find anything discernible. Judges who answered the survey think the same as the Exemplar Team: 80.3% would defer it.

Judges answering the survey were completely divided with this nomination. 50.7% would accept it, and 49.3% would defer it. The Exemplar Team has no doubts in this kind of nominations, and would defer it. Similar to nomination #4, judging events is something between the TO and the judge, so if they agree on terms and compensation there’s nothing significant on it.  We don’t consider judging frequently particularly relevant or praiseworthy. It just is. We do value what you make of yourself and the events in the frequent opportunities you create for yourself.

In this nomination we all have the same opinion: It’s a good nomination. Answering calls on the floor is something that we all do, but the judge went farther, and showed us that being a leader is not confined to a Team Lead role.The judge took care of his team and others, during a long tournament, and made sure that everybody had a good time and were not exhausted by the tournament. We all would accept this nomination (84.8% of the judges who answered the survey would accept it)

74.1% of the judges would defer this nomination, which is the majority of the judges who answered the survey. The Exemplar Team agrees, but there were different opinions, similar to question number 9. In this nomination we have something not significant (frequently judging events) combined with something expected (correct rulings) and something potentially exemplary (good customer service).However we need examples or a description related to the customer service in order to accept the nomination.

This is something that goes beyond what is expected from a judge. The nominated judge found an issue that was affecting other judges’ feelings, and acted in order to solve this issue, in a way that everyone could benefit from the tournament. So this is  a very very good nomination, and we all agree that we should accept it. The Exemplar Team has no doubts about it, and you also don’t have many doubts (89.3% would accept it).
[Bryan’s Thoughts: This is my favorite nomination in the survey.  And if you are reading this thinking “yeah, but I expect L3s to do this…this nomination was for an L1!]

Again, we find a very good nomination. It combines good mentoring to other judges, with making events better to all staff, all combined with specific examples, … By reading this nomination we all know what the nominated judge did during the weekend, so we would accept it. And 80.3% of the judges who answered the survey also would accept it.

These types of nominations tend to be contentious. In this nomination, judges who answered the survey were divided (53.4% would defer it against 46.6% of judges that would accept it) similar to the Exemplar Team. Some us think that this is a something that we want to see in judges at big events, since it motivates other judges to do their job (so some of us would accept it), while some of us think that this is something clearly good but a rather low bar for Exemplar, and while a nice thing, is not a nice judge thing not actually related to judging, and it’s more a Tournament Organiser issue, it’s something that the Exemplar Program should not take into account. Finally there were more votes for deferring the nomination rather than accepting it, but we were really divided on this one. Again, sometimes things are not as easy as they seem.
[Bryan’s Thoughts: I personally dislike
nominations like this.  I think the bar for Exemplar must be higher buying a bag or two of candy and leaving it in the Judge area, and I don’t want judges thinking the way to get Exemplar nominations is by spending money at the grocery store. However, this is also the type of nomination where I realize there is a lot of disagreement on and it can impact a judges event positively.  I’ve said in the past that there is a band of nominations that are subjective and could be accepted or deferred based on the reviewer. I’m ‘fine’ with this type of nomination being in that band.]

And finally a nomination for working on projects. Working on projects is something that many judges do, since our community is based on those projects and they are necessary for the community. But this nomination is not just for working, this nomination is for going beyond that point. Sometimes a project needs fresh air in order to continue and not disappear, and sometimes a project needs a judge that participates hard in order that it can continue helping the community. We want to see judges contributing in a positive way in projects, so we would accept this nomination. And a 90.3% of the judges who answered the survey would also accept it.
[Bryan’s thoughts:  Revitalizing a struggling project? If anything in the program needed to be held up as an example for others to follow, its that.]

We hope that you enjoyed answering the survey and reading the analysis we prepared for you. Putting yourself in the mindset of the reviewer can give you insight into how we look at things. We hope that this has served to understand better our task reviewing nominations.

See you in the next survey!!!


As a reminder, Exemplar Wave 16 closes on Tuesday Feb. 19th at 23:59 PST