Hello all,

Today, we are going to announce a change to the ongoing Exemplar wave ending May 2nd 23:59 PST.

Exemplar has had three waves without any visible changes. We’ve made some adjustments behind the scenes to speed up the process, but some changes can’t be invisible.  If we are going to move forward, then we have to be willing to try new things.

There are two consistent pieces of constructive feedback we get. One is that there is too much time between the end of the wave and when the nominations are released. The other is that judges with deferred nominations are frustrated because there is no direct feedback for why the nomination was deferred. For Wave 9, the Exemplar team is going to try an experiment to address both of those issues.

Before I get into the details of the experiment, I want to briefly explain the behind-the-scenes process of reviewing nominations. When a wave ends, the Exemplar team makes an initial pass at nominations. Nominations are separated into two categories: “Accepted” and “Needs a second look”. We then take all the second-look nominations and give them a second look. From there nominations are separated into three categories:

• ACCEPTED – This nomination is fine, let it through
• CONTACT – This nomination is almost fine, but we need revision from the author to include it.
• DEFERRED – Nomination is not fine, do not include it.

The CONTACT nominations are where we spend the bulk of our time and energy. Writing emails, waiting for replies, clarifying a response, waiting for replies, etc can take up to a month. It impacts the release timeline, requires a large amount of manual tracking, and is mentally draining. Getting revisions for the nominations that are ‘almost there’ has a lot of value to it, but the energy required to do it takes away from other initiative that might also have value, like sending emails to people explaining why their nominations were deferred.

Now the Experiment: For Wave 9, nominations will only be ACCEPTED or DEFERRED. There will be no emails requesting revisions. Instead, if one of your nominations is deferred, you will receive an email telling you why, after the wave closes.

Right now, this is just an experiment. This is not a permanent change. Inside the team we have spent some time discussing the pros and cons of this change. It’s rare that any decision is unanimous and this wasn’t one of them, but overall, I think the potential benefits outweigh the negatives enough to spend a wave investigating and see the reaction to it.

So let’s talk about those pros and cons.

Pro – Shorter releases – We no longer have to allow for replies back from emails we send. This has the potential to reduce the process by two or more weeks.

Pro – Judges with nominations deferred will now be told why the nomination was deferred. This should reduce frustration and result in successful resubmissions. This past wave I noticed a nomination that was submitted for a third time with nearly the exact same text. A brief chat with the judge was able to get him to understand what the problem was. Even if this experiment doesn’t transition into the new process, we will have informed a lot of judges as to what we were looking for in their nomination.

Pro (potential) – Higher quality nominations overall. This one is a bit meta. I suspect that some nominations are written hastily relying on the fact that if it’s “close”, they will have the ability to revise the nomination before the wave goes live. There is very little “cost” in writing a vague nomination. By removing this expectation, judges may spend more time on those hastily written nominations.

Con – This means we are going to defer more nominations, possibly as much as an additional 5% of the total nominations. This is the reason some people in the team think this experiment is a bad idea. It works against the philosophy of the project where we want to see as many judges as possible get recognized for their exceptional actions. It’s definitely a concern that we will be watching.

Con – Edge nominations may be adjudicated incorrectly (in either direction). It’s easy to see the difference between a good nomination and a poor one. However the line between a ‘barely ok’, and ‘barely not ok’ is subjective, and that is where a lot of CONTACT nominations come from.

Nomination Guidelines
Here is a quick reminder of what we are looking for in nominations, and you can find the full article here. A good nomination is specific and relevant to the Judge program, for praiseworthy actions, and positively representative of the Judge program.

Some of the most common reasons nominations are deferred:
Too Vague – The author leaves out information and the review team cannot determine what the judge did. Consider re-reading the nomination pretending you know nothing about the judge. Does the nomination have enough information in it?
For Judging events/Travelling – Every judge judges events. There has to be more to it than that. Doing something frequently is not the same as doing something exceptionally.
Working for free\Low Compensation – This is not something the Judge Program wishes to encourage, and Exemplar is about highlighting things we do want to encourage.
Passing an L1/L2/L3 test – This is something that every L1/L2/L3 has done. The bar for exceptional actions has to be higher than the minimum requirement for that level.
Copy/Paste Nominations – Please give your fellow judges personalized nominations. They are so much more meaningful.  This includes reusing nomination text from a previous wave.
Nomination for TO actions – Some TOs are also judges, but having a nice venue, overstaffing judges, having good prize payouts are not judge actions and not actions other judges can emulate.

Hopefully this change will be positive one.  Either way, we plan on looking at the results and feedback from this Wave and determine if that is something we wish to continue into Wave 10. If you have any questions, please email exemplarprogram@gmail.com.

Check back next time for some nomination highlights.