Hello all,

Let’s cut straight to it.  Nominations from Wave 4 have been posted!  Head on over to JudgeApps and check out the wonderful things we are saying about each other.

This past wave we drastically increased the number of slots each L2+ judge was allocated, and that resulted in nearly double the total nominations and over 1800 unique judges recognized!  To give perspective, that’s over 6 times the total staff of every judge/ scorekeeper/ admin at GPDC, and nearly 150% the number of unique judges for an entire year of GPs!

If you received a nomination, we will be sending you an email in the next day or two to get your mailing address and let you know the next steps.

That’s all we have for this update!  Thanks for joining us!  Check back next time when…

Next Time?  How about now?  I mean, I could talk about something.

The Most Common Question

One question I get often is:  “How come it takes so long for the nominations to come out?” It’s a question I hear frequently at varying degrees of candor.  (But not Kandor.)

When the Window closes, we begin reviewing nominations. In theory, we could start earlier.  In practice, there isn’t much point.  Nominations can be added/deleted/edited right up to the last minute, and well, you use those last minutes.  The graph of nominations over time looks like a Viking warship with the most intimidating prow you have ever seen.

The system to review nominations was developed by Gavin Duggan and is pretty slick. Each member of the team logs into a special screen in Judgeapps and is presented with 10 random nominations based on the language they selected.  Finish those 10, and you can move on to another 10…and another, or take a break.  This divides the task up into small manageable bites that can be done during a commercial of a television show or while waiting for a bus.

Each nomination gets a disposition.  The dispositions are:

No Comment – Typically reserved for a language you can’t read or it is for or by the reviewer
Promote – This nomination is TEH AWESOMZ!  Mark it for an article later.
Accept – This state is the default.  While not TEH AWESOMZ, it doesn’t have any blocking issues.  The vast majority of nominations fall into this category.
Dispute – This nomination has an issue with it.  Generally, it’s because it’s too vague, but there may be other problems as well.  These nominations go through the ‘Dispute’ process. Additionally, all nominations for and by suspended judges also get this assignment.

The process above takes about 2 weeks, or at least it did this time.  We also have a forum where we can discuss specific nominations, and get a sanity check from other members of the team.  It’s good to keep yourself calibrated, as it is easy to slip into a trend of being too strict or too lax. Typically about 5% of the reviewed nominations get “Promote” and 10% get ‘Dispute’.

For this wave, we didn’t check them all.  Nor will we in future waves.  Like deck checks at an event, we don’t check them all.  But unlike deck checks, we do check most. This is, more than anything, a concession to the amount of time it takes to touch every nomination.  For this wave, we checked about 70%.  We are currently teaching some natural language software what disputed nominations look like, but it will be a while before it’s ready.

Nominations Needing Further Review

Now comes the hard part. The first thing I do is take a pass through all the disputed nominations and make a cut.  Remember I said about 10% of the nominations are flagged for further review?  Well, for this wave, that means over 300 nominations need to be looked at again.  I want to reduce the workload on the team, and since the final call comes down to me, I go through all the disputes and break them down into three categories:

Accepted – I think the nomination is fine.  This is effectively an approval.
Rejected – Nominations that get this status will not be included in the Wave.  This is used rather sparingly, and is reserved for nominations that have a fundamental flaw that renders them unsalvageable.
Not Contacted – Accepted means it can go through without a change.  Rejected means it’s not going through.  Every other nomination gets a status of Not-Contacted.  This means that the nomination needs to be poked at and the author has not yet been contacted.  The majority of nominations get this state.   

There are other states too.  For Example, the status ‘Contacted’ means we have contacted the author.  There is also a “Done” state.  Can you guess what that means?

At this point the team begins working off the list. A team member will self-assign their name to a nomination and then they are the responsible individual for it.  They can decide to reject or approve the nomination, but at this point, it’s typically going to need a modification at some point as at least two other people have said it needs a revision.

We write an email to each author of a nomination that needs further work.  This is the most time consuming part because it can’t be a form letter.  Each nomination is different, and requires a different approach.  Also, each judge is different and responds differently to different approaches.  For example, a judge in the SE that I know well, the letter will be more direct with less fluff.  For an unknown judge in another region, we tend to spend more time explaining what we are looking for.  We quote each nomination and then explain what the issue is and typically give a mild push in a direction that can fix it.  The goal of this email is to get a revision that we can replace the old nomination text with.  While this process is time consuming, and sometimes there is a significant amount of back and forth, we feel it’s necessary to ‘save’ as many nominations as possible.  Also, a really cool consequence of this is that the revisions we get back often turn the nomination into one of the better ones. This is a great return on our investment of time.  Additionally, the hope is that once we work with an author on what sort of information we are looking for, future nominations will also be strong.  In the next few weeks Rob McKenzie will be posting an article inspired by his time on the dispute team.  Rob was also the MVP for this wave. This blog post would probably be two weeks later if not for him.  Tell him ‘Thanks’ next time you see him.

If the judge updates the nomination, the new text is re-reviewed by the person who claimed it. A re-re-submission is very rarely needed.  The new text is plugged into a spreadsheet, and I later go back through the spreadsheet and update the nomination in Judgeapps with the new text.

About a week out from when we are going to release the nominations, I send an email out to everyone who has been contacted, but hasn’t replied yet.  This is a ‘last chance’ email.  Then, the night before the release, I make one more pass of the remaining unresolved disputes, and reject or accept.  In previous waves this was a very hard decision; Waves were far apart and the number of nominations were relatively small.  Now?  Waves are 3 months apart and there are a lot more slots.  It’s still a tough call, but less so as the Judge can just re-nominate next wave.

Then, the final part, is that I write up a blog post.  That’s what I’m doing right now, and I see the light at the end of the tunnel.  Be sure to join us next time when Rob McKenzie brings an article on how to turn a Disputed Nomination into a Promoted Nomination.

See ya next time.