{"id":533,"date":"2016-03-21T16:52:33","date_gmt":"2016-03-21T16:52:33","guid":{"rendered":"http:\/\/blogs.magicjudges.org\/exemplar\/?p=533"},"modified":"2016-03-21T16:52:33","modified_gmt":"2016-03-21T16:52:33","slug":"wave-4-nominations-posted","status":"publish","type":"post","link":"https:\/\/blogs.magicjudges.org\/exemplar\/wave-4-nominations-posted\/","title":{"rendered":"Wave 4 Nominations Posted"},"content":{"rendered":"
<\/div>\n
\n

\"\"<\/p>\n

Written by Bryan Prillaman
\nLevel 3, United States, Florida<\/p>\n<\/div>\n

Hello all,<\/span><\/p>\n

Let\u2019s cut straight to it. \u00a0Nominations from Wave 4 have been posted! \u00a0Head on over to JudgeApps and check out the wonderful things we are saying about each other.<\/span><\/p>\n

This past wave we drastically increased the number of slots each L2+ judge was allocated, and that resulted in nearly double the total nominations and over 1800 unique judges recognized! \u00a0To give perspective, that\u2019s over 6 times the total staff of every judge\/ scorekeeper\/ admin at GPDC, and nearly 150% the number of unique judges for an entire year of GPs!<\/span><\/p>\n

If you received a nomination, we will be sending you an email in the next day or two to get your mailing address and let you know the next steps.<\/p>\n

That\u2019s all we have for this update! \u00a0Thanks for joining us! \u00a0Check back next time when\u2026<\/p>\n

Next Time? \u00a0How about now? \u00a0I mean, I could talk about something.<\/p>\n

The Most Common Question<\/b><\/p>\n

One question I get often is: \u00a0\u201cHow come it takes so long for the nominations to come out?\u201d It\u2019s a question I hear frequently at varying degrees of candor. \u00a0(But not Kandor.)<\/span><\/p>\n

When the Window closes, we begin reviewing nominations. In theory, we could start earlier. \u00a0In practice, there isn\u2019t much point. \u00a0Nominations can be added\/deleted\/edited right up to the last minute, and well, you use those last minutes. \u00a0The graph of nominations over time looks like a Viking warship with the most intimidating prow you have ever seen.<\/span><\/p>\n

The system to review nominations was developed by Gavin Duggan and is pretty slick. Each member of the team logs into a special screen in Judgeapps and is presented with 10 random nominations based on the language they selected. \u00a0Finish those 10, and you can move on to another 10\u2026and another, or take a break. \u00a0This divides the task up into small manageable bites that can be done during a commercial of a television show or while waiting for a bus.<\/span><\/p>\n

Each nomination gets a disposition. \u00a0The dispositions are:<\/span><\/p>\n

No Comment<\/b> \u2013 Typically reserved for a language you can\u2019t read or it is for or by the reviewer
\n<\/span>Promote<\/b> \u2013 This nomination is TEH AWESOMZ! \u00a0Mark it for an article later.
\n<\/span>Accept<\/b> \u2013 This state is the default. \u00a0While not TEH AWESOMZ, it doesn\u2019t have any blocking issues. \u00a0The vast majority of nominations fall into this category.
\n<\/span>Dispute<\/b> \u2013 This nomination has an issue with it. \u00a0Generally, it\u2019s because it\u2019s too vague, but there may be other problems as well. \u00a0These nominations go through the \u2018Dispute\u2019 process. Additionally, all nominations for and by suspended judges also get this assignment.<\/span><\/p>\n

The process above takes about 2 weeks, or at least it did this time. \u00a0We also have a forum where we can discuss specific nominations, and get a sanity check from other members of the team. \u00a0It\u2019s good to keep yourself calibrated, as it is easy to slip into a trend of being too strict or too lax. Typically about 5% of the reviewed nominations get \u201cPromote\u201d and 10% get \u2018Dispute\u2019.<\/p>\n

For this wave, we didn\u2019t check them all. \u00a0Nor will we in future waves. \u00a0Like deck checks at an event, we don\u2019t check them all. \u00a0But unlike deck checks, we do check most. This is, more than anything, a concession to the amount of time it takes to touch every nomination. \u00a0For this wave, we checked about 70%. \u00a0We are currently teaching some natural language software what disputed nominations look like, but it will be a while before it\u2019s ready.<\/span><\/p>\n

Nominations Needing Further Review<\/b><\/p>\n

Now comes the hard part. The first thing I do is take a pass through all the disputed nominations and make a cut. \u00a0Remember I said about 10% of the nominations are flagged for further review? \u00a0Well, for this wave, that means over 300 nominations need to be looked at again. \u00a0I want to reduce the workload on the team, and since the final call comes down to me, I go through all the disputes and break them down into three categories: <\/span><\/p>\n

Accepted<\/b> \u2013 I think the nomination is fine. \u00a0This is effectively an approval.
\n<\/span>Rejected<\/b> \u2013 Nominations that get this status will not be included in the Wave. \u00a0This is used rather sparingly, and is reserved for nominations that have a fundamental flaw that renders them unsalvageable.
\n<\/span>Not Contacted<\/b> – Accepted means it can go through without a change. \u00a0Rejected means it\u2019s not going through. \u00a0Every other nomination gets a status of Not-Contacted. \u00a0This means that the nomination needs to be poked at and the author has not yet been contacted. \u00a0The majority of nominations get this state. \u00a0\u00a0<\/span><\/p>\n

There are other states too. \u00a0For Example, the status \u2018Contacted\u2019 means we have contacted the author. \u00a0There is also a “Done” state. \u00a0Can you guess what that means?<\/p>\n

At this point the team begins working off the list. A team member will self-assign their name to a nomination and then they are the responsible individual for it. \u00a0They can decide to reject or approve the nomination, but at this point, it\u2019s typically going to need a modification at some point as at least two other people have said it needs a revision.<\/p>\n

We write an email to each author of a nomination that needs further work. \u00a0This is the most time consuming part because it can\u2019t be a form letter. \u00a0Each nomination is different, and requires a different approach. \u00a0Also, each judge is different and responds differently to different approaches. \u00a0For example, a judge in the SE that I know well, the letter will be more direct with less fluff. \u00a0For an unknown judge in another region, we tend to spend more time explaining what we are looking for. \u00a0We quote each nomination and then explain what the issue is and typically give a mild push in a direction that can fix it. \u00a0The goal of this email is to get a revision that we can replace the old nomination text with. \u00a0While this process is time consuming, and sometimes there is a significant amount of back and forth, we feel it\u2019s necessary to \u2018save\u2019 as many nominations as possible. \u00a0Also, a really cool consequence of this is that the revisions we get back often turn the nomination into one of the better ones. This is a great return on our investment of time. \u00a0Additionally, the hope is that once we work with an author on what sort of information we are looking for, future nominations will also be strong. \u00a0In the next few weeks Rob McKenzie will be posting an article inspired by his time on the dispute team. \u00a0Rob was also the MVP for this wave. This blog post would probably be two weeks later if not for him. \u00a0Tell him ‘Thanks’ next time you see him.<\/p>\n

If the judge updates the nomination, the new text is re-reviewed by the person who claimed it. A re-re-submission is very rarely needed. \u00a0The new text is plugged into a spreadsheet, and I later go back through the spreadsheet and update the nomination in Judgeapps with the new text.<\/p>\n

About a week out from when we are going to release the nominations, I send an email out to everyone who has been contacted, but hasn\u2019t replied yet. \u00a0This is a \u2018last chance\u2019 email. \u00a0Then, the night before the release, I make one more pass of the remaining unresolved disputes, and reject or accept. \u00a0In previous waves this was a very hard decision; Waves were far apart and the number of nominations were relatively small. \u00a0Now? \u00a0Waves are 3 months apart and there are a lot more slots. \u00a0It\u2019s still a tough call, but less so as the Judge can just re-nominate next wave.<\/p>\n

Then, the final part, is that I write up a blog post. \u00a0That\u2019s what I\u2019m doing right now, and I see the light at the end of the tunnel. \u00a0Be sure to join us next time when Rob McKenzie brings an article on how to turn a Disputed Nomination into a Promoted Nomination.<\/span><\/p>\n

See ya next time.<\/p>\n","protected":false},"excerpt":{"rendered":"

Wave 4 nominations are posted. Head over to Judge Apps to see the awesome results, then come back here and read about how we review nominations<\/p>\n","protected":false},"author":216,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[3,4],"tags":[],"language":[9],"class_list":["post-533","post","type-post","status-publish","format-standard","hentry","category-announcements","category-recognition-windows","language-en"],"_links":{"self":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts\/533","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/users\/216"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/comments?post=533"}],"version-history":[{"count":3,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts\/533\/revisions"}],"predecessor-version":[{"id":536,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts\/533\/revisions\/536"}],"wp:attachment":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/media?parent=533"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/categories?post=533"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/tags?post=533"},{"taxonomy":"language","embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/language?post=533"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}