Riding the (second) wave
The new Grand Prix Head Judges from Wave 2 have been selected (congratulations to you all!) and the project is now able to go into an introspection phase of the second wave to learn as much as possible from it and to make Wave 3 even better for everyone.
The process for the selection of the head judges is based on 3 core principles:
- Quality: We want to select the most suited for the job. Running a GP is a difficult task and it is important that certified Head Judges are skilled enough to make the event a success. That’s why the questions are focused on the core qualities of running an event, and properly interacting with others plays a non-negligible role in running a great event.
- Transparency: We tried to make the process as transparent as possible. We shared the whole process with the candidates beforehand with tentative dates to ensure everyone knew what they were getting into and we published the questions in an offline mode to every candidates at the start of each step. We were also available at all time to answer questions and shared both questions and answers to all candidates to be sure we would be fair to everybody.
- Fairness: The idea behind the New New World Order is a sense of “everyone has a chance”. We wanted to take this principle and anchor it at the heart of the selection process.
Getting better: What we changed between Wave 1 and 2
You can find the debrief of Wave 1 here (only accessible for Level 3 judges). Here are a few more details we can share with you all.
Time to breathe
One of the biggest changes between the first and second waves was a much more extended window of submission. As we weren’t bound by an externally defined deadline anymore, we were able to give candidates a much more comfortable window so that they could come up with the best possible application. We looked at the timeline not just in term of how many days we wanted to allocate to each step but also at the different events taking place during the phases.
Most candidates visit GPs and have to prepare for it either helping the TO or the HJ or being one of the two. Creating a 14 days window for a small step might look like a lot but can still be too short if you remove the days blocked for other things like travel, event preparation, significant others, etc.
After receiving some feedback on our timeframe, we also made it a point to have all our deadlines finish on a Tuesday or Wednesday. It sound kind of counterintuitive at first as we almost always associate week ends with free time but most candidates have GPs and other big events to judge and those just so happen to be on the weekend, making Saturday / Sunday the busiest days of the week.
Every step counts
In the first wave, if you passed the first step, you got a bunch of questions as the second step and were evaluated only on those. We decided that, starting with the second wave, the questions in the first step would count in the general evaluation of the candidates. In hindsight, It seems obvious that everything a candidate hands in should count in his final evaluation as the questions in the first step also shows the proficiency of the candidate in the role. It took us a while to realise this simple and obvious fact but we got there eventually.
Focus, focus, focus
Some of the questions we had in the first wave were very open ended or assumed a context that was hard to interpret. One or two question even drifted into the purely philosophical. This made the evaluation of the responses very difficult because the direction of some of the answers took the evaluator a little by surprise.
In the second wave, we tried to minimize the open ended questions and asked for more concrete aspects that could be evaluated using a predetermined set of criteria.
Conceal, don’t feel, don’t let them know
We devised a lot of mechanics to remove subjectivity from the evaluation. Unlike Wave 1, we tried to conceal the identity of the candidate as much as possible. We restricted the access of the answers to 1-2 questions per committee members by exporting the answers from the google sheet to different google docs (1 doc per answer).
We also created evaluation criteria for each question before handing out the candidates’ answers. The new system to anonymize all responses was used from the beginning of the wave on all essay questions. We believe we went further in running a system where the answer was more important that the person who wrote it.
If a candidate was to come with answers which would differ from the Evaluation Chart, as long as they made sense and were duly justified, evaluators were encouraged to consider them as potentially acceptable. No idea is bad just because it is new or deviant. Every new idea can be great, provided it is the consequence of a well-thought and well-explained reasoning.
Everyone’s a winner
Taking part in the selection process as a candidate in both waves meant taking on A LOT of work. From the survey we created after the second wave, we know that candidates had to pour an average of 4 hours for the first step and 20 hours for the second one. This is a huge amount of work!
We decided when we created the concept of the second wave that we wanted to be able to give everyone at least something in return for their work, even if the candidate wasn’t selected as a Grand Prix Head Judge. The basic principle was the following: Each evaluator had to explain why an answer was considered sub-par for a given candidate. We then consolidated the received feedback and tried to find pattern between the comments. This will give each candidate as much feedback as we could based on what we were able to see. We are still aggregating the feedback and will send it to all candidates soon.
Removing quotas
At the beginning of the second wave, we posted that, as the term of six head judges was expiring, there would be six slots available. After Wave 1, we got the feedback that this felt like a relic of an old boy’s club trying to protect their privileges. On our end, based on the results from Wave 1, we were afraid that setting a clear passing bar may end up with too few candidates passing, which could lead into a shortage of available GP HJs to sustain the GP needs.
This was a tough call to make since there were many pros and cons but after discussing it extensively, we felt that since we were not strongly-minded either way while there were some good arguments given against it, we should waive it… and so we did!
The next challenge arose when we realized we had no data to try to define that bar. So we looked at other exams we have in the Judge Program and felt we could use the L3 process as a source of inspiration. We defined the minimum passing criterias (at most x below average items) and how much (if) being excellent in one or more field(s) could mitigate being below average in another.
Some bumps in the road
It is very difficult to have bumps in a wave but sadly, Wave 2 wasn’t as smooth as we had hoped for and we encountered an unexpected problem along the way. The email address we created to contact the selection committee was gphj.program@gmail.com and the group email used for communication between the different GP HJs also starts with “gphj”. If you combine this fact with email programs that tries to help users with an autocomplete feature, you might get emails sent to the wrong address.
It happened twice that an email that should have gone to the selection committee went to the GP HJ list. In both cases, we thoroughly evaluated the content and the potential impact of the compromised items to check whether we should modify the process to mitigate or negate the effects of these two mistakes.
It turned out that, since we had already made the decision to move away from “competition between candidates” to “reaching a passing score”, the information shared would only have a limited impact since it could not be detrimental to candidates who didn’t have access to it.
The unseen heroes
We already mentioned the amount of work poured by the candidates but we wouldn’t have been able to do it if it weren’t for the tireless and mostly unseen work of all the members of the committee of the second wave who actively shaped the questions, the evaluation criteria and evaluated all candidates in the Christmas season.
What’s to come
New contact
We changed the email address of the group to prevent any misunderstanding as to what group was who. Anyone can now contact us with the following email address if they wish to give feedback and/or would ask a question: mtg.gphj.committee@gmail.com
Merging Step 1 and 2
Step 1 and 2 will be just two stage in the same evaluation and will be used to remove candidates that we think are not yet ready to continue with the process. Each question of the first step will count as much as the questions in step 2 and might therefore be more time consuming and involve the judges more. We think this is a good thing as the feedback we got from the second step was that candidates learned a lot just by having to think about the situations and what they would do.
Security will be a priority
We cut corners to make it more manageable and to speed things up. This is the primary cause why the leak could have been disastrous. If we had been more security conscious and used the normal paradigm of least access, we wouldn’t have had to worry. We will now use this paradigm in the handling of all questions, answers and evaluation criteria. This will probably make the administrative overhead bigger and might make the handling of some of the discussions more difficult for the committee members but it will be for the sake of safety.
Reversing the optional/mandatory deadline
The optional deadline of 3 weeks we had created to allow candidates to receive feedback if the questions weren’t understood was itself either not understood or not well received by the candidates. We don’t yet know exactly how the definitive Step 2 process will be for the next wave but we will most probably remove the optional deadline for a more familiar approach of a mandatory deadline, followed by an evaluation with optional follow-up questions and an optional resubmission.
Feedback, Feedback, Feedback
Having an exam style evaluation for new candidates is a great way to see if they know how to head judge a tournament at the GP level but is sub-optimal for judges who are already HJ GPs. It would make a lot more sense to actually evaluate their current performance instead of an exam every 15 months. Starting with the 2017 GPs, we will publish a feedback form aimed at evaluating the HJ and AJ but also to get general feedback on the event. The form (http://tinyurl.com/GPHJ-Feedback) will be open to all certified judges and all submitted information will be handled with absolute confidentiality.
Using this form will allow HJs to receive feedback on their performances and will allow them to get better. It will also allow new GP HJs to get faster up to speed on where they need to work on. If a HJ want to do an experiment (try a new DC style or a new team grouping), they will also receive more direct feedback on how it was perceived.
Filling out the form will, of course, not be mandatory for non-HJ/AJ but we really hope that each judge will integrate it into their “finishing a GP” routine. The form doesn’t take long to fill out and will be one of the most important evaluation criteria for GP HJs… This means, basically, that YOU are able to help us say who is, in your eyes, a good GP HJ or not.