Wave 2 - The Numbers - The Exemplar Program

A message from Kim:

“Me again!

Lots of people have been asking us for statistics on wave 1. I’m pleased to be able to present the results of some analysis of Wave 2 performed by a whole team of judges. Before I hand you over to Eric, though, a couple of reminders:

-If you were recognised in Wave 2 and have not yet provided us with your current mailing address via the Google Form that was emailed out (and you are not a judge based in either Russia or China), you have one week left to do it. I’ll be sending all of the mailing addresses to Andy on the morning of July 8th (BST). If you were recognised and did not receive the email linking you to this form, you need to contact us at ExemplarProgram@gmail.com in order to let us know as soon as possible so we can give it to you.

-There are about five weeks left to put in your recognitions for Exemplar Wave 3. Some people have mentioned being unwilling to enter recognitions early in the window in case they want to change them later; I’d like to take this opportunity to remind you that you are able to edit or delete recognitions that you have entered up to the point that the window closes, so there is no need to hold off to the last minute in case you change your mind. Recipient level is only verified against judge apps at the point that the recognition is submitted, so you don’t need to worry about a recognition being invalidated by the receiving judge levelling up part-way through.

And that’s done! Now for the post you have been waiting for…

Written by Eric Levine
Level 3, USA

Hey everyone! As we move forward into Wave 3 of the Exemplar program I want to talk a little bit about Wave 2, where Level 2 judges were added to the list of people who could submit recognitions. The hard-working team of Bryan Prillaman, Joe Hughto, Ronald Thompson, and Paul Baranay has taken the Wave 2 Exemplar data and wrangled it into a usable form, and I’m here to highlight some of the results.

What would data be without takeaways, though? Not very useful, that’s what. With each tidbit I give you I’m going to give you my opinion on the takeaway – that way, we can actually give meaning to these numbers I’m throwing around. Let’s get started with some interesting summary statistics:

62% of all eligible judges submitted at least one recognition. For a wide-angle program with global implications, I think 62% is a pretty good starting point. As we learn more about what a good recommendation is and how we can consider this program in our day-to-day judging lives, I think we can increase this number. Breaking this number down further, 59% of L2 judges and 87% of L3 judges submitted at least one recognition. Obviously the size of the L2 population (1,032 L2 judges compared to 132 L3 judges during Wave 2) means that they’ll be carrying that overall percentage figure for the foreseeable future.

Takeaway:
The Exemplar program had a good debut in terms of overall participation, but there’s room to grow – if you chose not to give a recognition the first time around, consider why and let us know at exemplarprogram@gmail.com if you have ideas for improvement. If you simply didn’t find the time, start early – Wave 3 is open!

So, time to go deeper and discuss some specific pieces of data, including some points of concern I’ve heard brought up from time to time. Let’s start first with where our exemplar recommendations went in terms of levels. I saw some conversations online that indicated concern about Level 1 judges not receiving recognition through the Exemplar program. I’m happy to report that 39% of Exemplar recognitions worldwide went to Level 1 judges! Here’s a little more detail in the form of a bar chart:

The Level 1 population and the Level 2 population received about the same amount of recognitions. As L2 judges are often the face of Competitive REL, and as many L1s interact mostly with a single local store, it makes sense that Level 2 judges would be well represented. That doesn’t mean there aren’t L1 judges out there working by themselves in stores doing exemplary jobs – of course there are. Oops, I’m eating into my “takeaway” section…

Takeaway:
Level 1 judges received a substantial amount of recognitions through Wave 2 of the Exemplar Program. We could probably find more Exemplary L1 behavior by heading to nearby stores and playing or judging there – L2s, I’m looking at you and that additional L1 slot you got for Wave 3!

Another concern I heard mentioned a few times was that judges would simply recognize other judges of the same level and leave lower-level judges feeling unmentioned and unfulfilled. I’m pleased to be able to tell you that, as a group, we used about 62% of our recognition slots on judges lower than us. Here are some more specific stats, focusing on L2s and L3s as they make up the bulk of this population:

51% of recognitions written by L2s went to L1s
42% of recognitions written by L3s went to L2s, and 29% went to L1s

(Fun fact: L5 judges used more of their “Any Level” slots on L1 judges than they did on judges of any other level – obviously not a huge population to look at, but still cool!)

Takeaway:
We’re not hesitant to recognize judges of lower levels – in fact, we seem to be rather excited about it! Of course, some slots are level-locked to judges of lower levels, but that’s by design, and I believe the data we have are telling us that the system is largely working the way it’s supposed to in this regard.

The last concern I’d like to address is the question of whether or not the Exemplar program was being used for “back-scratching” – specifically, whether judges were creating reciprocal recognitions simply to get one another recognized rather than in the spirit of the program. There are a few ways to examine this, but I think the best approach is the most direct one.

In about 8% of recognitions, the creator had been recognized by the subject as well. Put another way, 92% of recognitions don’t even require examination for this behavior, which is great – if this is a problem, it’s not a big one. I dove further into these recognitions and noticed a few things:

I recognized a lot of the names involved. Many of these judges work together regularly at events around the world or at large events in their own regions. While there are a few ways this could be interpreted, I think the simplest explanation fits best here: these judges see the great things the other ones do because they work together frequently.
Quite a few RCs showed up, indicating that they were working hard to recognize exceptional judges in their regions and that those judges also happened to recognize the hard work of those RCs. I think that’s a really good reason for reciprocal recognitions to happen organically!
I took a look at quite a few of the recognitions that popped up in this search, and to my delight, none seemed problematic. Even after looking through about 40% of the recognitions that fit this criterion, I didn’t find any that felt like “quid pro quo.” Instead what I found was pretty heartwarming. Examples:

Exemplary judges recognizing other exemplary judges in their region.
Judges doing verifiably strong work on large-scale projects, each recognizing the contributions that the other made.
One judge seeing an exemplary behavior in another at an event, exhibiting that behavior (or an improved version) at another event, and being recognized for learning/adapting/stepping up in a big way.

The overall rate of recognition rejection is extremely low – we only rejected or modified about 1%! For some good information about how to avoid having a recognition rejected, check out Jack Doyle’s excellent article.

Takeaway:
“Quid-pro-quo” recognitions probably aren’t happening very much, – maybe not even at all. Instead, we have judges recognizing each other organically for hard work and motivating one another to even greater heights. As we receive more data, we plan to look into larger clusters to learn more about what drives these organic reciprocal recognitions.

I’ve also got some cool data points to share that I think are interesting and fun! Here goes:

In the Spain & Portugal Region, 94% of eligible judges in that region wrote at least one recognition! Other all-star regions by this metric are Brazil (86%), Europe-Mid East (84%), Japan (78%), China (78%), and USA-Southeast (75%).
L2s in Spain & Portugal, Brazil, and Japan used 70% or more of their L1-specific slots, which is a great sign for community growth.
L3s in USA-Midwest, Spain & Portugal, Japan, Latin America – Spanish, and China used 70% or more of their L1-specific slots – again, this is a strong indicator of L3s supporting L1s.

There are a lot of people working hard in the Judge program, and I am glad to see some of that hard work being recognized through the Exemplar Program. I want to leave you with one last thought: For Exemplar Wave 3, try to expand your sphere of influence. If most of the Wave 2 recognitions you wrote came from large events like GPs, SCG Opens, or similar, try to look out for exemplary behavior on a local level as well. Maybe there are judges doing public-facing work online that you could recognize, or perhaps you know someone who just did some enormously difficult work on a judge project that has the potential to really help the program. Keep your eyes open for our next Exemplars!