{"id":349,"date":"2015-07-01T08:24:51","date_gmt":"2015-07-01T08:24:51","guid":{"rendered":"http:\/\/blogs.magicjudges.org\/exemplar\/?p=349"},"modified":"2015-07-21T15:57:26","modified_gmt":"2015-07-21T15:57:26","slug":"wave-2-the-numbers","status":"publish","type":"post","link":"https:\/\/blogs.magicjudges.org\/exemplar\/wave-2-the-numbers\/","title":{"rendered":"Wave 2 &#8211; The Numbers"},"content":{"rendered":"<div style=\"margin-top: 10px\"><\/div>\n<hr>\n<blockquote style=\"text-align: justify;line-height: 26px\"><p>A message from <span class='judge-tooltip'><a href='https:\/\/apps.magicjudges.org\/judges\/dci\/1206567626' >Kim<\/a><span class='avatar'><img width='200' height='200' src='https:\/\/apps.magicjudges.org\/dci\/avatar?dci=1206567626&size=200'><\/span><\/span>: <\/span><\/p>\n<p>&#8220;Me again!<\/p>\n<p>Lots of people have been asking us for statistics on wave 1. I&#8217;m pleased to be able to present the results of some analysis of Wave 2 performed by a whole team of judges. Before I hand you over to <span class='judge-tooltip'><a href='https:\/\/apps.magicjudges.org\/judges\/dci\/84750930' >Eric<\/a><span class='avatar'><img width='200' height='200' src='https:\/\/apps.magicjudges.org\/dci\/avatar?dci=84750930&size=200'><\/span><\/span>, though, a couple of reminders: <\/p>\n<p>-If you were recognised in Wave 2 and have not yet provided us with your current mailing address via the Google Form that was emailed out (and you are not a judge based in either Russia or China), you have one week left to do it. I&#8217;ll be sending all of the mailing addresses to Andy on the morning of July 8th (BST). If you were recognised and did not receive the email linking you to this form, you need to contact us at <a href=\"mailto:ExemplarProgram@gmail.com\">ExemplarProgram@gmail.com<\/a> in order to let us know as soon as possible so we can give it to you. <\/p>\n<p>-There are about five weeks left to put in your recognitions for Exemplar Wave 3. Some people have mentioned being unwilling to enter recognitions early in the window in case they want to change them later; I&#8217;d like to take this opportunity to remind you that you are able to edit or delete recognitions that you have entered up to the point that the window closes, so there is no need to hold off to the last minute in case you change your mind. Recipient level is only verified against judge apps at the point that the recognition is submitted, so you don&#8217;t need to worry about a recognition being invalidated by the receiving judge levelling up part-way through.<\/p>\n<p>And that&#8217;s done! Now for the post you have been waiting for&#8230;<\/p><\/blockquote>\n<hr>\n<div class=\"judgeimage\" style=\"margin-right: 20px;margin-bottom: 20px;float: left;padding: 10px;width: 200px;height: 240px;background: #f1f1f1\">\n<img decoding=\"async\" style=\"margin: auto\" src=\"http:\/\/apps.magicjudges.org\/media\/avatars\/Psychatrog\/resized\/200\/EricLevine.jpg\" alt=\"\" \/><\/p>\n<p style=\"margin-top: -23px;line-height: 18px;text-align: center;color: #828282;font-size: 12px\">Written by Eric Levine<br \/>\nLevel 3, USA<\/p>\n<\/div>\n<p style=\"text-align: justify;line-height: 26px\">Hey everyone! As we move forward into Wave 3 of the Exemplar program I want to talk a little bit about Wave 2, where Level 2 judges were added to the list of people who could submit recognitions. The hard-working team of <span class='judge-tooltip'><a href='https:\/\/apps.magicjudges.org\/judges\/dci\/1201757752' >Bryan Prillaman<\/a><span class='avatar'><img width='200' height='200' src='https:\/\/apps.magicjudges.org\/dci\/avatar?dci=1201757752&size=200'><\/span><\/span>, <span class='judge-tooltip'><a href='https:\/\/apps.magicjudges.org\/judges\/dci\/9102402960' >Joe Hughto<\/a><span class='avatar'><img width='200' height='200' src='https:\/\/apps.magicjudges.org\/dci\/avatar?dci=9102402960&size=200'><\/span><\/span>, <span class='judge-tooltip'><a href='https:\/\/apps.magicjudges.org\/judges\/dci\/8213969381' >Ronald Thompson<\/a><span class='avatar'><img width='200' height='200' src='https:\/\/apps.magicjudges.org\/dci\/avatar?dci=8213969381&size=200'><\/span><\/span>, and <span class='judge-tooltip'><a href='https:\/\/apps.magicjudges.org\/judges\/dci\/5223921934' >Paul Baranay<\/a><span class='avatar'><img width='200' height='200' src='https:\/\/apps.magicjudges.org\/dci\/avatar?dci=5223921934&size=200'><\/span><\/span> has taken the Wave 2 Exemplar data and wrangled it into a usable form, and I\u2019m here to highlight some of the results.<\/p>\n<div style=\"text-indent: -9999px\">a<\/div>\n<p style=\"text-align: justify;line-height: 26px\">What would data be without takeaways, though? Not very useful, that\u2019s what. With each tidbit I give you I\u2019m going to give you my opinion on the takeaway \u2013 that way, we can actually give meaning to these numbers I\u2019m throwing around. Let\u2019s get started with some interesting summary statistics:<\/p>\n<p style=\"text-align: justify;line-height: 26px\"><strong>62% of all eligible judges submitted at least one recognition<\/strong>. For a wide-angle program with global implications, I think 62% is a pretty good starting point. As we learn more about what a good recommendation is and how we can consider this program in our day-to-day judging lives, I think we can increase this number. Breaking this number down further, <strong>59% of L2 judges<\/strong> and <strong>87% of L3 judges submitted at least one recognition<\/strong>. Obviously the size of the L2 population (1,032 L2 judges compared to 132 L3 judges during Wave 2) means that they\u2019ll be carrying that overall percentage figure for the foreseeable future.<\/p>\n<hr>\n<blockquote style=\"text-align: justify;line-height: 26px\"><p><strong>Takeaway:<\/strong><br \/>The Exemplar program had a good debut in terms of overall participation, but there\u2019s room to grow \u2013 if you chose not to give a recognition the first time around, consider why and let us know at exemplarprogram@gmail.com if you have ideas for improvement. If you simply didn\u2019t find the time, start early \u2013 Wave 3 is open!<\/p><\/blockquote>\n<hr>\n<p style=\"text-align: justify;line-height: 26px\">So, time to go deeper and discuss some specific pieces of data, including some points of concern I\u2019ve heard brought up from time to time. Let\u2019s start first with where our exemplar recommendations went in terms of levels. I saw some conversations online that indicated concern about Level 1 judges not receiving recognition through the Exemplar program. I\u2019m happy to report that <strong>39% of Exemplar recognitions worldwide went to Level 1 judges<\/strong>! Here\u2019s a little more detail in the form of a bar chart:<\/p>\n<div style=\"width: 360px;height: 670px;margin: auto\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/puu.sh\/iB0tY\/cc2d876547.png\" height=\"662\" width=\"353\" style=\"margin: auto\" \/><\/div>\n<p style=\"text-align: justify;line-height: 26px;margin-top: 10px\">The Level 1 population and the Level 2 population received about the same amount of recognitions. As L2 judges are often the face of Competitive REL, and as many L1s interact mostly with a single local store, it makes sense that Level 2 judges would be well represented. That doesn\u2019t mean there aren\u2019t L1 judges out there working by themselves in stores doing exemplary jobs \u2013 of course there are. Oops, I\u2019m eating into my \u201ctakeaway\u201d section&#8230;<\/p>\n<hr>\n<blockquote style=\"text-align: justify;line-height: 26px\"><p><strong>Takeaway:<\/strong><br \/>Level 1 judges received a substantial amount of recognitions through Wave 2 of the Exemplar Program. We could probably find more Exemplary L1 behavior by heading to nearby stores and playing or judging there \u2013 L2s, I\u2019m looking at you and that additional L1 slot you got for Wave 3!<\/p><\/blockquote>\n<hr>\n<p style=\"text-align: justify;line-height: 26px\">Another concern I heard mentioned a few times was that judges would simply recognize other judges of the same level and leave lower-level judges feeling unmentioned and unfulfilled. I\u2019m pleased to be able to tell you that, as a group, <strong>we used about 62% of our recognition slots on judges lower than us<\/strong>. Here are some more specific stats, focusing on L2s and L3s as they make up the bulk of this population:<\/p>\n<ul>\n<li style=\"line-height: 26px;text-align: justify\">51% of recognitions written by L2s went to L1s<\/li>\n<li style=\"line-height: 26px;text-align: justify\">42% of recognitions written by L3s went to L2s, and 29% went to L1s<\/li>\n<\/ul>\n<p style=\"text-align: justify;line-height: 26px\">(Fun fact: L5 judges used more of their \u201cAny Level\u201d slots on L1 judges than they did on judges of any other level &#8211; obviously not a huge population to look at, but still cool!)<\/p>\n<hr>\n<blockquote style=\"text-align: justify;line-height: 26px\"><p><strong>Takeaway:<\/strong><br \/>We\u2019re not hesitant to recognize judges of lower levels \u2013 in fact, we seem to be rather excited about it! Of course, some slots are level-locked to judges of lower levels, but that\u2019s by design, and I believe the data we have are telling us that the system is largely working the way it\u2019s supposed to in this regard.<\/p><\/blockquote>\n<hr>\n<p style=\"text-align: justify;line-height: 26px\">The last concern I\u2019d like to address is the question of whether or not the Exemplar program was being used for \u201cback-scratching\u201d \u2013 specifically, whether judges were creating reciprocal recognitions simply to get one another recognized rather than in the spirit of the program. There are a few ways to examine this, but I think the best approach is the most direct one.<\/p>\n<p style=\"text-align: justify;line-height: 26px\"><strong>In about 8% of recognitions, the creator had been recognized by the subject as well<\/strong>. Put another way, 92% of recognitions don\u2019t even require examination for this behavior, which is great \u2013 if this is a problem, it\u2019s not a big one. I dove further into these recognitions and noticed a few things:<\/p>\n<ul>\n<li style=\"line-height: 26px;text-align: justify\">I recognized a lot of the names involved. Many of these judges work together regularly at events around the world or at large events in their own regions. While there are a few ways this could be interpreted, I think the simplest explanation fits best here: these judges see the great things the other ones do because they work together frequently.<\/li>\n<li style=\"line-height: 26px;text-align: justify\">Quite a few RCs showed up, indicating that they were working hard to recognize exceptional judges in their regions and that those judges also happened to recognize the hard work of those RCs. I think that\u2019s a really good reason for reciprocal recognitions to happen organically!<\/li>\n<li style=\"line-height: 26px;text-align: justify\">I took a look at quite a few of the recognitions that popped up in this search, and to my delight, none seemed problematic. Even after looking through about 40% of the recognitions that fit this criterion, I didn\u2019t find any that felt like \u201cquid pro quo.\u201d Instead what I found was pretty heartwarming. Examples:<\/li>\n<ul>\n<li style=\"line-height: 26px;text-align: justify\">Exemplary judges recognizing other exemplary judges in their region.<\/li>\n<\/li>\n<li style=\"line-height: 26px;text-align: justify\">Judges doing verifiably strong work on large-scale projects, each recognizing the contributions that the other made.<\/li>\n<\/li>\n<li style=\"line-height: 26px;text-align: justify\">One judge seeing an exemplary behavior in another at an event, exhibiting that behavior (or an improved version) at another event, and being recognized for learning\/adapting\/stepping up in a big way.<\/li>\n<\/ul>\n<li style=\"line-height: 26px;text-align: justify\">The overall rate of recognition rejection is extremely low &#8211; we only rejected or modified about 1%!  For some good information about how to avoid having a recognition rejected, check out <a href=\"http:\/\/blogs.magicjudges.org\/exemplar\/the-anatomy-of-a-recognition\/\">Jack Doyle\u2019s excellent article<\/a>.<\/li>\n<\/ul>\n<hr>\n<blockquote style=\"text-align: justify;line-height: 26px\"><p><strong>Takeaway:<\/strong><br \/>\u201cQuid-pro-quo\u201d recognitions probably aren\u2019t happening very much, &#8211; maybe not even at all. Instead, we have judges recognizing each other organically for hard work and motivating one another to even greater heights. As we receive more data, we plan to look into larger clusters to learn more about what drives these organic reciprocal recognitions.<\/p><\/blockquote>\n<hr>\n<p style=\"text-align: justify;line-height: 26px\">I\u2019ve also got some cool data points to share that I think are interesting and fun! Here goes:<\/p>\n<ul>\n<li style=\"line-height: 26px;text-align: justify\">In the Spain &amp; Portugal Region, <strong>94%<\/strong> of eligible judges in that region wrote at least one recognition! Other all-star regions by this metric are Brazil (86%), Europe-Mid East (84%), Japan (78%), China (78%), and USA-Southeast (75%).<\/li>\n<li style=\"line-height: 26px;text-align: justify\">L2s in Spain &amp; Portugal, Brazil, and Japan used 70% or more of their L1-specific slots, which is a great sign for community growth.<\/li>\n<li style=\"line-height: 26px;text-align: justify\">L3s in USA-Midwest, Spain &amp; Portugal, Japan, Latin America &#8211; Spanish, and China used 70% or more of their L1-specific slots &#8211; again, this is a strong indicator of L3s supporting L1s.<\/li>\n<\/ul>\n<p style=\"text-align: justify;line-height: 26px\">There are a lot of people working hard in the Judge program, and I am glad to see some of that hard work being recognized through the Exemplar Program. I want to leave you with one last thought: For Exemplar Wave 3, try to expand your sphere of influence. If most of the Wave 2 recognitions you wrote came from large events like GPs, SCG Opens, or similar, try to look out for exemplary behavior on a local level as well. Maybe there are judges doing public-facing work online that you could recognize, or perhaps you know someone who just did some enormously difficult work on a judge project that has the potential to really help the program. Keep your eyes open for our next Exemplars!<\/p>\n<span class='judge-tooltip'><a href='https:\/\/apps.magicjudges.org\/judges\/dci\/84750930' >Eric Levine<\/a><span class='avatar'><img width='200' height='200' src='https:\/\/apps.magicjudges.org\/dci\/avatar?dci=84750930&size=200'><\/span><\/span>\n<div style=\"margin-bottom: 20px;text-indent:-9999px\">a<\/div>\n","protected":false},"excerpt":{"rendered":"<p>A message from : &#8220;Me again! Lots of people have been asking us for statistics on wave 1. I&#8217;m pleased to be able to present the results of some analysis of Wave 2 performed by a whole team of judges. Before I hand you over to , though, a couple of reminders: -If you were [&hellip;]<\/p>\n","protected":false},"author":68,"featured_media":405,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[3,7],"tags":[],"language":[9],"class_list":["post-349","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-announcements","category-article","language-en"],"_links":{"self":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts\/349","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/users\/68"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/comments?post=349"}],"version-history":[{"count":23,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts\/349\/revisions"}],"predecessor-version":[{"id":448,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/posts\/349\/revisions\/448"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/media\/405"}],"wp:attachment":[{"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/media?parent=349"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/categories?post=349"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/tags?post=349"},{"taxonomy":"language","embeddable":true,"href":"https:\/\/blogs.magicjudges.org\/exemplar\/wp-json\/wp\/v2\/language?post=349"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}