Hello all,

We have completed an arbitrary cycle around the sun, and that seems to be a perfectly arbitrary time to do a retrospective. When I made a post on Facebook asking if there were any topics people wanted to cover, the three most frequently requested topics were:
1) Do I think Exemplar is meeting its goals?
2) How do I respond to criticisms on Exemplar?
3) MOAR METRICS PLZ

I’m going to tackle the first two in this supersized blog post. I will touch on metrics some, but I think that is best reserved for a separate article.

Do I think Exemplar is meeting it goals?

At Exemplar’s creation, the goal was declared in Exemplar’s initial blog post as:

The goal of the Exemplar Program is to recognize the contributions that judges make to the Magic Community. Its aim is to grow the quality of judges through peer recognition and provide positive examples for others to follow. By providing examples of judges who exhibit good behavior and provide a positive impact on the Judge Community, the belief is all judges will have role-models and clear examples of conduct which will improve the quality of judging globally.

So, does Exemplar recognize contributions that judges make? Absolutely. In 15 months, we have recognized nearly five thousand judges for their contributions to the program.

Previous recognition efforts were focused on RCs or L4+ Judges recognizing contributions. Exemplar, through the peer to peer approach, reaches a much wider audience than RCs and L4s ever could. The peer to peer approach allows us to get personal with nominations, and you can recognize what you value. A personal example of something I value is the ability to recognize a problem, figure out how to address it, and then actively pursue a resolution. I want to hold up positive examples of that like Link wants to hold up pieces of the Tri-force.

There are a lot of judges out there who enjoy reading the great things their fellow judges do, and use them as inspirations. The examples and the role-models are out there. Each Exemplar wave contains thousands of examples of behavior that we want to see more of. There are multiple regional blogs and projects that highlight the awesome things judges do in those regions, and once a wave, the Exemplar team makes a blog post highlighting nominations we like. We can do more. For 2017, the Exemplar Sphere has formed a separate project, led by Nicolette Apraez, designed to highlight exceptional judges and exceptional actions as well as cross-coordinate with regional efforts.

Has Exemplar improved the quality of judges? That’s more difficult to quantify. We don’t have a mechanism to measure that, and isolate the impact Exemplar has in a vacuum. I have anecdotal evidence that being nominated for a specific behavior provides positive reinforcement of that behavior. I’ve also seen many people post on social media that they made a specific positive change and the received a nomination as a result. I also know that personally, I’m more likely to continue a behavior I was nominated for. So, does it improve the quality of Judges? I think it does.

Random Questions

How has leading Exemplar changed my perception of the Judge program?

We are going to start with my favorite question I was asked, because it wasn’t one that I had previously thought about. About six months ago, I was reviewing the judges that repeatedly received a lot of nominations and I realized something.
I used to think that success in the Judge program was based on being willing to do things. If someone asks you to do something, you do it. If a HJ opportunity comes up; yeah, that’s something you’re willing to do. You are asked about joining a project, yeah, you’re willing to help.

I now think success in the Judge program is based on being eager rather than just willing. Eager judges have the drive and the passion to go beyond the task at hand and beyond the walls of their store. Being eager sustains you when things get rough and your project slows down. You strive to improve; you don’t let setbacks actually set you back and you don’t confuse repetition with growth. You grab others and try to transfer some of your excitement into them. When you plan your event, you don’t just go through the motions, you try and make it exciting for the players and you make sure the judges get something out of it.

Regardless of your level, you can only go so far by being willing. True sustainable success comes from being eager.

Also, I may have been listening to a particular song when writing this answer.

How Many Nominations do you get per wave; how many could there be?

It differs based on the number of slots, but this last wave 3385 nominations were written. On average, 55-60% of eligible judges make at least one nomination.

All those nominations going to waste! What are you doing to increase those numbers?

Nothing.

That may sound surprising, but hear me out. Nominations aren’t actually going to waste. Judges are given more nominations than they are expected to use. The slot allocations are based on several factors, with some padding built in for judges that are exceptionally active in the judge community and have more opportunities to observe behaviors worth recognizing. The average participating judge is using nearly their full expected amount for their respective level. It is not a failure when you do not use all your nominations, nor is it something to be ashamed of.

Also, one of this past year’s goals was to increase the quality of nominations. Looking at Waves Four and Five, the quality of nominations seems to go down as the quantity of nominations go up. We wanted to focus on making the nominations we already were receiving better before looking at the nominations we weren’t receiving.

How many nominations get deferred/rejected?

We are currently fluctuating between 4-8% depending on the wave. About 25% of those nominations are deferred because judges don’t reply to emails requesting more information. Nominations flagged for a “second look” though, fluctuates wildly. It’s been as little as 4% in Wave 2 and peaked in Wave 5 at 19%, and is currently around 11%.

Why the peak in Wave 5? I think a few things converged. Between Wave 4 and Wave 5 we had finally documented internally what makes an acceptable/unacceptable nomination. The act of writing down a standard makes you more likely to enforce it. Wave 5 was also still relatively early in Exemplars life and it was the second “large” wave. Public guidelines on nominations were rather vague and judges were “stretching” to use their slots. Some judges had learned that if their nomination didn’t pass the guidelines, they would be contacted and they could fix it, so there wasn’t really any cost to submitting a poor quality nomination. Wave 5 was also the wave that we introduced “deferrals”. Coincidence?

What do I think was the most successful change to Exemplar over the last year?

There were several changes that had a huge impact. However, I’m going to say that the most successful change has been that we have become predictable. 15 months ago, Exemplar was very unpredictable. When waves started, when waves were released, when foils were distributed, what was an acceptable nomination. It was all uncertain. Uncertainty makes people anxious.
Since then, there have been several blogs laying out expectations, so you know what we expect from you. We adhere to a predictable schedule, so you know what to expect from us.

What do I think was the biggest failure of Exemplar over the last year?

In my opinion, the biggest failure of Exemplar this year was the lack of attention to non-English languages. While that’s mostly fixed in the nomination review team now, the fixes were reactive instead of proactive. Blog posts tend to be in English (uh…like this one), and a subset of auto generated emails tend to be in English. Going forward, my hope is to bring in the translation team to help with the internationalization of blog posts.

How does the Exemplar Team handle nomination abuse?

There are ‘no action’, formal warnings, suspensions (from Exemplar). Just like in a tournament, multiple warnings can be upgraded.  Also like the PIC, the criteria we use for decisions is not published publicly. If a case of abuse is brought forward, or we discover it internally, we do an investigation. Conclusions and actions are reached in agreement with the affected judges RC. I do want to encourage anyone that is aware of abuse send an email to exemplarprogram@gmail.com and we will look into it.

How does Exemplar handle Judges suspended by the JCC?

I don’t remember if I ever published the previous Suspension policy. If I didn’t, you didn’t miss much. It was awful.
Suspensions are unique, and it seems anything I wrote down to address one wasn’t applicable to the next situation. Suspensions are also rare. So spending time documenting a comprehensive policy for so few cases isn’t worth it.
So, the current policy has been changed to this: If your suspension completely overlaps a Wave, you can not receive or give nominations. If your suspension (or investigation) partially overlaps a wave, the exact course of action will be handled by an agreement between the Exemplar Lead, the JCC Lead, and the judge’s RC on a case by case basis.

What are some big complaints about the program?

Oh boy, buckle in. Most of the items below come from emails or FB posts, or even concerns I had before I joined the Exemplar team.

Isn’t the Exemplar program is just a popularity contest?

Not quite, but I can understand how someone would reach this conclusion. In fact, when Exemplar was first announced, I thought it was going to be a popularity contest where GP Judges just nominated each other.

I was wrong.

The difference is what makes a person popular. In high school, it might be being on the football team, or having rich parents, or a car, or the older brother that will buy you beer. In the Judge program, none of that helps you become popular. You don’t become popular in the judge program because your parents are rich. You become popular by helping others, establishing yourself as someone dependable, and performing with excellence and being a leader. The same traits that result a person to becoming popular in the Judge program are also traits that result in Exemplar nominations. Popular judges are popular because they are involved.

The system is flawed and should be abandoned.

I typically see this on Facebook or Reddit and it confounds me. Im not confused by the claim that Exemplar isn’t perfect; it’s not perfect. The part that confuses me is the belief that only a perfect solution should be allowed to exist. There are over 7000 judges listed in Judge Center. You are not going to find a solution that everyone will agree is ‘flaw free’. However, this point almost always comes right before the next complaint.

Why don’t you send out foils based on the number of events worked instead of this subjective criteria?

This is probably the most common complaint I get. Before we start getting into the answer, I want to address the fact that questions like this ignore all the goals of the Exemplar program, and try to turn it into a business transaction. At it’s core, that’s the source of the disconnect, and it makes it hard to come to an agreement.

There is some pretty seductive logic in play here though. If foils are sent based on events worked, then it’s easier to get foils because you are already doing what is required to get them. You feel in control.

However, all of that strips away the point of Exemplar. Exemplar is about recognizing excellence, not recognizing attendance. They aren’t the same thing. There is already a system in place for recognizing attendance.

We want to recognize mentoring, innovations in how events are done, overcoming obstacles and creating new tools and hundreds of other things. In short, we want to recognize judges that make a strong positive impact on the program, the judges in it and the community around them. And there is no spreadsheet or matrix that can objectively calculate that.

I live in a remote area and it’s hard to get noticed for Exemplar.

This is the complaint that I have the most empathy for, and out of all complaints, it’s the most legitimate. It is more difficult to be recognized for growing your local community when you are the only judge in it, and judges in more populated areas are more likely to be recognized.

But it’s only an obstacle if you let it be one.

As I said above, we want to encourage and recognize judges that make an impact on the program. You can do that without ever leaving your house. There are forums, Facebook study groups, and projects that you can join or even start! Ask to host a mini-conference. Mentor judges up to Level 1 skill and then find an L2 to test them. You have so many options and opportunities available to you if you leave your store, physically or virtually. Those are the types of things we want to encourage. We want you to interact with the larger judge program.  If you have an idea, or you tried something, send an email to your RC asking if something similar has been done elsewhere.

If you aren’t interested in doing those things, that’s ok, but you are removing possibilities

Exemplar Creates “Feel Bads” for people that don’t get nominations.

This is another tricky one. It’s dealing with emotions, and there is a delicate dance around acknowledging the emotion is real, while looking objectively at the validity of the reasons the emotion is there in the first place. I can’t tell you not to feel bad. I can help address the reason though. Consider reaching out to someone you trust to be honest with you; your mentor, your RC, a member of the Exemplar team, me. We can talk through your feelings if you are willing.
It’s really too personal and too nuanced a topic to handle in a generic blog post.

What I can say in a generic blog post is that disappointment isn’t necessarily a bad thing. You can learn a lot from what you are disappointed about. It shows you care about something to feel bad when it doesn’t happen, and where there is passion, there can be change for the positive.

Nominations take too long to get out.

They sure do. Currently it takes about six weeks to make a first pass at the nominations, and then go through a second time, and send emails out to the judges for corrections, get those corrections and manually enter them into the system. It’s a relatively grueling process and I am constantly in awe at the Exemplar team’s dedication to the project. No other project on the program puts out as much work on as tight a timeline, on such a frequent basis. (I base that assertion on purely anecdotal evidence while ignoring any facts that might prove me wrong.)

Right now, I could probably trim about three weeks off the timeline by not contacting judges with nominations that are ‘on the line’ and instead just deferring them. We may eventually go that route. However, at this time, I’d rather spend those three weeks sending emails and providing some minor course corrections to nominations and education. I think of it as an investment in the quality of future nominations.

I do want to address the common feedback of “if you start reviewing nominations earlier, you can release them earlier”. This would be a great idea if nominations were entered uniformly across the Nomination window. Unfortunately 60% of all nominations come in the last week, with about half of all nominations being made in the 3 days right before the window ends.

Speaking of taking too long to get out,, I hoped you enjoyed this mega-sized blog post. If you have any questions or concerns, please email exemplarprogram@gmail.com. Join us next time when we talk MOAR METRICS!