Magic the Gathering was released on the summer of 1993, soon to be followed by tournaments, and the judges needed to run them. While no one really expects a person to carry the mantle of a “Judge” for 25 years, just how long are judges expected to last on the role?

My purpose in this article is to raise awareness of a possible problem within the judge program, and take the first steps in trying to solve it.

JudgeApps has been online since August 2012, and currently (March 2018) lists nearly 6,500 active judges (Levels 1, 2 and 3) compared to an overwhelming 11,800 uncertified ones. While it’s true many of those people were never certified to begin with, such a large number over a period of less than 6 years – that’s over 150 people per month – should for the very least raise some eyebrows.

A level 3 judge once told me that “It is perfectly normal and natural for people to lose interest over time, and we have to be mindful of how people come in and out of judging.”
I was left pondering – How many judges lose interest? How long does it take them to lose interest? Why do they lose interest anyhow? And is it really that, or are there other reasons which cause them to quit?

Let’s delve into the numbers a little deeper, and for a more accurate evaluation use the tool of “Judge Anniversaries“.
For those of you who don’t know, Judge Anniversaries is a monthly publication intended to commemorate judge’s “landmarks” in the program – the 5 years, 10 years, and 15 years marks. The data being used by them is the first certification date of all judges, so all the judges who certified that month 5/10/15 years ago are listed there. However, not all of them are still certified.

I’ve checked the 5 years anniversaries over the last 3 months:
For the January 2018 Anniversaries, 73 out of 117 (62%, with 2 profiles who couldn’t be verified) judges were uncertified.
For the February 2018 Anniversaries, 51 out of 82 (62%) judges were uncertified.
And lastly, for the March 2018 Anniversaries, 73 out of 123 (Slightly better, at 59%) judges were uncertified.

The percentages look quite constant, so let me put this into words for you:
5 years into certification, only 40 out of 100 people are still active as judges.
Of course some quit a lot earlier than 5 years, and some will inevitably quit after that. Those are rather staggering numbers, especially considering all those people once took the time and effort to become certified.

I usually like to dive into the numbers even deeper, but unfortunately finding the exact “certification decay rate” would be impossible:
For starters, there is no global database keeping track of all the judge level changes and when they became uncertified. Secondly, even if there was one, the maintenance of changing inactive judges to “uncertified” status is something that only recently picked up. For a long time “Judge Center” was used to keep track of judge activity and levels, with “JudgeApps” being secondary, thus listing many inactive judges as still certified.

Only upside to this is that current judge statuses should be rather accurate, after a start of the year purging for many of the people who didn’t meet their level maintenance requirements for 2017.
So… instead of seeing how all the judges “decay” over time (that is what the percentages are after people have been certified for 2 years, 3 years, and so on) I’ll try and do the next best thing: I’ll cross reference a current JudgeApps 2018 master list against the data used by Judge Anniversaries of when people became certified.
That way I’ll be able to tell that as of 2018, from the judges certified on 2016 – 2 years ago – X% became uncertified, from the judges certified on 2015 – 3 years ago – Y% became uncertified, and so on. Assuming people behave the same way each year, this should allow me to “reverse engineer” an approximation of the actual certification decay rate.

One more thing about the numbers: The database being used by Judge Anniversaries had to have been cleansed and managed, as it included missing DCI numbers, multiple mentions of when someone got his or her level 1 (most likely because of failing a level 2 exam, thus re-entered as L1), and such. Moreover, the database only goes back so far.
As a result, I expect for someone else who tries to replicate this calculation to get slightly different numbers.
So without further ado, here are the results:

YEAR	Number of new judges certified that year	Number of uncertified judges from among them, as of 2018	RATIO
2006	104	59	57%
2007	130	70	54%
2008	132	79	60%
2009	208	116	56%
2010	331	203	61%
2011	713	430	60%
2012	1,133	663	59%
2013	1,302	711	55%
2014	1,712	820	48%
2015	1,719	671	39%
2016	1,286	331	26%

And here is the certification decay rate in a visual graph form:

* The 1 year 5% is an estimate based on partial information from 2017. I suspect actual percentage to be lower.
** The little bump on the 9th year is because we’re dealing with people from 2009, and not the actual decay rate. It was a different time, with different kind of people who joined the program.

If these results are true (and we know that for the very least they’re true for the class of 2016) then they are quite surprising.
A decay rate of 25% as early as the second year is much more than one would expect, and it could be an indication of certifying the wrong kind of people – that is the certifier missing something or somethings that should have been picked up on during the interview.
As a side note I could also argue that this is in part because Level 2 judges who want to advance are required to certify new judges, but that’s a discussion for a different time.

The judge program is very inclusive: L2s and L3s are always on the lookout for potential candidates, and a lot of people show interest in becoming judges. Once you’re a candidate basically judge two events to demonstrate some soft skills, chat a bit with the interviewer, show that you know the rules to an Okay degree, and WHAM! We’ve got ourselves a new level 1!
So while the judge program seems to be doing great with “Acquisition”, I would argue – in accordance with the above findings – that it lacks in “Retention”.

To better help the people who steer the program, I’ve conducted a survey among uncertified judges. My hope is that by knowing the reasons that cause so many people to quit from being active judges, the program coordinators and sphere leaders could come up with ways to combat it, or at least delay it, thus keeping the program with experienced judges for longer periods.

For this survey, 4 uncertified people were selected from each region, from among all the uncertified profiles on JudgeApps. Those names were checked with an external source in order to make sure they were indeed certified as L1s at some point.
Each participant was asked several questions, including the open ended question of “Why were you uncertified?” Some replied with multiple reasons.
Out of the selected 108 people, 46 replied and provided 76 distinct reasons.
Those 46 people were certified for an average duration of 3 years, with 10 of them becoming uncertified after only 1-1.5 years. There was no one dominant reason for quitting after such a short duration.
41 people only went as far as level 1, though some were L1 back when those could judge competitive events. The rest made it to level 2.
27 still play MTG to some degree, and another 5 only MTGO/Arena. The rest quit the game altogether.

After going over the different answers I’ve divided the reasons into different categories.
Here is the full sorted list of reasons. Some could be placed under multiple categories, but where divided according to what I believe is the root cause (for example in “I moved for a new job, there for I quickly didn’t have time to stay a judge” the root cause isn’t moving to a new location, or the financial aspect of work, but rather the lack of time caused by the new job).
Though many of these reasons won’t surprise you, their amount might. Results are as follow:

Lack of resources

(28/76) There were two major resources mentioned here:

1. Time (25/76) – This one is self explanatory: people thought judging would easily fit into their schedule, only to find out they were having a hard time keeping up with it, usually due to a big life event such as changing work or family status.
This is by far the most common reason causing judges to quit their judging, making up nearly a third of all the reasons.
[cets_callout_box style=’gray’ align=’full’ title=”]
“I was only a student then an intern when I joined the judge program. When I started to work full time I had less spare time and started to have trouble to split this time between playing, judging and having a girlfriend.”
[/cets_callout_box]

2. Finance (4/76) – While “time is money”, sometimes people lacked the later more, and found it not worthwhile to continue acting as a judge.
[cets_callout_box style=’gray’ align=’full’ title=”]
“I was very poor being a student without a job. I couldn’t afford traveling or going to bigger events.”
[/cets_callout_box]

Losing interest

(17/76) While many people can’t quite place their finger on what exactly made them lose interest in judging (usually it wouldn’t be just one thing), some people provided interesting insights. The following topics seem to fit as sub-categories under losing interest:

1. General change of heart (8/76).
[cets_callout_box style=’gray’ align=’full’ title=”]
“Magic itself was no longer a huge part of my life, as I was finding more enjoyment in other games.”
[/cets_callout_box]
Interesting thing to note here is that more than one person lost interest after only one year. You would expect something like this to take longer.

2. Feeling unappreciated (2/76).
[cets_callout_box style=’gray’ align=’full’ title=”]
“I worked at a games store where I judged events 2-3 times weekly. I worked there for 4 years and it ended with me quitting because I didn’t get the respect for the work I did.”
[/cets_callout_box]
[cets_callout_box style=’gray’ align=’full’ title=”]
“The lack of recognition for my efforts made me feel like my energy was better spent elsewhere.”
[/cets_callout_box]

3. Connections and Community (5/76) – You don’t judge in a vacuum, rather you meet many of the same people over and over again in each tournament, both players and colleagues. A change in that environment can have an effect on you, since it’s also the people who keep you engaged in acting as a judge.
[cets_callout_box style=’gray’ align=’full’ title=”]
“[…] mentality of players changes. So I have decided to stop.”
[/cets_callout_box]
[cets_callout_box style=’gray’ align=’full’ title=”]
“[Person] moved away from [place], and he was a HUGE factor in my becoming a judge and staying enthusiastic about it.”
[/cets_callout_box]

4. Burn out (2/76).
[cets_callout_box style=’gray’ align=’full’ title=”]
“It was a combination of I didn’t want to work for the store in principle and I was emotionally and physically burned out and no longer wanted to judge.”
[/cets_callout_box]
[cets_callout_box style=’gray’ align=’full’ title=”]
“Perhaps needing a break from the game.”
[/cets_callout_box]

Lack of opportunities to judge

[cets_callout_box style=’gray’ align=’full’ title=”]
“I obviously didn’t have a lot of opportunities to judge so judging just sort of faded away from me.”
[/cets_callout_box]
(14/76) This one can also be divided into several sub-categories:

1. Scarce options (3/76) – Some judges get certified in areas with very few stores who need their help. They might not be needed from the start, or it’s enough for something to happen at the one local store, and then they’re left with nowhere to judge.
[cets_callout_box style=’gray’ align=’full’ title=”]
“We got only 2 card stores in my state, so I couldn’t judge anywhere thanks to my previous boss.”
[/cets_callout_box]

2. Moving (6/76) – Some judges were needed in their old area, but moved to a new location were their judging skills weren’t as needed.
[cets_callout_box style=’gray’ align=’full’ title=”]
“Graduated, moved to a new state where I didn’t know the judges and couldn’t find a location that needed a judge.”
[/cets_callout_box]

3. Too many judges (5/76) – Sometimes an area has a large MTG community which seems like it could use more judges, but things don’t always work out that way. Judges find themselves competing for positions, or a store is content with just its regular judge/s.
[cets_callout_box style=’gray’ align=’full’ title=”]
“A lot of players were becoming judges. I believe there were like 15 judges in our local store when I quit. Only 4 of them were like serious judges, but still, my help wasn’t needed at all.”
[/cets_callout_box]
This doesn’t just mean that new judges have a hard time finding places to judge. Sometimes the new judges actually come at the expense of the older ones.
[cets_callout_box style=’gray’ align=’full’ title=”]
“I saw all the newer judges receiving more opportunities and felt more and more left out.”
[/cets_callout_box]

Prefer to play

(8/76) A lot of judges enjoy the game too much, and didn’t like it that having to judge comes at the expense of playing. Not surprisingly, everyone who quit because of this reason is still playing MTG.
[cets_callout_box style=’gray’ align=’full’ title=”]
“I didn’t want to recertify because I like playing much more than judging at the moment. I like some things (discussing rules or how to resolve interesting situations) but overall judging is a bit not my thing.”
[/cets_callout_box]
[cets_callout_box style=’gray’ align=’full’ title=”]
“I considered judging at GPs, however I am also a very competitive player and if I’m going to spend the time and money to travel to a GP, I plan on playing in the main event.”
[/cets_callout_box]

Discontent with the program

(3/76) A few people quit being judges on rather bad terms:
[cets_callout_box style=’gray’ align=’full’ title=”]
“I never felt welcome in the program, I never felt I had fair chances.”
[/cets_callout_box]
[cets_callout_box style=’gray’ align=’full’ title=”]
“I stopped judging events. This was due to internal politics and policy change with the DCI. I wanted to focus on community building and the DCI focused its rewards programs away from that work and onto judging major regional events.
Additionally, I saw too much corruption in higher level judges.”
[/cets_callout_box]

Self doubt

(2/76) One person felt (among other things) that he wasn’t handling the job well enough, and instead of investing on improving himself, decided to take the way out:
[cets_callout_box style=’gray’ align=’full’ title=”]
“I also felt like I wasn’t doing an adequate job at higher level events where it mattered, which was a blow to my confidence”.
[/cets_callout_box]
Another person made a more conscious decision of quitting, since he felt it was hard for him to remain updated:
[cets_callout_box style=’gray’ align=’full’ title=”]
“It was difficult to keep up with new cards and rule changes, so I felt I would not be an effective judge. I declined a couple opportunities to judge events and haven’t had an interest in returning since.”
[/cets_callout_box]

Suspension

(1/76) One former judge was willing to talk about being suspended from the program:
[cets_callout_box style=’gray’ align=’full’ title=”]
I had a disagreement with this person who insulted [me and] my partner [on Facebook]. The situation was only with insults and things like that but without physical aggression. As a result of this on [date] they take me off my conditions of level 1 judge and a few hours later bring me it back arguing that ‘they take the decision without think it very well’, due to this, they began to investigate and until today the judge with the higher degree didn’t communicate with me to know about the situation or what had happened (and he also didn’t go to the shop to ask what happened). This judge is a good friend of the person who insult[ed] me and my partner and I want to emphasize that he never come to ask what happened. The report was sent without ask[ing] me almost nothing, denying me the possibility of defending myself, and they sentenced my with a life banning from magic the gathering.
[/cets_callout_box]

Misunderstandings

(2/76) Lastly, two people who should have remained certified became uncertified because of unclear issues:
[cets_callout_box style=’gray’ align=’full’ title=”]
“I lost my certification because I did not judge any events. At the time I was playing a little bit and thought that to keep my level I needed to judge a GPT. In my area at that time a single judge had a monopoly on all the GPTs so I never judged one.”
[/cets_callout_box]
[cets_callout_box style=’gray’ align=’full’ title=”]
“The TO never reported due to WER errors that resulted in me decertifying.”
[/cets_callout_box]
Discussion

These are my own thoughts and interpretations on the above findings and what can be done about them. Feel free to disagree, but regardless, this is something for others to start from and to expand on in the future.

The number one reason why judges become uncertified is a lack of time. While maintenance requirements for level 1 only include a sanctioned event once every 6 months (lately also a yearly exam), keeping up to date with most recent sets and rules also takes a lot of time in preparation. In addition, if someone is currently too busy in life, he or she probably doesn’t just want to do the bare minimum to remain certified. Lastly, this is not just a matter of lacking time, but of priorities – there are those who manage to find the time to judge despite having demanding studies, jobs and families, and some of them even make it to level 3.

When it comes to losing interest, sometimes people will just find out that acting as a judge isn’t what they had in mind. This should show the importance of judging two sanctioned events before certification, and of the interview portion. You probably shouldn’t expect judging to “grow on you” – If you didn’t like it the first two times, it won’t get any better. In addition, the people conducting the interview should have this in mind, and not just think about certifying another new judge.
After certification – recognition, community and connections could have some weight in keeping your interest.

A lack of opportunities to judge is also something that should be noticed beforehand – Is there a real need for another judge in that area? Will that judge have somewhere to judge at? If there are other active judges in the area, how are they planning to split the load among themselves? All of these points should be taken into account.
Also, is the candidate facing a major life change in the near future? If so, maybe it will be better to wait with the certification afterwards, just so we’ll see it’s still relevant.

When it comes to the preference for playing over judging, this is an interesting one: The MTR allows “Tournament officials” to play in non-Premier Magic Tournaments, and while some of the people in this survey quit before this change was introduced, many others became uncertified only after it. Perhaps it doesn’t work as planned?
Anyway, this is another something that can be brought up during the interview, seeing how the candidate feels about the events he or she judges, as opposed to plays.

Suspension is a more prevalent cause than one might expect, and the reason it didn’t show up more here could be because those who were suspended might have been too embarrassed to reply and mention it. I know of other judges who are or have been under investigation, and even a judge I personally certified ended up suspended due to theft. The Judge Conduct Committee exists now to deal with such issues, but better care could be given as to who people certify.
As far as retention, this isn’t an issue here – If the suspension is warranted (either by WoTC or the JCC) then these aren’t the kind of people the program wants to retain. All we can do is try to make sure no one is wrongfully suspended.

I don’t want to get into discontent, self doubt and misunderstanding too deep, but in general – talk. Talk to your colleagues, mentors, RC, forums on JudgeApps… Communication can help sort these types of things out.

Two final clarifications:
1. None of these reasons should be an “auto decline” for someone who wants to be certified. Just because you’re about to move, or maybe get married, doesn’t mean you’ll quit the judge program afterwards. Instead those are just more things to think about and discuss during the interview.
2. It’s important to note that this list of reasons is not a comprehensive one. For example, I know of a judge who left the program due to health issues, which could fit as a new category. There will always be more reasons for lapsing.

Conclusion

Many of us can relate to at least some of the reasons that were mentioned as causing people to become uncertified. There’s a quote – “The same boiling water that softens the potato hardens the egg. It’s about what you’re made of, not the circumstances.”
So while a lot of people quit judging because of these reasons, there are also those who face the same dilemmas and still continue judging, despite it all.
Having said that, with so many people sharing the same circumstances which do cause them to quit, surely there are things that can be done about them as well.

I’d like to thank the following judges for their help at getting the data and providing feedback on the article:
Georgi Benev, George Gavrilita, Rob McKenzie and Giorgos Trichopoulos.