I guess it’s that time of year. I’ve seen at least three “New Year’s Resolutions” posts having to do with advancing to Level 3. I don’t have a particular problem with this as a goal. Goals are good. The problem I see is that too many judges have very vague goals pertaining to L3. The one that sticks out in my mind is Andrew Rula’s goal to “Make significant inroads towards L3.” This isn’t meant to pick on Andrew because the other posts used similar terminology, and I’ve heard this type of language even more in face-to-face talks with various judges: “I want to make progress to L3…”

What does that mean? To me, it sounds like a hedge. The judge wants to be L3, but doesn’t feel that they can complete that journey in the given time frame, the calendar year for the purpose of Resolutions. Thus, 2014 becomes the “progress” year and 2015 is potentially set up as the payoff year. But what does progress look like and how do you measure it?

There is a pretty well defined L3 Advancement Process now and it has various metrics and “checklist” items that one can aspire to, but one thing that isn’t measured is receiving feedback from others. I suppose technically the 3 recommendations fit this bill, but those are often seen as the final gatekeepers of the Advancement Process. What should one “on the road to L3” do?

I’ve been compiling statistics on reviews to try to get a better understanding of what matters. Much of the research is focused on L3s, specifically what it takes to get to L3. The first statistic I want to throw out there is:

1020 days

That’s the average time it takes a judge to go from L2 to L3 (numbers compiled from 63 judges). The median number is 855 days. This gap in these two numbers (and the standard deviation of 550 days) stems from some of the longtime L2s who finally made the jump recently. At the very least, the average is over 2 years, which suggests that if you just became L2, a year to consolidate your general skills and gain experience, then a year to really focus on trying to become an L3 are not just recommended, but a reality of the complexity of the process. This is why the the L3 Advancement Process states that you must be an L2 for 1 year as a prerequisite.

The next number I have is:

34 reviews

That’s the average number of reviews that new L3s had written of them on their advancement date. The mean for my 63 judge sample was 33, so very close. Standard deviation was 13. Based on that, from a statistical standpoint, you should have at least 21 reviews written of you (1 standard deviation from the average). The final number is:

11

That’s the average number of reviews written per year for judges from their L2 advancement date to their L3 advancement date, the “1020 days.” That’s less than 1 review per month. That doesn’t seem like much, but take a look at your history; have you been reviewed 11 times in the past year? And perhaps more importantly, why is this so important compared to reviews written by an L3 candidate?

Well, I have numbers on that too, but the spread from outliers makes the information unreliable. Take, my good friend David de la Iglesia. He wrote a whopping 101 reviews as of his L3 advancement date, an average of 35 per year from L2 to L3 (it took him 742 days). This stunning number speaks more to David’s devotion to the art of feedback, as do other outliers.

Meanwhile, the number of reviews written about someone is not directly under one’s personal control. It has a lot more to do with overall event attendance and garnering the attention of others. If you attend a certain number of events, you are bound to catch someone’s eye–“I like that judge. He or she has a lot of potential. I’m going to write a review.” It’s my belief that the people who become L3 tend to get more of this attention for several reasons:

1) It’s easy. These judges are the cream of the crop, often even before they become L3. They have obvious strengths that make for an easy opening to a written review.

2) People want to help. If the judge has weaknesses, other people want to point them out and help the judge overcome them so that they can hurry up and get to L3.

3) It’s easy (part two). Even when there are weaknesses to be addressed, the judge’s easy-going personality or openness to feedback makes it much easier for other people to write reviews of them.

Taken as a whole, a judge who gets reviewed a lot is someone who the community is collectively deciding to expend time and energy on developing. That sounds like an L3 to me. Of course, the correlation isn’t 100%. There are exceptional judges who make it through the process with very few reviews written of them. Sometimes this has to do with geographic isolation and a lack of big events. It may also be that a judge is “fully formed” and people don’t feel like they need a lot of feedback in order to make the leap to L3.

On the other side of the coin, there are certainly examples of L2s who have tons of reviews written of them and never make it to L3. Statistics aren’t perfect. They tell an aggregate picture, but on the micro level, the individual matters the most. Maybe the L2 just never decides to go through the l3 Advancement Process. Maybe they have a hang up that they can’t or are unwilling to fix (like rules knowledge).

This blog isn’t meant as the magical potion. It’s a brief analysis of the statistical profile of L3s and what I think it means for you if you want to join those ranks. My next few blogs will focus a little on how you can take matters into your own hands in terms of getting reviewed more.