Clarification: Expectations for Filling Out Evaluations

Officiating (Public Read-Only)
1

The WFTDA Officiating Pillar, and Certification Oversight specifically, would like to clarify our expectations for filling out evaluations. It is a core expectation for higher-level head officials to fill out evaluations for their crews for WFTDA Sanctioned Games and Tournaments, within three months of the conclusion of the game or tournament.

This expectation applies to:

  • Higher-level officials (level 2 or level 3)
  • Who focus on the “head official” role
  • Who are acting as a Head Official, Crew Head Official, or Tournament Head Official
  • For a WFTDA-Sanctioned game or tournament
  • Who have people on their crew (including the other Crew Head) who have opted in to the WFTDA Officials Certification program.

For Tournament Heads, the expectation only applies to evaluations for crew heads (not for every tournament).

To this end, Certification has begun exploring how to count “number and quality of evals submitted,” by officials who are themselves applying for Level 2 and Level 3, as a component of determining whether they are meeting expectations in the Head Official role. While filling out evals for others has always been an expectation, we will soon be formalizing it by looking at the raw numbers.

This is in line with many of the long-stated expectations of Level 2 and Level 3 officials, as published for many years in our website, What do Certification Levels Mean?. The Head Official role requires:

For both Level 2 and Level 3 certification:

  • Advanced understanding and consistent execution of at least one role across multiple styles of gameplay
    • (because providing feedback is a core part of being a head official)
  • Completes paperwork, hand signals, and verbal cues correctly, to the level necessary for sanctioning, in positions of expertise.
    • (because formalized feedback is a paperwork expectation for head officials)
  • When working in a position of expertise, has a holistic view and understands the interaction between all roles that their position comes into contact with. Able to support another official in a related role.
    • (because formal feedback is a key way that head officials support their crew)

Additional expectations at Level 3:

  • Exceptional ability to communicate with all participants, including conveying detailed information in a concise manner.
  • Can correct others’ mistakes to rectify problems for the game.
    • (this is widely understood to apply to in-game scenarios, but it also applies via feedback to preventing such problems from recurring)
  • Can be trusted to perform in a leadership position planning and staffing a multi-day event and will follow through with all necessary work after the event.

We also want to clarify that any official who can be evaluated should be assumed to have opted in, unless they have explicitly opted out for a specific game or tournament. Head Officials should not require officials to “opt in” a second time, either per-game or per-tournament, to receive an evaluation. There are sometimes barriers to explicitly opt-in to ask for feedback, especially when there is an imbalanced power dynamic. By clarifying that it is a core expectation for experienced head officials to automatically write evaluations, we hope to remove the possibilities of those barriers.

Some caveats to the above:

  • Perfection is not the expectation, as with everything, we are looking for a pattern of providing timely feedback.
  • The expectation is only for Head Officials who are themselves on the Certification path
  • The expectation different per level:
    • Level 1 applications will not be affected, though we may note if a Level 1 applicant frequently serves as Head Official but does not show a pattern of providing timely feedback, as a point to improve prior to applying to Level 2.
    • Level 2 applications already require that an official be excellent in at least one role. If they do not show a routine pattern of providing timely feedback to those who have opted in to the Certification system, they will not be considered excellent at the Head Official role, but they could still attain Level 2 through excellence in another role or roles. Working as Head Official is not itself a requirement.
    • Level 3 requires that an official be exemplary in at least one role and excellent in all roles. Writing evals should be a habit for exemplary Head Officials, and should be routine for applicants who focus in a different role.

We do not believe that this should be a big change to most high-level Certified officials who routinely work in the Head Official role. This has long been a requirement for Playoffs, Cups, Championships, and Recognized Tournaments, and most sanctioned tournaments make a similar promise and/or set a similar expectation.

Who has “opted in”? Anyone on the Evaluation Form itself, but they are also listed here in a spreadsheet.

Any questions, please ask here!

2

If every head official starts filling out an eval for every “opted in official” for every single sanctioned game or tournament…

…won’t that represent a massive increase in the amount of evals that Cert has to wade though for each applicant? Have estimates been made of how that number compares to the current number of submitted evals and how that would affect the workload of the panels if it were actually to occur?

3 Likes
3

From the guidance on writing an evaluation for an official, it says:

performance evaluations should take ten to thirty minutes to write

At a typical game there are probably something like ten non-skating officials (JT, 2xPLT, SBO, 2xSK, PMB, 2xPBT, ALT) including the HNSO. So that’s potentially nine evaluations to write, which if the guidance on how long it should take to write is followed means the HNSO should spend at least 90 minutes but potentially up to 4½ hours writing evaluations. This could be even higher if there’s an extra role such as PW, or if there’s multiple games and some of the NSOs change between games.

I recognise that not every official is opted in, but as more and more do this scenario becomes more and more likely. Is this level of workload actually the intention?

5 Likes
4

Great questions! I’m going to bundle responses together because the new forums don’t like it when people reply multiple times. :slight_smile:

@AdamSmasher:

If every head official

This expectation is for officials who focus on the Head Official role and are aiming for Level 2 or Level 3 themselves. That’s hidden in the post but this is worth saying explicitly.

We also note that perfection is not an expectation, so we do not expect 100%.

massive increase in the amount of evals that Cert has to wade though for each applicant

But as an extreme case where literally everybody always wrote evals for everybody, we would expect to have a lot more evals. If they turn out to be highly redundant or of low quality, we will revise our expectations accordingly.

Have estimates been made of how that number compares to the current number of submitted evals

For most officials who apply for certification, there are about 100 new games to evaluate. Tournaments represent 5 or 6 games plus a bunch of singles so probably 20-40 opportunities for evals. Head officials from those tournaments, who qualify for these expectations, are around 10-15% of games so…we’re looking at two to six additional evaluations per packet.

This would likely reduce the workload of the panels, by making the decisions easier to make and making the summaries easier to write. That would probably be true if we started seeing 10-11 additional evals per packet. After that, the downside is not huge because redundancy is just reading the eval and saying “+1 to what we already noted.” It would probably get to be “too much” if a packet ended up with 30+ evals in it, but we have only seen that twice in history. But as noted, we will revise our advice if we get to this point.

@Twixxi:

At a typical game there are probably something like ten non-skating officials … Is this level of workload actually the intention?

The idea of having to write nine evals for every single game every HNSO works is not the intention, nor is it the advice, nor is it likely in our view.

  • This expectation only applies to HNSOs who are themselves focusing on the HNSO role and targeting L2 or L3
  • For single games most officials on the crew are not in the Certification system at all
  • For single games we think officials will be more likely to opt out
  • For most sanctioned tournaments, this expectation is already in place
  • We do not expect perfection in any case and if someone is doing a lot of evals but misses some we would not necessarily say they missed the mark
  • We think that the “ten to thirty minutes to write” expectation will actually get much shorter as head officials get into the practice of writing evaluations.
5
  • Why is the focus on Sanctioned Games and Tournaments specifically? For many reasons (regions, rankings, b teams, borderless teams, etc), my local derby-sphere has lots of high-level Regulation games, while Sanctioned games are rare and often low-level. E.g. my last Sanctioned game was in August, and only three of the refs had more than 5 games experience. Since then, I’ve officiated dozens of Regulation games, several with gameplay and/or crews I would consider postseason-level.
  • An obstacle I have found is that newer officials seem unfamiliar and uninvested in Cert. E.g. after Skate Wars this year (a tournament explicitly structured around mentorship and growth), I asked my crew who wanted evals. 4 said yes, but none of those 4 had opted in. I explained to them what that meant and how to opt in, but none of them did.
  • I wanted to single out a remark that resonated with me:

There are sometimes barriers to explicitly opt-in to ask for feedback, especially when there is an imbalanced power dynamic.

This is a big thing I’ve seen in my teaching: making something “By Request” tends to reward those who aggressively advocate for themselves and punish those who don’t. Since rewarding entitlement tends to propagate inequality, removing these structures is a simple way to make systems more equitable. I like that.

  • There was a survey making the rounds a year or so ago about Cert. Does Cert have any plans to discuss the results of that survey and how they are incorporating that feedback? I ask because this clarification has an oddly frustrated tone, like the community is letting Cert down by not writing enough evals. Reading between the lines, I can see how the community doing more for Cert helps Cert do more for the community. However, I think this conversation would be more collaborative and less adversarial if we had a sense of what Cert is trying to change or improve, and how normalizing evals-by-default fits into that plan.
7 Likes
6
  • How is it determined which role an official is focusing on? And when you break down expectations by level, you write evals are expected from all L3 appplicants, while in other places you state it’s only those focusing on the head role. Which is it?
  • How do you determine the number of evaluations expected from an applicant? Will you go through all the officials rosters of their sanctioned games to see who was in the system at the time? And even if you do that how would you know if an official opted out of an eval? (You write that you expect that to happen regularly at single games after all.)
  • Your assumptions regarding the expected amount of extra work do not match my experience:
    • Most of the time single sanctioned games are staffed by essentially the same people that officiate in tournaments. There is no significant difference in the number of opted in people per crew
    • The only event I officiated post Covid that had an expectation of evals being written was RCs. All other tournaments, sanctioned or not, were on an opt-in basis.
    • I don’t see how it is possible to write a meaningful eval with significantly less than 10 minutes of time investment. You have to recall what happened in the game, identify points worth mentioning, and write them down. Plus filling in all the metadata. Sure, if I just write “Did their job, nothing special happened” that can be done faster. But is that meaningful?
  • “Completes paperwork […] to the level necessary for sanctioning […] (because formalized feedback is a paperwork expectation for head officials)”
    Citation needed
  • And the implication that evals are the only (or best) way to give feedback in your other reasons why you consider this a clarification and not a change is just plain wrong. If e.g. I want to give someone feedback on their hand signals, doing it in person where I can demonstrate what I want to see is much easier than writing an eval the week after.
  • By introducing this requirement you are creating a barrier for people for whom writing things does take considerable effort.
  • “any official who can be evaluated should be assumed to have opted in,”
    Is there a way to get removed from that list or get yourself marked as generally opting out for the time being? This change has further decreased my already low interest in getting certified and I’d prefer not to generate unnecessary work for others.
7 Likes
7

I fear that this expectation might scare FLINTA Officials applying for leadership positions. Those are more often than not also Skaters, coaches… i.e. already dedicate a lot of time to our sport.

5 Likes
8

This expectation applies to:

  • Higher-level officials (level 2 or level 3)

  • Who focus on the “head official” role

  • Who are acting as a Head Official, Crew Head Official, or Tournament Head Official

  • For a WFTDA-Sanctioned game or tournament

  • Who have people on their crew (including the other Crew Head) who have opted in to the WFTDA Officials Certification program.

  1. Is this meant to be considered five “and” statements? As in, only if all 5 of these apply then the expectation is evals are the default?

  2. Does this mean the T/C/HOing JRDA and MRDA sanctioned games/events will no longer “count” the same as WFTDA games for the purposes of Certification/TOSP/Playoffs qualifications?

3 Likes
9

I have questions:
I know this is WFTDA and WFTDA comes first in this group. But I am affiliated with a MRDA league as an official, and while I do WFTDA derby locally, the higher level sanctioned work I do is mostly MRDA and JRDA. Are evals for MRDA and JRDA going to count? If I CHR a MRDA tournament and write evals, is that labor going to matter for myself or for the person I am writing about, as the event is not WFTDA? Or will this expectation reach officials despite their affiliation and the affiliation of their event(s)?

Is adding additional labor to folks who may not be able to handle more labor- time wise, economically, or mental health wise- really a wise choice? While many of us make time for the games and tournaments we officiate, we may not have additional time or resources to do more. Some are working multiple jobs, some struggle with health and mental health issues that monopolize time, some have children or families that require attention. It seems like gatekeeping to offer higher certification levels only to those who have the resources to write evals. There are a lot of great officials that are going to miss out because they don’t have extra to devote to derby beyond what they already do.

I chose certification to assist with getting picked for tournament and it is important to me, which is why I’m here, even though I affiliate with MRDA. But I feel like the goal should be to certify as many excellent officials as possible and make advancement based on skill set and knowledge, not on having the spoons to write evals for others.

6 Likes
10

Thanks all for your thoughts and comments – I’ll try to reply once a day.

@DrMath

Why is the focus on Sanctioned Games and Tournaments?

Because this is what we (WFTDA Officiating and Cert) have scope over. Other evals for other games matter and are helpful but we don’t feel like we have license to say that we expect anybody to do things outside of games of our purview. (We also wanted to limit the expectation to not stress anybody out more than necessary.)

newer officials seem unfamiliar and uninvested in Cert

To put it bluntly, this is okay too. If they don’t see a need for certification that is good information for us. Someday they might, but, you don’t really need evals to get to Level 1. That’s why we don’t automatically opt anybody in for evals until they reach Level 1. The eval system is for (and from) those who value Certification.

There was a survey making the rounds

Certification Oversight only got access to the responses last month. It was run by the Board of Directors and they only just now asked us to help them make sense of it. Expect a report on this sometime in Q1!

@speedy

How is it determined which role an official is focusing on

When an official applies for, e.g., Level 2, we will look at their performance across roles, and we will look at their “eval rate” as part of our evaluation of their Head Official role. This is one more piece of info we will consider when evaluating how each applicant is performing as a Head Official.

How do you determine the number of evaluations expected

This will be sort of collective, based on the rates we see from other officials with similar game counts as Head Official in WFTDA Sanctioned Games and Tournaments.

Most of the time single sanctioned games are staffed by [the same people]

If you find that one or two people are always the “head official” of games and tournaments, this could be relieved by helping, encouraging, and training others to fill the Head Official role sometimes.

All other tournaments, sanctioned or not, were on an opt-in basis

This is why we feel the need to clarify the expectation, as it has not been consistently understood.

write a meaningful eval with significantly less than 10 minutes of time investment

For a single game for a single official who did their job and did not have anything notable happen, writing a short paragraph about that should not take ten minutes. The metadata can be copy/pasted.

Is an eval that says “nothing went wrong” useful to us? Yes, to an extent. It is not the sort of eval that would make or break someone’s level, but it tells us that there wasn’t a problematic issue that occurred. Officials who are seeking certification and don’t opt out can also tell their Head Official what they are working on and then ask them to note it, or tell their Head Official about their experience in the game in order to get the content they think is most important into their record, if the Head Official is willing or cares.

This is also not a specific or precise thing, but, an official who says “went fine, nothing happened” in eval after eval is probably not paying very close attention to their crew during games. Most games have at least a few interesting things happen.

Citation needed

For which part exactly? “Paperwork is filled out” is listed on the linked sheet; “providing feedback and helping crews grow is part of being a Head Official” is not so we are clarifying it in this post, and “evals count as paperwork” is also clarified in this post.

evals are the only (or best) way to give feedback

We do not claim this. We are clarifying that evals are an expected way to give feedback. Not the only, nor the best.

“I spoke to them about their hand signals and they improved” is an extremely helpful thing to see on an eval. Especially for someone who had been noted to have that issue earlier. The issue is on-record. Then if you say “I spoke to them about the issue, and they listened and applied it,” that shows they can accept feedback and adapt. Then in two weeks someone else just says, “no notes, they did fine,” that confirms that your feedback “stuck” and the previous issue has been resolved.

Is there a way to get removed from that list

Yes, email certification@wftda.com and we will remove anybody who asks from the list. Note, however, that you must remain opted in to feedback for one year prior to applying to Level 2 or Level 3.

@micro

might scare FLINTA Officials applying for leadership positions

Can you help me understand why this group might be more affected by this expectation than others? I answer the general case of “people who already do many different things in roller derby” below but if I need to answer more specifically, please let me know.

also Skaters, coaches… i.e. already dedicate a lot of time to our sport

This is totally true, and this is a big part of why we are clarifying that this is an expectation for higher-level officials focused on the Head Official role. If they have enough time to learn to officiate, and to officiate many games, and to get Level 1 and after that decide they want to go for Level 2 and that they want to do it by focusing on the Head Official role, then feedback and mentorship is already an expectation.

We acknowledge that every person has limited time to dedicate to Roller Derby and if they have multiple interests or multiple roles that they may not be able to dedicate as much time to others. Ultimately we need to make choices about how to spend our time. The idea of “I want to be a head official but I don’t have time to write evals” remains fine for the vast majority of potential head officials who are not working towards a Level 2+ certification level, focusing on the Head Official role.

@wishbonebreaker

  1. Is this meant to be considered five “and” statements? As in, only if all 5 of these apply then the expectation is evals are the default?

Yes, and yes.

Does this mean the T/C/HOing JRDA and MRDA sanctioned games/events will no longer “count” the same as WFTDA games for the purposes of Certification/TOSP/Playoffs qualifications

These games and these evals will continue to count as much as they always have, on par with WFTDA games.

We (WFTDA Officiating and Certification Oversight specifically) do not feel we have the standing or the “right” to tell people working other orgs’ games what “we” expect of them (see also my answer to @DrMath above).

3 Likes
11
  • This expectation only applies to HNSOs who are themselves focusing on the HNSO role and targeting L2 or L3

That is a fundamentally different statement than:

“It is a core expectation for higher-level head officials to fill out evaluations for their crews for WFTDA Sanctioned Games”

Either filling out an eval is an expectation or it isn’t.

Can one be a high level official without being certified? I certainly think so.

Expecting only high level CERTIFIED officials to fill out evals and not expecting uncertified officials to do so implies that the evaluations of uncertified officials don’t matter. It reads as a slap in the face to the abilities of uncertified officials to offer competent evaluation.

The other way to read it, of course, is that “as a price of your certification, you are expected to provide us with evals,” but if that’s the intent, it should be explicit, and not imply that submitting evals is actually part of what makes someone a good official. Rather, this means that submitting evals makes you a good member of the certification program.

  • For single games most officials on the crew are not in the Certification system at all

As Speedy said, [citation needed]. Sanctioned play attracts a much higher proportion of people seeking cert. Is there data on what proportion of officials in Sanctioned play have opted in?

  • For single games we think officials will be more likely to opt out

What does “opt out” mean in this context? Has it been made clear that it is possible for officials to “opt out” of evals, other than by dropping out of the system entirely? What’s the process for doing so on a game by game basis?

And if you’re really JUDGING officials on whether or not they submit evals, is there a process for ensuring that they aren’t dinged for officials on the crew who opted out? That sounds like something that could easily go wrong, unless things are carefully correlated.

“We see that for the last four sanctioned games you HRed, you only submitted evals for 50% of your crew,” is a problem unless someone is actually looking at who opted out.

  • For most sanctioned tournaments, this expectation is already in place

Is it? Has data been gathered on this point, or is this purely anecdotal?

  • We think that the “ten to thirty minutes to write” expectation will actually get much shorter as head officials get into the practice of writing evaluations.

Will it? Why does Cert think so? Writing a thoughtful eval takes time, and I’m assuming pro forma ones aren’t much use. And this doesn’t even begin to account for non-native speakers or people with neuro-spicyness that makes writing evals challenging.

8 Likes
12

Have you considered that there’s also similar barriers to opting-out on a game-by-game basis? There’s a lot of connotations that can be erroneously drawn from opting out from evaluation if the default is that everyone is opted in.

I would imagine that opting-out will at least on some occasions be because of those exact power dynamics, or because the head official for that game is not someone the official believes will give a fair evaluation. In those kinds of circumstances, those barriers to opting-out will be even higher.

Additionally there will be numerous officials who are at level 1 but who, for whatever reason, are not planning to progress to level 2. This situation was fine with an opt-in approach because only those seeking to advance would ask for one, but with an opt-out approach it means Head Officials will spend time writing evaluations that will never be used.

The guidance says that it should take ten to thirty minutes. This strongly implies that if you take less time than that, you are doing it wrong. I also agree with Smasher that there are whole segments of our community for who this will never get shorter, and may indeed take longer. This expectation may in fact discourage certain groups from focusing on the Head Official role. I can imagine realistic, plausible scenarios where the Head Official spends more time writing Evaluations than they spent actually in the sports hall.

Have you considered how this will impact officials who aren’t focusing on the Head Official role, and who don’t live in an area where many are? Their packs are going to look awfully slim compared to those who can regularly attend games where this expectation is in place. Will this in turn lead to them having to ask for even more evaluations from Head Officials who are not under that expectation, which means they have to overcome that power dynamic fuelled barrier even more frequently.

Instead of expecting evaluations to be done on all opted-in Officials, couldn’t it instead be that Head Officials are expected to ASK all opted-in Officials whether or not they would like one? This could be a simple yes or no question on the application form when signing up.

Alternatively, set up an indirect system. Set the expectation that certified Head Officials (not just those with a focus in it) should be prepared to provide an evaluation after any game. Then set up a form for the Official to complete either before or within a week after the game, where they select the Head Official and provide game/tournament details such as date. After the game the Head Official would be informed that an evaluation has been requested, and provide the relevant details. Reminders can be sent on a monthly basis until it is completed. As Cert would have access to the list of requests made, this would provide a quantifiable metric in terms of percentage of evaluation requests that were actually fulfilled.

The current system in the OP just seems overly laborious and inconsistent, creates wasted work, and introduces new problems. There must be a better way.

8 Likes
13

This is assuming that all Head Officials across the globe will see an approximately equal ratio of opted-in officials in their games. I don’t have numbers but would be very surprised if this is even remotely the case.

You are claiming this is just clarifying existing requirements and list this as one of those existing requirements. But if this is an existing requirement for sanctioning a game or tournament it should be listed in the Sanctioning Policy or a document referenced from there.

All the examples you gave in this thread so far are examples of evals being useful feedback for Cert. But the reasons why you consider this to only be a clarification is that heads are already expected to give feedback to the officials on their crew. It looks like you are treating these as the same when they aren’t.

The old cert system was closed because it became clear that the amount of evals necessary to consistently get this kind of a picture was too high to be manageable. Instead OOSs were supposed to paint this part of the picture while evals should highlight games/events where an official’s performance was noteworthy for some reason. If you are now saying that you need evals to get this part of the picture, what purpose are OOSs still serving? And why are you expecting the amount of evals to remain manageable this time?

5 Likes
14

This feels like a barrier for those who are amazing Head Officials but struggle to sit and write evals for people. Feels like a knock against them applying to higher levels if this is a heavily weighted piece of the puzzle. Wouldn’t it make more sense for them to receive evals on performance rather than have to write evals for others?

This will again penalise geographically isolated officials. Someone who lives in the middle of a derby metropolis who might have 10 sanctioned games in a year, and do more evals than someone who has to fly internationally to get more than 1 sanctioned game a year. I don’t see how you can compare the rates of game counts across certified head officials?

I’m going to be honest, this sounds like a slap in the face to those of us who spent the last decade attempting this to little or no avail … I feel like this should have been worded much better.

I agree. Does Cert expect a Head Official to do a self eval and state that no officials on their crew were eligible or interested in evals and why? Do you want an officials roster with it to confirm that we had certified officials on the roster?
Example - my league has 4 certified refs and we officiate together all year long - when does our doing evals for each other all year get to be repetitive and unhelpful to Cert?

6 Likes
15

I agree with others that this create a significant work from people that are already dedicating quite a lot of their time/energy to derby.
Certification provides a gain (trying to accurately assess officials + providing feedback), at a cost (time spent by evaluating officials).
I feel the cost is already not insignificant, so increasing it seems a questionable strategy.
From the answers in this thread, I get the impression that Certification is not accurately doing this cost/analysis balance, by underweighting the part they are less exposed to (time/energy needed to write evaluations (or OOS)), and overweighting the part they are more exposed to (certification goal, i.e. accurate assesment/feedback).

I also agree it seems also likely to create an extra barrier. I’ve seen officials who could be excellent HNSO refusing to be HNSO if they were expected to do the stats (because they did not have the time or energy for it). HNSO doing the stats is less an expectation nowadays, so it is less an issue, but I would expect this phenomenon to occur again, if the expectation is to do some extra administrative work when you are CH.

5 Likes
16

I’ll echo what other’s have said above, these expectations are not remotely reasonable, and pose a number of accessibility issues, both for geographically isolated and neurodiverse officials. The most common criticism that crops up in discussions of the Certification system (at least in my area) is the sheer volume of administrative labor involved in participating. This “clarification” compounds that existing issue immensely. I can see why these expectations may not be seen as burdensome by Certification, as volunteering to be a member of Certification likely selects for individuals for whom these tasks do not pose a significant challenge, but I don’t believe that skill profile can be generalized. If implemented, this set of expectations may artificially exclude many extremely talented officials from the certification program, and/or reduce the perceived value of higher level certification by equating those higher levels not with officiating skill or leadership skill, but with administrative skill. I sincerely hope that Certification takes the time to review community feedback and meaningfully address what steps they can take to make Certification more equitable, not less.

6 Likes
17

Tournaments here in Australia typically have the same (Crew) Head Officials. Why? Because 98% of the officials in Australia don’t want to take on the role of a Head Official because they don’t want the responsibility.
It’s not that they can’t so the role, it’s not that any level of help, encouragement, or training would help - it’s that they just flat out don’t want to do the role.

As a Level 2 NSO, I understand there is an expectation of providing Evals and will do it for all games when requested, not just Sanc games (of which I’m still waiting on evals that I requested over 6 months ago, and have no expectation of receiving. [Yes, I’ve followed up multiple times, but it’s now at a point where it would not be accurate.]). That is going to impact me much more in my attempt to get L3, than the Head Officials who haven’t done the Evals.

And echoing other statements here - How does it then become fair on those who are aspiring to higher levels but have other barriers for them to complete evals? You say it should get easier over time, but I’ve done my fair share of OOSes, and they have not gotten easier and still take me 3+ hours to complete

I absolutely 100% agree with this. It’s the main reason I didn’t apply to be part of Cert - I know I don’t have the capability of writing feedback that would be worth the paper it’s printed on, and I know this also applies to my Evals and OOSes, which is why I work so hard on them.

8 Likes
18

Well then to be honest with you, that sucks. That sucks that someone who is CH at a MRDA event or HO at a MRDA game doesn’t have to do evals to be considered a high quality official, according to you, but a CH or HO at a WFTDA event will be considered less-than if they don’t fill out un-requested evals. And yet when WFTDA playoffs come around again, the HO from the WFTDA event who DOES fill out all the eval hoops is considered to have put in equal effort to the HO at the MRDA event that did not necessarily get the requirement to fill out evals. That the WFTDA HO gig requires more effort and yet counts equal to gig that require less effort sucks as a system the powers that be are choosing to put in place.

Yes, I know evals can be filled out for MRDA games. I’m talking about the “expectations” that are being listed in the OP. MRDA games have less expectations, then, than WFTDA games and yet are counted as equal experience?

19

If Cert only received the results of the survey about the certification process a month ago and hasn’t yet reported on the conclusions, it seems that that step is needed before processes are revised or clarified, as they may change with that input. The responses here suggest the community would like to see data supporting the need for a clarification at this time.

5 Likes
20

Hi – we’ve got too many items to respond point-by-point to so I’m going to try to address the gist of what I’m seeing. First I will address the implied question – why are we doing this? What is our motivation for this, and why now?

(First, please note that the term “level” in the original post and in our follow-ups, is meant to refer to certification level, and “higher” levels refer to Levels 2 and 3. We do not use the term “higher-level” as a synonym for “better” or as a way to express that they are officiating higher-stakes games.)

  1. Complaints received during office hours, in person, and via email, regarding:
  • Applicants not having evals on-record, which delays them from applying
  • Timeline for evals being unclear, or about how applicants’ head officials break promises about when the evals will be fulfilled by.
  1. Incomplete packets, with missing information from games an official has already worked, leading to:
  • Extra work for our panels and panelists as the decisions are harder to make and the summaries are harder to write and harder to review
  • A pattern of undercertification of officials, especially uncertified officials who apply directly for Level 2 or Level 3
  • Delay, sometimes by a month or more, of packets while we go around in circles;
  • Further delay, sometimes by a month or more, if we need to go back to an official and list the missing pieces of the puzzle; we usually ask them about specific events in their officiating history as ones for which an eval would be extremely helpful – and the head official is usually quite willing to provide it!
  1. A repeated pattern of those extra asked-for evals coming in, and being very very helpful, and putting things in order for confident decision-making.
  2. A pattern of missing information for higher-level officials regarding how they are providing feedback and helping to grow the community. For higher-level officials who are often Head Officials and who do not often work in “all” roles – how are they showing training or mentorship for those other roles and how are they helping their crews to improve.

Together, items 1-3 these suggest that expectations for head officials are unclear, and that expecting every official who wants an eval to set expectations on their own is a significant barrier; this implies a need for this Clarification.

Items 2 and 3 also tell us that missing information from evals is highly impactful on our panel’s self-efficacy and on the results we are able to provide, because of the struggles before we have the information, and because of the clarity we experience when we delay or pause packets until we receive it.

Item 4 tells us that the solution is also a win-win, as the individuals whose evals are most needed, themselves often need evidence that they are providing feedback and helping their crews to grow.

It should be noted that the above is a qualitative analysis. The WFTDA does not have a “data team” and data analysis is a highly-paid and highly time-consuming discipline, and we are a volunteer-run org. If you or anyone you know would like to take on a data analyst role for Certification please email us at certification@wftda.com and we will work to create / train the role.

(Yes there are other points noted above which we will try to respond to in due course.)