Wednesday, April 08, 2009

Cosplay Judging Criteria and System

While I'm never going to be as active as I used to be in, I am reposting this so the world that cares about costuming and cosplay may know how it's done here in Manila.

And for the record, while it's me a while to grasp the logic and I did not at first find myself in agreement with it, I understand now that this new system has its merits, perhaps more than I gave it credit for.

So here it is, the post of Kero/Pablo of on the issue at hand:

Ever since has managed cosplay events, we've been tweaking the judging system to try to find the right mix of aspects to ensure a system that is the most efficient, leads to the least errors, and allows the judges to focus on aspects that really matter.

With this in mind, we have been implementing a system where each judge is assigned a criteria instead of all the criteria for various reasons:

1) It allows the judge to focus on their expertise. They can ignore other aspects and really focus on their criteria. It allows them to consider their score at length given the only 1-2 minutes he or she has to observe the cosplayer.

2) It addresses the concern that a person might be judging something that is not necessarily their field. For instance, a judge might be more into making costumes for other people and simply allows someone else to perform in his costumes or another judge who is good at performances might simply be commissioning his or her costumes.

3) It reduces the chance of "score copying" that occurs when a judge starts to get behind and simply copies the scores of the judge beside him or her to catch up (the judge can't really stop the cosplay just to catch up.)

4) Each criteria really makes an impact. For instance, if there were four criteria and one was 10%, most judges would just score a 5 in almost all contestants thinking that it's insignificant. This undermines that criteria and renders it useless. By making it their sole concern, they can award a 0 or 5 or 10 or anything in between if they feel they really deserve it.

5) It reduces errors simply because you are reducing the numbers that are being tracked and the considerations being made by 1/3 (for 3 criteria) or 1/4 (for 4 criteria). So, in a cosplay with 100 contestants, instead of 300 or 400 numbers, you just need to keep track of 100 numbers. In addition, this gives judges time to double-check their scores and keep track of their results and even reconsider some of their scores.

Yes, indeed cosplay is "the whole package", but their weight should not be treated equally. In the case of E-Games, it was mostly Craftsmanship/Accuracy > Performance > Audience Impact. Does this mean that Audience Impact is useless? Perhaps in the old system where most judges would've just scored all the contestants a 5 it would be, but in a system where the judge gets to study the impact of the cosplayer and his or her projection carefully, it becomes pivotal.

Just to prove how pivotal it was, the top 3 winners were decided on point differences of 1-2 points with the Best Female winner being decided by a single point! The Audience Impact score could have, in fact, DECIDED the outcome of the winners. You're talking about 10 points: something that could have put the 6th placer in 1st place.

About the fact that newbies aren't being represented and that scoring was manipulated, let's take a look at the results:

Best Male: 1st Time Cosplayer
Best Female: 1st Time Cosplayer

1st Placer: Cosplayer Since November 2008 (Kid Robocop)
2nd Placer: Brand New Costume. Last time he cosplayed was last year (Spongebob Squarepants)
3rd Placer: Cosplaying Since this year (Kid Terminator)

Where is the manipulation and prejudice against newbie cosplayers? In fact, Hitman and Katsune (both multi-award winning, veteran cosplayers) participated and did not even get 3rd place simply because we let the scores stand for themselves. We adhered to the rules that were set and did not "tweak" or "manipulate" them in any way.

To be honest, I had concerns because the 1st and 3rd placers were both siblings, but despite this we allowed the scores to stand BECAUSE we respected the judges and the rules and made their decisions final. Integrity in the cosplay contest and the adherence to the rules was our main concern. We may not have entirely agreed with the decision of the judges, but we trusted their judgment and felt that the winners were indeed deserving.

With the choice of judges and their expertise, how would you have ranked the following in the different criteria (Craftsmanship/Accuracy, Performance, Audience Impact):

JM Chua
Robert Wong

With the criteria having to be equal, let's be honest. Who would want Audience Impact to have the same percentage as Craftsmanship/Accuracy and Performance? Does this represent cosplay properly that a cosplayer having a great costume and poor projection is scored the same as a cosplayer with great projection and a poor costume?

In fact, if there was a problem with the judging system, it could have been brought up when Alodia herself was a judge during Tagcom 3. In the end, you can't argue with results. Using this system, score tabulation has been FASTER and MORE ACCURATE. Errors have been minimized (if not completely eradicated) and all the criteria emphasized. In the end, the result is a smoother cosplay contest that has resulted in deserving winners.

Or would it be better to wait 3 hours to come up with the winners and to ask cosplayers to come back on stage because a judge didn't catch their performance or wasn't able to look at their costume properly?

Original source: (sorry, members only)

This is in response to Alodia's allegations as stipulated in her deviantArt account:

As I am no longer in activity as a CosPh junior administrator (due to personal and work-related circumstances), this will be my first and last public post in the matter. For those interested to take this up with me, do so at your risk.

No comments: