Event Structure Review: Rankings List

[DJM]

Graeme is spot on in his reply. We need to find a way of kick-starting the new rankings list; thereafter it should take care of itself.

But the kick-starting should be done as carefully as possible, the major problem to be solved being how we take existing data for all age groups and combine them into one list. How do M40L points relate to W55L points for example? Well, we think we know something about running speeds of the leading M40s and the leading W55s so we can multiply their existing points by the published speed factors and add them to the new list, just as SimonE has already done.

But we don't really have much data about how S runners compare with their corresponding L ones. In some classes, the difference is not great whilst in others it's much greater; using a one size fits all conversion from S points to L points will inevitably result in large anomalies. But why do we need to consider existing S class rankings points at all? Simon's spreadsheet has 1609 L runners with 6 sets of results and these should be more than enough to "seed" the production of the new overarching rankings list. As soon as these runners start competing at the proposed new "standard" events, they will allow others to acquire points. These others then go to further events, creating more opportunities for points to be gained, and the process repeats itself. The overall list will expand very quickly indeed until it reaches the steady state Graeme refers to.

In fact, I'm not sure that we need as many as 1609 to kick start the process - maybe we could just use the leading 25% in each L class instead, thereby minimising the problem that the speed ratios of the best may not be the same as those for the average in each class.

[Adventure Racer]

In fact, with a bit more twiddling, wouldn't it be easier to use this as the basis for the new ranking (rather than trying to retroactively incorporate data from 2007 CC events)? There will be lot more ranking events under the new system - so it shouldn't take too long for "fresh" scores to predominate

The point is that you do all these adjustments, but do them with the data at the end of 2006. If you then incorporate data from 2007 events it provides you with a much better ranking than if you hadn't - effectively puts you a year ahead. Of course doing this exercise also allows you to have rankings based on a rolling 12 month period as at present - something which otherwise wouldn't be possible until the end of 2008 (at least not without potentially making a huge mess of the whole thing).

[jac]

But we don't really have much data about how S runners compare with their corresponding L ones.

But we have lots of data about how S runners compare with the runners in other L classes on the same course. For example, M21S is often on the same course as W21L / M45L / M50L. So the S course runners can be directly ranked against L course runners of different classes. And we already think we know how to relate the various L classes to each other.

This is what I was getting at when I wrote:

The data for each race will need some pre-processing to decide which classes ran the same course. This might be tedious, but it will help bring the Short course runners in to the main ranking list.

As a quick experiment, I looked at my results for the last year on M21S as if they were on the 'other' L classes. My hypothetical ranking points for M21S, W221L, M45L, and M50L come out to be reasonably similar to the awarded M21S ranking points. Noting that the 'official' running speed ratios are W21L = 0.8, M45L = 0.84, and M50L = 0.79, this tends to suggest that the M21S derating factor should be around 0.8 to 0.85. This is about the same number as I got when comparing runners ranked on both L and S classes for their age group.

Inspection of more results from more events and more competitors might give a better estimate of the M21S ratio. And similarly for the other S classes that share courses with L classes.

I might be persuaded to help with this preprocessing.

jac

[2006, 2007: M21S / 2008: M45L]

[PG]

In the discussion on kick-starting the new rankings list it seems the assumption is that we start with current ranking list total points per runner.

But in some classes it seems easier to get a high number of points than on others. Picking a few leaders: W21L Jenny Johnson 7526, M21L Oli Johnson 7766, M35L Steve Nicholson 7690, W35L Heather Hartman 7643, W50L Jackie Hallett 7971, W60L Pauline Ward 7947 , but M50L Martin Dean 8253, M55L Axel Blomquist 8359, W55L Inara Gipsle 8374 , M60L Mike Murray 8226. So Inara and Axel have 11% more points than Jenny.

There are two possible explanations. Firstly that those with over 8000 are a greater distance ahead of their class average, but secondly that the whole class has inflated points compared to others. If it is the second then converting to a new single list using standard running speed ratios will provide a distorted result.

Maybe the points per runner needed to be factored by the average number of points of all runners in the class with 5 scores? Or by the average number of points of the top 20% in the class?

[DJM]

Or by rebasing each class so that its mean is (six times) 1000 and standard deviation is (six times) 200 to see if this brings the leaders into line. Or, alternatively, by scaling each class so that the leader has the same number of points and then taking things from there.

[Oldman]

Should I be concerned that (as what I think of as one of your average punters) I understand next to nothing of all the previous posts in this thread?

Is comprehension of a ranking list by the "average punter" (and possibly also how it works) not also important?

[greywolf]

Re: PG and DJM 's posts: As i understand it, the current ranking algorithm is intended to ensure that the class average score is around 1000 points per event. If the ranking list utilises competitors best 6 scores, then clearly the mean score from the ranking list is going to be more than 6000, and it makes no sense to try and reduce it.

As for the disparity in class leading scores: those with scores over 8000 are indeed a greater distance ahead of their class average - but I think this reflects the "missing " intake - the sport isn't recruiting M/W 21s to fill out the lower reaches of these classes...the better the "average" performance, the harder it is for the very best to be way ahead of the average.

[DJM]

Oldman wrote:

Is comprehension of a ranking list by the "average punter" (and possibly also how it works) not also important?

Yes, of course it's important that the ranking list should be comprehensible but it's also important that it produces sensible results too and these two criteria are unfortunately at odds with one another. If all runners ran at every ranking event then we could devise a much simpler, utterly transparent and accurate ranking system.

However, we have to take into account event A, where the best runners turn up and where the terrain is especially challenging so that times are spread out, and also event B, where the turnout is much less strong (and no one from event A is there either) and the technical and physical difficulty is lower meaning times are more bunched. And of course all the other possible combinations of "who turns up" and terrain nature in between.

No transparent system can cope with these challenges (the old BOF rankings system tried valiantly but failed to allocate points fairly enough) and the only way to do so is to mathematically compensate for strength of field and terrain type. The principle is that the same standard of run should get the same number of points irrespective of who else is running the course or whether the terrain is The Trossachs or Brandon Forest.

The maths used to take into account these factors is well within the capability of an A Level Statistician and I'm sure that many nopers are able to follow it too. But of course many others are not and their only option is to accept the results on trust, much as I have to accept on trust the tennis rankings, the FTSE index, the cricket Duckworth-Lewis calculations etc etc.

[Adventure Racer]

The other understanding issue is that this thread is going into greater depth about both how to prevent distortion in the rankings and how to provide a baseline to kickstart the rankings based on existing data. Understanding the actual rankings system itself once it's going should be easier, and no more difficult than understainding the current rankings system (I'm afraid if you don't understand that, then you'll have to continue to take it on trust!)

[roadrunner]

I have to say that I have some sympathy with Oldman's views - and as a professional engineer I'm perfectly capable of understanding the maths involved and the reasons behind it. It would be interesting to know how different the current ranking list would look had we stuck with the previous, simpler but less "fair", system.

There's another point which hasn't been raised yet, though, and that's the order in which data is processed. One thing I hate about the present list is that my score for an event can change as a result of an earlier event submitting its results late - even if I didn't take part in that earlier event! Will the new scheme be the same? Maybe it would be better to process data in the order received rather than the order of the events (after all, for multiple events on the same day, what's the correct order anyway?), and to place a stricter cutoff on submitting final results - i.e. if they aren't in within (say) 2 weeks after the event, they won't be included. OK, there's a risk that this might distort the results a little, but I can't see that anyone could use this to their advantage.

[Roger]

...after all, for multiple events on the same day, what's the correct order anyway?

If they're on the same day then there will probably be no runners who do both, in which case the order doesn't matter, I believe. (Night events are the likely exception that proves the rule.)

[Simon E]

...in which case the order doesn't matter.

The proposal seems to be that the whole ranking list is renormalised after every event. Therefore the order of two events on the same day does have an impact, even if (in theory) quite a small one.

[Adventure Racer]

...in which case the order doesn't matter.

The proposal seems to be that the whole ranking list is renormalised after every event. Therefore the order of two events on the same day does have an impact, even if (in theory) quite a small one.

Surely the answer then is to renormalise every day (or after every day of events) instead. Could even just be done once a week (on a Sunday evening) without making a significant difference.

The issue over day and night events is quite easily resolved by inserting the day event first (or given somebody running in two daytime events, inserting them in chronological order - I presume you wouldn't get 2 people both doing 2 different events and running different ones first?)

[DJM]

Adventure Racer wrote:
The issue over day and night events ...

This raises again the issue over whether or not night events should be included in the rankings scheme. The argument in favour is that night orienteering is just another branch of orienteering, along with e.g. Sprint and Middle, and also that those who run at night also run at day time.

The argument against is that, unlike with day orienteering, a small mistake at night can have a totally disproportionate effect on the time taken and some highly anomalous results can occur as a consequence.

My own view is that night results should not be included in the rankings lists as they can distort the order quite signifcantly, and that some other way should be found of recognising runners' status at night. I'd be interested to hear how many others agree with this ...

[Snail]

[quote]I presume you wouldn't get 2 people both doing 2 different events and running different ones first?)

Not impossible.

Some clubs offer a 3-in-1 series of 3 events on the same day, and a few years back SEOA offered 4 events around the M25 on the same day. You choose how many to go to, and in which order.