Results and Rankings.

[martin]

The anomalies keep coming, and none of the tweaks I have seen so far will fix them.

The reason is the innocuous looking clause 2.2.5 in appendix K "The
current scores of all ranked runners are rebased after each event to ensure that the
mean current score of all ranked runners is 1000 and the standard deviation of their
current scores is 200."

The numbers 1000 and 200 came from the old ranking scheme, which only ranked individual classes. Imagine if every ranked orienteer ran the same course, the distribution of times would be much wider and the standard deviation/mean ratio would be nowhere near 200/1000, probably nearer 300/1000. Until this clause is modified, lower ranked (usually shorter course) competitors will continue to have inflated points, thus giving inflated points to anyone competing against them.

[Homer]

It will be interesting to see how you go on and whether your Black score is comparable with your usual scores.

In my experience you rarely get any ranking points running Black as there are usually less than 10 competitors.

[mappingmum]

I find all this discussion and dis-satisfaction with the single ranking list quite strange and bizarre.

It's not an exact science therefore it will inevitably display nuances. Each course is a single statistical population with no correlation, rule, or logic which stipulates that the longer courses generally contain faster runners and should therefore score higher.

We are very lucky to have it. Road races don't feel the need to produce a ranking list for the whole of the UK do they?

I suggest you accept it for what it is; mild amusement. Keep moaning about it and I'm sure the powers-that-be will just withdraw it from service. Why maintain a system which everyone moans about?

[DJM]

Until this clause is modified, lower ranked (usually shorter course) competitors will continue to have inflated points, thus giving inflated points to anyone competing against them.

This is nonsense! Changing the standard deviation from 200 to, say, 300 merely increases the separation of everyone's points by 50%. In other words, the distribution of points just gets spread out more and no one group will get inflated points as a result.

[superstartradesman]

I find all this discussion and dis-satisfaction with the single ranking list quite strange and bizarre.

...

I suggest you accept it for what it is; mild amusement. Keep moaning about it and I'm sure the powers-that-be will just withdraw it from service. Why maintain a system which everyone moans about?

Well, that's us told! But, d'you honestly think that a reply like that will shut us up?

Seems to me the existence of a ranking list provides motivation for enough orienteers such that they will be inclined to point out anomalies and suggest improvements to what is a valued and appreciated part of our sport. I'd hope it's all seen as constructive input.

And since this thread's title is pretty indicative of the topic, people who are uninterested in this sort of debate should easily be able to avoid it. Right?

[mappingmum]

Yes, but Dave, I have avoided [contributing to] it for 16 pages !!

I just reached breaking point!

[DJM]

This thread has provided excellent feedback on the rankings in general and the calculation in particular. As one who co-wrote the spec for it, I've taken a particular interest in what's been said (yes, I've read all of the hundreds of posts) and have been grateful to those who've pointed out anomalous results which I would not otherwise have seen.

Having analysed the data behind many of the problems, I'm now pretty convinced that the vast majority of the discrepancies (all, maybe?) originate from courses with small numbers of competitors on them. These discrepancies then get fed into subsequent course calculations, thereby "infecting" their results.

To cut a long story short, when we changed the definition of a counting course to one with at least 10 previously ranked runners, we should also have removed a "fudge factor" formula designed to cope with very small classes and which still applied where there were up to 20 previously ranked runners. Now, this formula worked quite well in most cases, but under certain circumstances, e.g. a particularly wide spread of ability on a course, it produced the anomalies previously highlighted.

When I removed the unnecessary "fudge formula" and recalculated in Excel, the problem for each suspect course was solved and the generated points looked entirely sensible.

I'm therefore optimistic that when the changes are made to the real database, we'll get a major improvement to the list.

[martin]

DJM thinks "This is nonsense! Changing the standard deviation from 200 to, say, 300 merely increases the separation of everyone's points by 50%. In other words, the distribution of points just gets spread out more and no one group will get inflated points as a result."

Updating a data distribution(the ranking list) with new sample data (the latest event) requires the distributions to have similar statistics for the process to be meaningful. Artificially constraining the width of one of the distributions will give errors particularly for the data outliers (the course winners and losers). This is just basic statistics!

[andypat]

To cut a long story short, when we changed the definition of a counting course to one with at least 10 previously ranked runners, .

Is that right? Is it not possible to gain any ranking points from a course with less than 10 previously ranked runners? This seems a bit arbitrary and possibly unfair to those (poss older or on black) competitors who run on courses with less people in. Anyone know how often this cut off applies?

[DJM]

martin wrote:
Updating a data distribution(the ranking list) with new sample data (the latest event) requires the distributions to have similar statistics for the process to be meaningful

Absolutely - you have to compare like with like. The implication behind my reply was that I was comparing what the list would look like if the whole of it had an SD of 200 with its shape if the whole had an SD of 300. I therefore stand by what I said!

[Spookster]

To cut a long story short, when we changed the definition of a counting course to one with at least 10 previously ranked runners, .

Is that right? Is it not possible to gain any ranking points from a course with less than 10 previously ranked runners? This seems a bit arbitrary and possibly unfair to those (poss older or on black) competitors who run on courses with less people in. Anyone know how often this cut off applies?

Yes (refer to earlier posts for discussion about small numbers of ranked runners on a course being a cause of anomalous results). Looking at events in the last 3 months or so, about half the Black courses have had enough ranked runners, including the last two SOLs. There is no problem with older runners - the courses they run have plenty on.

[AndyC]

To cut a long story short, when we changed the definition of a counting course to one with at least 10 previously ranked runners, .

Is that right? Is it not possible to gain any ranking points from a course with less than 10 previously ranked runners? This seems a bit arbitrary and possibly unfair to those (poss older or on black) competitors who run on courses with less people in. Anyone know how often this cut off applies?

On Courses other than Blue & Green in the NE the answer is "mostly" -often we don't have 10 runners let alone 10 ranked runners on the shorter courses. At the last event I planned The Kielder Trophy only the Orange, LG & G had more than 10 runners -and I suspect only the G had a chance of having 10 previously ranked competitors

[DJM]

andypat wrote:
Is that right? Is it not possible to gain any ranking points from a course with less than 10 previously ranked runners? This seems a bit arbitrary and possibly unfair to those (poss older or on black) competitors who run on courses with less people in. Anyone know how often this cut off applies?

The cut-off has to come somewhere and 10 matched the previously accepted definition of a statistically "small" class. Yes, it's unfair if you happen to run on a course which has fewer than 10 previously ranked finishers on it, but (a) this in practice affects a very small percentage of runs, and (b) to reduce below 10 would result in an increased number of courses with anomalously awarded points ... and the subsequent nopesport storm!

How often? I did a check on all the results submitted in January this year and, of 4066 competitor runs, 48 would have been eliminated under the "more than 10" rule. NB this rule has been in operation since April this year without any adverse feedback ...

[Scott]

Looking at events in the last 3 months or so, about half the Black courses have had enough ranked runners, including the last two SOLs. There is no problem with older runners - the courses they run have plenty on.

I suspect Very Short Green would fall foul of this rule as often as Black does, but at the moment relatively few clubs bother to put it on, even at big events.

But as we established in the earlier discussion, the cut-off needs to happen somewhere.

[peawet08]

To cut a long story short, when we changed the definition of a counting course to one with at least 10 previously ranked runners, .

Now I think this system is even more flawed than I did before. So when I am entering in advance on Fabian 4 I need to wait until the last minute to make sure that there are enough competitors for it to actually make a difference to my ranking. Others may also be doing this and as they want their run to count towards the rankings so hold off. As a result they don't enter the Black course as there aren't enough runners (beieng as they are waiting to see the number of competitors entering) so I end up entering the Blue to be sure of a ranking score.

If courses where graded based on distance, climb, TD and number of controls then a multiplier can be assigned to each course to give competitors a comparative score based on the event as a whole. Isn't a similar thing done on the street O events I do but with time being the decider (Which make the calculation easier). Incidentally they also produce an overall result and age adjusted results.

I find all this discussion and dis-satisfaction with the single ranking list quite strange and bizarre.
...
I suggest you accept it for what it is; mild amusement. Keep moaning about it and I'm sure the powers-that-be will just withdraw it from service. Why maintain a system which everyone moans about?

Well, that's us told! But, d'you honestly think that a reply like that will shut us up?

The way I see it is that with all the fees we pay to enter and our membership of BOF, along with subsidies received from government bodies (Which we paid for in tax) I think we have a right to comment on ranking system create by BOF.

This thread has provided excellent feedback on the rankings in general and the calculation in particular. As one who co-wrote the spec for it, I've taken a particular interest in what's been said (yes, I've read all of the hundreds of posts) and have been grateful to those who've pointed out anomalous results which I would not otherwise have seen.

Just because it's free doesn't mean we shouldn't question it! Feedback only helps to evolve things.