Event Structure Review: Rankings List

[greywolf]

An ideal ranking system would be based on events where everybody turns out.

er...if everybody turned out then you wouldn't need a ranking system would you? I know how I rank against the people I race against locally, that's what we have results for.

of course i want to have as high a ranking as possible, because I want to be as good an orienteer as possible, but I want my ranking to reflect my orienteering ability (or lack of), not whether i can afford (time & money) lots of 1000+ mile round trips to go to National events where I know I will get inflated ranking points.

and i'm aware that the S6D is a holiday event - but other posters in favour of weighting have suggested that quantity and quantity of competitors at Nationals justifies the weighting - in which case i'm pointing out that the S6D exceeds them all on both these grounds and probably on quality of terrain too.

[Gnitworp]

(but then I'd personally only use national events anyway!)

Did you know, awk, that there are already ranking lists using National Events only; the best kept secret in orienteering, the Masters Cup? It's been running, believe it or not, since 2002!

[awk]

er...if everybody turned out then you wouldn't need a ranking system would you? I know how I rank against the people I race against locally, that's what we have results for.

Yes, you would still 'need' a ranking system, especially in a sport like orienteering where results week on week can be so different, even at international level. Similarly for local competition: there are plenty of people I race against locally where they will beat me one week and I will beat them the next.

of course i want to have as high a ranking as possible, because I want to be as good an orienteer as possible, but I want my ranking to reflect my orienteering ability (or lack of), not whether i can afford (time & money) lots of 1000+ mile round trips to go to National events where I know I will get inflated ranking points.

Orienteering ability includes the ability to produce a performance when it matters and on a wide variety of terrain; I don't care if somebody beats me at some local race, which I regard as training, if I'm beating them at the big ones (and vice-versa!). On 'inflated' points, I see it as the nationals as the standard score, and more local events being reduced - but the maths is simpler this way round.

However much you argue it, I simply don't see how you can equate performances at local 'badge' events with those at the British Champs or regional championships. Simply doesn't stack up. I accept that this then means that the ranking structure has an element of being an attendance prize in it, but that's the nature of sport: if you want to rate against others nationally, then you must be prepared to compete in national competitions, and I know that will be more difficult for some than others. This was discussed in some depth at the event review group, but whilst the "every event the same" case was argued very coherently, on the balance of the arguments a significant majority came out in favour of weighting. We'll see if the same happens in the consultation process.

Gnitworp - yes I did know (I tried setting something similar up some years before the Masters Cup came about through the efforts of Peter Guillaume I think), although of course the scoring system isn't really about ranking in the same way (personally, I prefer it!) which is why the two can run alongside each other. The event review group are recommending giving it more prominence.

[greywolf]

Yes awk, I agree that the championships are more "important" than local events, and that the best orienteers will be those who perform best at the biggest events...but i can't see any evidence that the current unweighted list is top heavy with runners who've accumulated massive points at remote minor badge events.

simple test - look at your own position in the current ranking list and those around you (ignoring anyone with less than 6 scores, obv.) Now add 10% to your points total, and see what company you're in. And then subtract 10% from your current points total and do the same. Which of the three most honestly reflects your ability?

and it's worth noting that (according to the consultation doc anyway) the proposed weighting has nothing to do with producing a better rankings list, it's being done to encourage participation at and improve the status of Area Championships - fine objectives, but not worth screwing up the ranking list for...

[Gnitworp]

The event review group are recommending giving it (the Masters Cup) more prominence.

Shouldn't be difficult!

How about a Cup!

[Godders]

but i can't see any evidence that the current unweighted list is top heavy with runners who've accumulated massive points at remote minor badge events.

M21 is often influenced by small events. I was ranked number one for a long time by running fairly well (but rarely winning) at small, technical and physical Scottish races. The same applies to any class where you can get 6 technical & physical races with less than 20 runners. I knew I wasn't the best, everyone else who knew the M21's knew. I'd favour removing races with less than about 20 runners as not statistically significant.

[The Loofa]

Sorry Greywolf but I reckon you've got the wrong end of the stick entirely with this subject

Let's start with your first post - weightings will reward carbon footprint
The ranking list rewards the big carbon footprint already. You may go to 6 ranking events a year, but someone the same standard as you goes to 30 they will rank higher because their best 6 will outscore your best 6 (assuming of course you both have a fairly normal spread of good/bad runs). As other posts on the this subject already said any ranking list will reward higher attendance (a.k.a big carbon footprint) whatever you do.

Now weighting of more prestigious events
This has nothing to do with encouraging partcipation or better performance. It has everything to do with reacting to better performance. To explain
You run your nearby Regional event and score x points. Next weekend you go to the BOC and have the same standard run, but all the others in your class were treating last week as training, and at BOC they''ve pulled out all the stops and run 10% faster. So (without any weighting) you'll get 10% less points than you got the last week, or to put it another way you've got to pull out all the stops and run 10% faster just to get the same points (if there were no weighting).
Now if you've had a 10% better run shouldn't you get 10% more points? The weighting is needed to refect the fact that people are trying harder and having better runs.

Adding or subtracting 10% to your score in the current list is a pointless exercise, unless you know that your top 6 runs were all at Regionals and everyone else's was all at Nationals (really!!??). In practice most people would have a similiar sort of balance of Regional/National events.

have you ever heard someone say "I had a really good run at BOC and it doesn't count for me in the rankings". OK maybe not the most common question, simply becuase few people analyse their ranking scores, but among those that do, they ask it pretty often. In 2006 Colin Dickson was undisputed best M50. He was top of the rankings and won BOC and both days of the JK, but none of these 3 races counted in his best 6 scores; his best points came from one of the smallest Regionals going held a week before the JK - somehow don't think he was trying his hardest there.

So we need the weighting so that our best ranking points come from our best runs, and for most people that's at the most prestigious races

Andy

[greywolf]

Now weighting of more prestigious events This has nothing to do with encouraging partcipation or better performance. It has everything to do with reacting to better performance.

The consultation document (Recommendations 5 & 6 quoted previously) is pretty explicit that the purpose of the weighting is to encourage participation in and increase the status of the rebranded Area Championships. If that's not what the Event Structure Group thinks or meant then someone needs to rewrite the document.

[deebee]

There's got to be something wrong with a ranking system that gives 1100 points for winning a National , 1200 for finishing second in the following day's Badge event and 1300 for winning from two other competitors in another Badge event a few weeks later.

[awk]

The consultation document (Recommendations 5 & 6 quoted previously) is pretty explicit that the purpose of the weighting is to encourage participation in and increase the status of the rebranded Area Championships. If that's not what the Event Structure Group thinks or meant then someone needs to rewrite the document.

Yes they were certainly two of the main reasons. We spent many hours discussing many items - I have a huge file of emails just from the review group. If the report had included all the reasons for everything and everything the group thinks, you would still be reading the document rather than discussing it!

[DJM]

deebee wrote:

There's got to be something wrong with a ranking system that gives 1100 points for winning a National , 1200 for finishing second in the following day's Badge event and 1300 for winning from two other competitors in another Badge event a few weeks later.

Quite right! There has got to be something wrong with such a system: the main issue is that the current system is designed to work well when there are large numbers on a course whereas, with the exception of the older men's classes, most Badge events have classes which are far too small to give reliable results.

For example, a typical C3 has 48 rankable classes whilst a typical CC event (including Black) has 4 or 5 rankable courses. If 300 seniors attended the C3, there would be an average of 300/48 = 6.25 runners per class - far too small to give reliable rankings results. However, if those same runners competed at a new-style "Standard" course with, say, 5 rankable courses, there would be an average of 300/5 = 60 runners per course. Because the single rankings database takes into account every one of the 60 runners, the points calculation will be hugely more reliable.

The difference is dramatic and will lead to a huge improvement in the Rankings scheme. Quite apart from the Event Structure Review group's proposals, a change to a single rankings database rather than having 48 separate ones could be made within the current event structure and would lead to some worthwhile improvement in the Rankings reliability.

[Simon E]

The single ranking list is a great idea in theory. I'm not sure that it is so easy to implement in practice.

You will need to find a way of correlating performances across courses. Let's say GG wins the brown, whilst GG's dad has a flyer and wins the blue. How many points should each get? How do I compare an M21 to an M55? This presumably either relies on arbitrary weighting factors between courses and classes, or needs to find enough people who run on a spread of courses to allow their results to be used to scale things as necessary.

I did start looking at various possibilities some time ago. The ELO rating system is used in chess to rank people who haven't necessarily played against each other and might be a place to start. The Glicko system is an extension of ELO which looks interesting. Microsoft use a system called TrueSkill to "to both identify and track the skills of gamers in a game (mode) in order to be able to match them into competitive matches". The American College Hockey Association uses a pairwise comparison method to rank teams even when they haven't played each other, which is also close to what we are trying to do.

One thing all of these have in common is that they aren't going to be "simple", so I'd fundamentally disagree with the second sentence of Recommendation 10 which states that "A simple system of producing ranking points from raw event data needs to be developed".

[greywolf]

You will need to find a way of correlating performances across courses. Let's say GG wins the brown, whilst GG's dad has a flyer and wins the blue. How many points should each get? How do I compare an M21 to an M55?

As I understand it, how many points they get will still depend on not whether they win but on how fast they are compared to everyone else competing - as DJM points out, fewer courses, and not slicing up the field into ridiculous numbers of age and L/S classes should mean the calculations are more robust.

It's a bit more complex to compare rankable classes: presumably the simplest way (assuming they're all the same TD) is a mins/(km + height/100 gain) calculation - the question then would be does the (usual) increased density of controls on a Green balance out the longer distance of a Brown, or does it require some kind of fiddle factor whereby e.g. 10 min /(km+height/100) on brown = 10.30 min /(km+height/100) on blue = 11 min /(km+height/100) on green?

You'd generally expect that the winner of brown (and sorry, but I haven't got my head round the new colours yet) will get the most points ... but also probably that the winners of the blue and green courses would score more than some slower competitors on brown. The best thing about a combined list is that a given performance will have a single score - under the current system, 8 athletes can complete the same course in exactly the same time - and get 8 very different scores.

[Roger]

...I'd fundamentally disagree with the second sentence of Recommendation 10 which states that "A simple system of producing ranking points from raw event data needs to be developed".

I'm with Simon on this. However, a system can be 'simple' to operate (event software exports the results in a suitable format, user uploads file to server, ranking lists are updated) even if the underlying calculations and data manipulation are quite complicated. Maybe this is what the report means by 'a simple system'? Would one of the authors like to enlighten us?

[DJM]

SimonE wrote:

You will need to find a way of correlating performances across courses.

I was told to keep the rankings bits of the report as non-technical as possible in order not to put people off so preliminary thoughts about how to start off the overarching database were left out ...

The simple way to kick start the new rankings system is to take existing rankings points for all (Long) classes and then multiply them by published ratios of running speeds. Thus, if a W55L has an average of 1000 points under the current system, she would be included in the proposed overarching database with 1000x0.53 = 530 points (BOF speed ratio = 0.53). An M21L with 1000 points now would start with 1000 points in the new database (ratio = 1.00).

There are a couple of problems with this simplicity. Firstly, the BOF speed ratios date from 1997 and may not be appropriate today. These ratios are currently under review and any revised figures should be used instead. Secondly, the ratios apply to winners' speeds only and may be different for the average runner in each age class. So appropriate correction needs to be made for this factor too.

One way or another some time and effort needs to be put into a once and for all production of the single database of rankings points. Once this is done, then the Brown and Blue courses which Simon refers to are simply treated as separate "events" (just as if they actually occurred at different venues) by the rankings calculation system and there is no need for any course length/height climb data ever to be considered.

This answers greywolf's point too.

Simon E wrote:
...I'd fundamentally disagree with the second sentence of Recommendation 10 which states that "A simple system of producing ranking points from raw event data needs to be developed".

Roger wrote:
I'm with Simon on this. However, a system can be 'simple' to operate (event software exports the results in a suitable format, user uploads file to server, ranking lists are updated) even if the underlying calculations and data manipulation are quite complicated. Maybe this is what the report means by 'a simple system'? Would one of the authors like to enlighten us?

I wasn't very comfortable with this recommendation as I thought that the present way of entering race data to the BOF rankings system was pretty straightforward. However, I was assured that others reported nightmares in trying to do the same, hence the request to simplify the data upload method. This recommendation does not and cannot apply to the points calculation algorithm which we propose remains similar to the present method, i.e. based on the statistics of standardisation.