The point is that nobody should have to. And it's not about simply running a longer course per se, it's about having to make sure you run the course the highest ranked are on. Often but not always the same thing. At the event we're discussing, you could run slower on Brown than on Black because that's where the competition was.
Personally, I've largely given up on the ranking scheme - I knew for instance that running M65 this weekend would lose me points. But yes, the blatant failings as demoed there in what is meant to be a nationally recognised scheme do piss me off, and yes it is galling seeing others running much slower scoring more simply because they've lighted on the 'right' course... The one saving grace is that it's not really used for anything substantive, but that in itself says so much. And that damn email keeps coming through each week....must get my filters working properly!
Ranking Scheme bias?
Moderators: [nope] cartel, team nopesport
Re: Ranking Scheme bias?
@awk: for someone with no interest in the ranking scheme you seem to spend an inordinate amount of time and effort criticising it .
- MIE
- green
- Posts: 370
- Joined: Fri Jan 11, 2008 10:05 pm
Re: Ranking Scheme bias?
MIE wrote:@awk: for someone with no interest in the ranking scheme you seem to spend an inordinate amount of time and effort criticising it .
I think I've made it clear that I'm very interested in it. I've pretty much given up trying to participate in it (it would be good to be able to withdraw from it), but that doesn't mean I'm not interested in having a scheme that works.
There are a lot of others who would also like that too - it's surprised me how many people have come up to me and said that they agree, but have simply mentally walked away from it because of the issues discussed above. I agree, I probably do spend an inordinate amount of time on it - part of having a rather obsessive nature I suspect - but it remains very much less than I spend on both other aspects of the sport and other activities.
-
awk - god
- Posts: 3224
- Joined: Sun Mar 21, 2004 5:29 pm
- Location: Bradford
Re: Ranking Scheme bias?
FWIW I agree with you awk.
It's totally bonkers that I'm ranked above Alistair Landels (largely based on running Elite at all the major events).
He beat me on all 5 days of the Scottish 6 Days...
It's totally bonkers that I'm ranked above Alistair Landels (largely based on running Elite at all the major events).
He beat me on all 5 days of the Scottish 6 Days...
-
Homer - diehard
- Posts: 969
- Joined: Wed Apr 13, 2005 12:10 pm
- Location: Springfield
Re: Ranking Scheme bias?
Totally agree! Dislike the ranking system and hate the emails.
- JennyJ
- red
- Posts: 158
- Joined: Tue Nov 14, 2006 12:00 pm
- Location: Sheffield
Re: Ranking Scheme bias?
I think the principle / theory behind the ranking list is sound. So if there are anomalies it is probably because there are not enough people running running often enough outside of their 'usual' class to reduce any cross-class discrepancies. If all the say M65s nearly always only run against other M60/65/70s then any anomalies with younger age classes will persist, and take a long time to reduce or be eliminated.
It certainly isn't helped that many (most?) level B/C events are now in some form of regional / urban league where age groups are encouraged to enter a particular course, combined with level As being largely (solely?) age based. Fewer leagues, with everyone choosing a course at level A/B/C events based on there own preference / ability rather than what they think their (age) peers will be running, would reduce any anomalies more quickly.
It certainly isn't helped that many (most?) level B/C events are now in some form of regional / urban league where age groups are encouraged to enter a particular course, combined with level As being largely (solely?) age based. Fewer leagues, with everyone choosing a course at level A/B/C events based on there own preference / ability rather than what they think their (age) peers will be running, would reduce any anomalies more quickly.
- Snail
- diehard
- Posts: 709
- Joined: Wed Nov 30, 2005 8:37 pm
Re: Ranking Scheme bias?
Snail wrote: If all the say M65s nearly always only run against other M60/65/70s then any anomalies with younger age classes will persist, and take a long time to reduce or be eliminated.
The anomalies are not just with younger age classes, they are within age classes as well, and that to my mind is the more significant effect of the failings in the system.
See Homer's example above.
In M65 there is no way (on terrain events) I should be ranked above, for example, John Tullie, Don Petrie, Roger Coombs, Keith Tonkin, Peter Haines and probably others as well. (though I would give myself a chance on urbans/sprints). Evidence for this - S5D, where I was 8th Brit.
The only one of these 4 I run against regularly is Keith, and I would estimate he beats me on 80%+ of terrain events.
Another point I have made previously - the rankings should have a filter on overall vs terrain vs urban. This should be quite straightforward to implement, even without addressing the issues with the calculations.
Like AWK, I do not spend all my days fretting about the rankings, but what is the point in having them at all unless they work and are reasonably accurate ? If something is worth doing, it is worth doing well.
curro ergo sum
-
King Penguin - addict
- Posts: 1453
- Joined: Wed Sep 21, 2005 6:56 pm
- Location: Kendal
Re: Ranking Scheme bias?
JennyJ wrote:Totally agree! Dislike the ranking system and hate the emails.
Totally disagree! I like the ranking system and find the emails useful.
There we go - now it is a proper forum discussion! FWIW I genuinely mean that - not because I am anywhere interesting in the rankings, but because it provides me with a metric to measure myself against. Am I getting better/worse. Did I do better/worse at this year's event. Simply comparing times is impossible as the planners and distances are not consistent enough. Comparing to position isn't that helpful as it depends who turns up / enters that class etc.
I do agree though that Urban/Terrain rankings don't seem to be very comparable. Mathematically I'm not sure I understand why that is - the stats should be sensible regardless of the terrain. I suspect its not actually a terrain (or nav difficulty) directly but a consequence of the issue Awk is highlighting - the mix of competitors on particular courses is different. To me that just reiterates that our random five year and gender split is nonsense. Wouldn't it be better to put people in categories, based on past performance (rankings!) so the people on each course are closer matched in expected result rather than have similar birth certificates? Is winning M65 really so much better than winning "Class 5"? I think the ranking scheme should actually work well if the distribution of people on each course is more closely matched (fewer outliers). My hypothetical Class 5 would probably have a good number of M65s but also some slower M/W50s, and faster M70s. The really fast M65s (who run up just now meaning the winning M65 isn't the fasted 65 yr old man) would be in class 4 (or 3/2 etc) Perhaps controversially that would actually allow men and women to compete together. You could still keep M/W elite or M/W open separate. At all but the very biggest events we don't really have enough people for the range of classes.
Another problem with the rankings is in Scotland there is a lot of orienteering, but for many good but not obsessive orienteers, they aren't BOF members so don't appear on the rankings. If you think are 600th in the country you are probably actually 650th if you add in the missing Scots and perhaps 675th if you include the under 16s who don't count!
- Atomic
- orange
- Posts: 140
- Joined: Thu Nov 25, 2021 11:56 am
Re: Ranking Scheme bias?
King Penguin wrote:...The anomalies are not just with younger age classes, they are within age classes as well, and that to my mind is the more significant effect of the failings in the system.
See Homer's example above.
In M65 there is no way (on terrain events) I should be ranked above, for example, John Tullie, Don Petrie, Roger Coombs, Keith Tonkin, Peter Haines and probably others as well. (though I would give myself a chance on urbans/sprints). Evidence for this - S5D, where I was 8th Brit.
The only one of these 4 I run against regularly is Keith, and I would estimate he beats me on 80%+ ..
But presumably you aren't ranked higher based on the races you run against Keith (and others). I am guessing you have obtained the higher points from races you have run with /against younger classes?
If all M21s points are currently say 10% higher than they should be then clearly a few older competitors running in a race of mainly M21s are going to get a high score. But the way to address that is to encourage more (ranking) events where mixed age classes are choosing to run the same course. In the past there would have been many people running a variety of courses from Yellow up to Black as they either improved or declined. Over time the anomalies would then disappear.
But the current event and league structures are encouraging the opposite. In the example above (which is possibly similar to what is happening in practice) anyone running up will get a high score - but it could easily have been the other way around, where running up would always give you a low score. The more that most people only run against just their age peers in ranking events the longer any anomalies will persist.
- Snail
- diehard
- Posts: 709
- Joined: Wed Nov 30, 2005 8:37 pm
Re: Ranking Scheme bias?
I think that the "holy grail" of a ranking scheme that fairly compares runners on different courses and different types of event is something that can never be achieved.
Some people may perform comparably at urban and terrain events, others not so. For example, I'm useless at running through terrain, so if you look at my rankings on the very few recent terrain events I've done, they're abysmal (836 and 866 points) whereas my urban scores are usually more like 1070 points. There will doubtless be others with far better orienteering skills but poorer running ability than me who will see the opposite. Yes, the basic skills are the same for both types of event, but the balance between them is different. I think you'd see pretty much the same if you compared rankings for, say, 10k road races and cross-country races of a similar distance.
It's much the same comparing people of vastly different level (say a top M21 vs a mediocre M70); they'll never compete directly, so you're relying on a chain of people to relate their scores. Maybe there are some M21s who sometimes compete against M55s, and some M55s who are sometimes on the same course as M70s, or something like that - it's never going to be that precise.
Add to that the problems that any sort of statistical scheme has with small numbers of people, and it's not surprising that, in spite of all the effort put it to make it as good as possible, there will be a lot of complaints.
By contrast, when I started orienteering in the 80s everything was a lot simpler. Most (if not all) ranking events were age-category based, and there were separate ranking lists for each age category, so while you couldn't compare yourself against someone in a different age category, within a single category the rankings were probably quite reasonable. Also, sprint and urban events didn't exist (or if they did, weren't used for ranking) back then, so the list was based on broadly similar types of event. I'm not suggesting that was better or that we should go back to it, but I do think we should manage expectations.
Some people may perform comparably at urban and terrain events, others not so. For example, I'm useless at running through terrain, so if you look at my rankings on the very few recent terrain events I've done, they're abysmal (836 and 866 points) whereas my urban scores are usually more like 1070 points. There will doubtless be others with far better orienteering skills but poorer running ability than me who will see the opposite. Yes, the basic skills are the same for both types of event, but the balance between them is different. I think you'd see pretty much the same if you compared rankings for, say, 10k road races and cross-country races of a similar distance.
It's much the same comparing people of vastly different level (say a top M21 vs a mediocre M70); they'll never compete directly, so you're relying on a chain of people to relate their scores. Maybe there are some M21s who sometimes compete against M55s, and some M55s who are sometimes on the same course as M70s, or something like that - it's never going to be that precise.
Add to that the problems that any sort of statistical scheme has with small numbers of people, and it's not surprising that, in spite of all the effort put it to make it as good as possible, there will be a lot of complaints.
By contrast, when I started orienteering in the 80s everything was a lot simpler. Most (if not all) ranking events were age-category based, and there were separate ranking lists for each age category, so while you couldn't compare yourself against someone in a different age category, within a single category the rankings were probably quite reasonable. Also, sprint and urban events didn't exist (or if they did, weren't used for ranking) back then, so the list was based on broadly similar types of event. I'm not suggesting that was better or that we should go back to it, but I do think we should manage expectations.
- roadrunner
- addict
- Posts: 1059
- Joined: Fri Sep 02, 2005 8:30 pm
Re: Ranking Scheme bias?
awk wrote:Well, well!
I was that winner ...
and by a significant margin.
Thus by definition a statistical outlier, which (as others have pointed out) will always produce unreliable scores using statistical methods. Nothing to do with the system being biased against particular courses or age classes.
Your score in that particular race was produced by extrapolation of a best fit line matching all the competitors on the course - most running very much slower than yourself. The base time for that course was about 67 minutes and half the field were within 10 minutes of that. The line will have produced reliable results within that range, but a slight difference of gradient will have a big lever arm effect for someone beating the base time by 25 minutes.
If you want the ranking system to produce reliable results for you then you need to run courses where you compete against your peers (ie people with similar running speeds to you rather than people of a similar age)
- pete.owens
- diehard
- Posts: 752
- Joined: Tue Jan 15, 2008 12:25 am
Re: Ranking Scheme bias?
So what several abouve are now suggesting is that it's not the ranking scheme that is a problem, but the sport, and the fact that people (want to) compete against people of their own age and gender.
This is total back to front thinking: it may well be true that there is too much age class based competition (I don't find it as such - I just pick and choose which age class competitions I compete in, and otherwise choose courses on other criteria), but a ranking scheme needs to fit the sport, not the sport fit the ranking scheme. "We've got this super-duper statistical formula, and it's so brilliant that the whole sport needs to be changed to make it work properly"
This is total back to front thinking: it may well be true that there is too much age class based competition (I don't find it as such - I just pick and choose which age class competitions I compete in, and otherwise choose courses on other criteria), but a ranking scheme needs to fit the sport, not the sport fit the ranking scheme. "We've got this super-duper statistical formula, and it's so brilliant that the whole sport needs to be changed to make it work properly"
You may well be right. So, perhaps we should have something different.roadrunner wrote:I think that the "holy grail" of a ranking scheme that fairly compares runners on different courses and different types of event is something that can never be achieved.
Thanks for the explanation Pete, but whilst it explains, it, IMO, doesn't justify. And this statement, again IMO, represents that back-to-front thinking I referred to. Given how this ranking scheme is presented, I should be able to run any course and get a 'reliable result'. I should certainly be able to run my age class at an age class based event and get a 'reliable result'. At the White Rose, I was running a course that was a perfectly appropriate middle distance course for someone of my age/ability. ( You are, BTW, doing down several runners on my course who regularly run at similar speeds to me - and faster). I shouldn't need to go hunting down my 'ability peer group' whoever they are (scattered across a range of courses). To repeat, the ranking scheme needs to fit the sport, not the sport to fit the ranking scheme.If it doesn't, and it doesn't, then it's not fit for purpose.pete.owens wrote:If you want the ranking system to produce reliable results for you then you need to run courses where you compete against your peers (ie people with similar running speeds to you rather than people of a similar age)
-
awk - god
- Posts: 3224
- Joined: Sun Mar 21, 2004 5:29 pm
- Location: Bradford
Re: Ranking Scheme bias?
JennyJ wrote:Totally agree! Dislike the ranking system and hate the emails.
I've just found that you can unsubscribe from the latter. Just go to the end of one of the emails, and there's an unsubscribe option. I've just made use of it!
-
awk - god
- Posts: 3224
- Joined: Sun Mar 21, 2004 5:29 pm
- Location: Bradford
Re: Ranking Scheme bias?
awk wrote:Thanks for the explanation Pete, but whilst it explains, it, IMO, doesn't justify. And this statement, again IMO, represents that back-to-front thinking I referred to. Given how this ranking scheme is presented, I should be able to run any course and get a 'reliable result'. I should certainly be able to run my age class at an age class based event and get a 'reliable result'. At the White Rose, I was running a course that was a perfectly appropriate middle distance course for someone of my age/ability. ( You are, BTW, doing down several runners on my course who regularly run at similar speeds to me - and faster). I shouldn't need to go hunting down my 'ability peer group' whoever they are (scattered across a range of courses). To repeat, the ranking scheme needs to fit the sport, not the sport to fit the ranking scheme.If it doesn't, and it doesn't, then it's not fit for purpose.pete.owens wrote:If you want the ranking system to produce reliable results for you then you need to run courses where you compete against your peers (ie people with similar running speeds to you rather than people of a similar age)
What you are demanding mathematically impossible. Any empirical scientific calculation can only work with the data input and there is always a risk of extrapolating any conclusions beyond the range of observed data; conclusions will be increasingly less reliable the further away you are from the main body of data.
In the case of O results, the ranking system can only work on the results and current rankings of those who complete the course. ie the explanation IS the justification. If you are very much faster or very much slower than the main body of the field than you will be a statistical outlier and NO amount of data manipulation can get over the fact that comparable data simply isn't there.
One thing that could improve matters is to reduce the number of courses. As you pointed out, blue and short blue were very similar with a total entry of 93. With any statistical method you get more reliable results from a bigger data set.
- pete.owens
- diehard
- Posts: 752
- Joined: Tue Jan 15, 2008 12:25 am
Re: Ranking Scheme bias?
If there are anomalies, such as younger class scores being too high, then I think there are only two ways of resolving them.
Either mix up age classes across courses a bit more often. Eg age class X sometimes runs with class X+5 / X+10 and sometimes with X-5 / X-10. Over time (and it could be a long time) the anomalies would gradually reduce. One way of doing that might be to remove the suggested class / course combinations from the rules, and just leave the age class ratios so that planners can put different combinations together. (As well as leagues dropping required class / course mixes).
Or make a one-off adjustment to everyone's scores, with the adjustment varying by age class (or age). But that probably needs more evidence on the scale of the anomalies. Eg enough people running out of class sufficiently that they can reliably say: when I run in my own age class I get N points on average, but when I run (say) Black / M21 I always get N+y%.
Either mix up age classes across courses a bit more often. Eg age class X sometimes runs with class X+5 / X+10 and sometimes with X-5 / X-10. Over time (and it could be a long time) the anomalies would gradually reduce. One way of doing that might be to remove the suggested class / course combinations from the rules, and just leave the age class ratios so that planners can put different combinations together. (As well as leagues dropping required class / course mixes).
Or make a one-off adjustment to everyone's scores, with the adjustment varying by age class (or age). But that probably needs more evidence on the scale of the anomalies. Eg enough people running out of class sufficiently that they can reliably say: when I run in my own age class I get N points on average, but when I run (say) Black / M21 I always get N+y%.
- Snail
- diehard
- Posts: 709
- Joined: Wed Nov 30, 2005 8:37 pm
Who is online
Users browsing this forum: No registered users and 16 guests