I would still like to see the official analysis for this course.
It appears that the points awarded to the winner were too high, but therein lies the problem, with the algorithm coming up with a score of 1130 for a winning time that was good but certainly not exceptional. Had the points been , for example, 1030, which is much more in line with what I would have expected, she would not have been classified as an outlier, and maybe some of the outliers at the low end would also have escaped their fate.
I know the original points were too high, as before they were totally removed by the new calculations using the updated algorithm, I had been awarded a score of 1040 or thereabouts, which was about 80 points more than any of my previous scores - and I had a somewhat less then perfect run, making approximately 3- 4 minutes of mistakes.
So the question remains - why did the algorithm get it so wrong and deprive us all of our points?
BOF Updated Ranking Algorithm - effect on older vets.
Moderators: [nope] cartel, team nopesport
Re: BOF Updated Ranking Algorithm - effect on older vets.
xxx wrote:So the question remains - why did the algorithm get it so wrong and deprive us all of our points?
That's actually two questions...
It got it so wrong because there isn't enough reliable data to do a statistical analysis.
No points were awarded because it realised how bad the data was.
I would still like to see the official analysis for this course.
So would I... Even the development team has no access to see the data or the code. I think DJM wrote to Mike Hamilton to ask. I think the formula on the website is before the outliers are removed, it would be particularly interesting to see how the scores would have looked on the second pass.
WOC2024 Edinburgh
Test races at SprintScotland (Alloa/Falkirk) and Euromeeting (near Stirling).
Test races at SprintScotland (Alloa/Falkirk) and Euromeeting (near Stirling).
-
graeme - god
- Posts: 4726
- Joined: Thu Feb 12, 2004 6:04 pm
- Location: struggling with an pɹɐɔ ʇıɯǝ
Re: BOF Updated Ranking Algorithm - effect on older vets.
I've just looked at my scores over the past year and they fluctuate between 850 and 1100 depending on whether I've run too fast for my brain to keep up and seriously messed up on one or two controls. Does this mean all my scores would be deemed anomalous? I'd love to be more consistent but I just haven't mastered it yet.
-
Miner - white
- Posts: 77
- Joined: Mon Oct 12, 2009 3:33 pm
- Location: In the pits
Re: BOF Updated Ranking Algorithm - effect on older vets.
I've just looked at my scores over the past year and they fluctuate between 850 and 1100 depending on whether I've run too fast for my brain to keep up and seriously messed up on one or two controls. Does this mean all my scores would be deemed anomalous?
No, it doesn't! Taking the current situation, your average score is probably around 1000 (I can't tell unless I have your complete portfolio of points).
Let's say it's exactly 1000. Then, only if you score 1100 or more or 900 or fewer in the next race will your score be treated as an outlier by the algorithm and will not be included in the points calculation algorithm. However, as graeme has been correctly stressing, you will still be awarded points (assuming the minimum of 8 criterion is met).
Going back in time, the variation in your points suggests that, on at least one occasion, your result will have been treated as an outlier by the algorithm. But you will still have got the correct points for your race time ...
- DJM
- diehard
- Posts: 979
- Joined: Sun Sep 30, 2007 8:19 pm
- Location: Wye Valley
Re: BOF Updated Ranking Algorithm - effect on older vets.
DJM wrote:
Going back in time, the variation in your points suggests that, on at least one occasion, your result will have been treated as an outlier by the algorithm. But you will still have got the correct points for your race time ...
I think the concern that a lot of people are having is precisely this, that the "anomalous" score will be much more frequent than is being suggested and when you add in the vagaries of say "less consistent" orienteers you'll end up with more and more cases where multiple anomalous scores mean noone gets points.
Please dont underestimate the negative impact this could have on the regular club orienteer.
Orienteering - its no walk in the park
- andypat
- god
- Posts: 2856
- Joined: Sat Apr 11, 2009 9:58 pm
- Location: Houston, we have a problem.
Re: BOF Updated Ranking Algorithm - effect on older vets.
xxx wrote: why did the algorithm get it so wrong ...?
I don't think it did "get it wrong" - it did what it was supposed to do with the data it was given. (However the outcome illustrates a vulnerability and potentially a flaw in the implementation). AFAICS the primary anomaly is that an M55 (Dave Ryder) with a previous average of 1188 points produced a performance worth 863 points - at least 20 minutes slower than might've been expected. Given that an M55 would normally be on course 3 not course 5 and he isn't on the pre-entry list (http://www.fabian4.co.uk/start/list.aspx?EventID=1814) the first task is probably to check that this isn't a competitor ID issue.
Why does this matter? Because (crudely) on any given course the algorithm uses the mean of runners' previous scores to work out the mean score for that course (so e.g. the average score on Men's Open is going to be much higher than on WSV) and it uses the standard deviation (a measure of how spread out the numbers are) of previous scores (and the SD of times achieved on the day) to calculate how widely to distribute points - on any given course the higher the standard deviation of runners' previous scores, the greater the spread of ranking points awarded. (Generally this is sensible - at an event with just one course for all ages and abilities it's reasonable to expect a wider range of performances than say, the Men's Elite final at the British sprints.)
In the formula shown on the results page:
824.06 = mean of runners' previous scores
186.18 = SD of runners' previous scores
3259.85 = mean time on the day (in seconds)
672.99 = SD of times on the day
T = time on the day of competitor
so achieving the mean time of 3260 seconds (54.20) would score 824 points (the mean of runners' previous scores) whist finishing in 2587 seconds (43.07) i.e. 1 SD (673 seconds) better than the mean time would score 1010 points (mean of runners' previous scores plus 186).
What's happened in this case is that the inclusion in the calculation of a competitor with a mean previous score of 1188 in a field where most of the others were clustered around 800 - 900 has greatly increased the value of "standard deviation of runner's previous scores" cf what it would have been otherwise - the "186" number in the equation. This is a relatively high figure and has the effect of ensuring that there will be a wide spread of points awarded: e.g. if the SD of runners' previous scores had been 120 then Heather S (whose performance was 1.645 sd better than the mean time) would have scored 824+120*1.645 points = 1021 rather than 824+186*1.645 =1130)
Had Mr Ryder performed as expected (<30 minutes) then the algorithm would have delivered "sensible" scores. If he had done exactly 30 mins then (the mean and SD times on the day would've have changed and) he would've got 1149 points, Heather S 1064, Mary Nixon 969, Denise Harper 963 etc, none of which would've been judged anomalous).
Two implementation issues:
it would be preferable for the algorithm to work iteratively - identify the largest anomaly, remove it, and repeat - unfortunately as discussed previously this is apparently beyond the capacity of the developers
also I noted when doing the above calculations that it appears that the final finisher(s) are no longer being excluded as outliers - not sure this a great idea.
-
greywolf - addict
- Posts: 1416
- Joined: Sat Apr 07, 2007 12:45 pm
- Location: far far away
Re: BOF Updated Ranking Algorithm - effect on older vets.
greywolf wrote: it would be preferable for the algorithm to work iteratively
Yes, and that would have solved the current problem.
100 points slower than gg wrote:I don't think it did "get it wrong"
Indeed, the algorithm flagged Heather's run as an anomaly. LesS confirmed that this was true - she's just back running properly and it could be wrong to compare fit-Heather with injured-Heather.
100 points slower than gg wrote:
the final finisher(s) are no longer being excluded as outliers - not sure this a great idea.
That's right - they're only excluded if they had a disastrous run by their own standards.
And it is a good idea for several reasons.
1/ More data is generally better: including the last-3 finishers makes it more likely the required 8 sensible score
2/ It give a direct connection between high-ranked and low-ranked competitors.
3/ Being slow is no reason to exclude people. If someone runs as slowly as expected, it helps to include them.
4/ Notwithstanding #3, the final finishers are more like to have had a poor run than a good one. So the scores would be based on people having above-average runs: a systematic bias which, over time, led to the well-known problem of scores drifting downwards and older results being worth more than recent ones*.
WOC2024 Edinburgh
Test races at SprintScotland (Alloa/Falkirk) and Euromeeting (near Stirling).
Test races at SprintScotland (Alloa/Falkirk) and Euromeeting (near Stirling).
-
graeme - god
- Posts: 4726
- Joined: Thu Feb 12, 2004 6:04 pm
- Location: struggling with an pɹɐɔ ʇıɯǝ
Re: BOF Updated Ranking Algorithm - effect on older vets.
graeme wrote:100 points slower than gg wrote:I don't think it did "get it wrong"
Indeed, the algorithm flagged Heather's run as an anomaly. LesS confirmed that this was true - she's just back running properly and it could be wrong to compare fit-Heather with injured-Heather.
I'm still not getting why Heather running better than expected shouldnt have an adverse effect on my ranking points if she beat me?
Orienteering - its no walk in the park
- andypat
- god
- Posts: 2856
- Joined: Sat Apr 11, 2009 9:58 pm
- Location: Houston, we have a problem.
Re: BOF Updated Ranking Algorithm - effect on older vets.
graeme wrote:"100 points slower than gg"
An excellent illustration of how big a difference 100 points covers!
graeme wrote:Indeed, the algorithm flagged Heather's run as an anomaly. LesS confirmed that this was true - she's just back running properly and it could be wrong to compare fit-Heather with injured-Heather.
Except that, whilst it may have been a "better than recent" run, it might not've been flagged as anomalous without the outliers in runners' previous scores which cause the algorithm to exaggerate the spread of points.
-
greywolf - addict
- Posts: 1416
- Joined: Sat Apr 07, 2007 12:45 pm
- Location: far far away
Re: BOF Updated Ranking Algorithm - effect on older vets.
So it wasn't Heather's fault - it was Dave's! Another possibility about Dave on this course was what we've all done - coming back from injury or illness, we have a walk round a short course and this clearly has a big effect on low-competitor-numbers courses.
Can someone confirm - the Rules for the Ranking Scheme on the BOF website say 'Effective Jan 2014' - and given Greywolf's helpful analysis, it appears that the formula for calculating the points has remained the same but its implementation has been improved but has not yet been reflected in the Rules?
Can someone confirm - the Rules for the Ranking Scheme on the BOF website say 'Effective Jan 2014' - and given Greywolf's helpful analysis, it appears that the formula for calculating the points has remained the same but its implementation has been improved but has not yet been reflected in the Rules?
- LesS
- off string
- Posts: 27
- Joined: Tue Mar 24, 2009 3:49 pm
Re: BOF Updated Ranking Algorithm - effect on older vets.
LesS wrote: the Rules for the Ranking Scheme on the BOF website say 'Effective Jan 2014'
That is at least two iterations out of date. The final 2016 sent for implementation was
1. General information
1.1 Purpose
1.1.1 The aim of the British Orienteering Ranking Scheme is to rank regular orienteers in order of orienteering ability. The Ranking list may also be used for International selection purposes as well as for seeding competitors at Competitions.
2. Eligibility
2.1.1 All Events at levels A, B and C are part of the ranking scheme except for Relay, score, chasing start and other mass start events.
2.1.2 All British Orienteering members who gain points at ranked events will have their points stored in one ranking list.
2.1.3 Only competitors who are members of British Orienteering and will be 16 or over on 31st December of the year of the event will have their results included in the ranking calculations and score ranking points at Ranking events.
2.1.4 Membership numbers are used to link results from different events for a given runner. Only runners who provide their British Orienteering Membership number at the time of entry will have their results carried forward for ranking.
2.1.5 The total of a runner’s best six scores over the past 12 months determines a runner’s ranking.
3. Administration
3.1.1 The ranking list is accessible from the British Orienteering website and may be filtered.
3.1.2 The system is written in such a way that it can accept results data from both age based courses (e.g. Area Championships) and also “ageless” courses, e.g. colour coded courses and Urban series courses and produce valid rankings points.
3.1.3 When a competitor is to be ranked as a member of a new club or under a change of name, the member must inform British Orienteering so that the membership records can be updated. This will then update the information in the ranking lists.
4. Submission of results
4.1.1 Final results must be submitted electronically to the results page in the British Orienteering website within 7 days of the event. Results not uploaded within this period may be excluded from rankings calculations.
4.1.2 Rankings points will then be calculated and displayed automatically.
4.1.3 Results must be submitted in the British Orienteering file format (see ’User Guide’ available on the British Orienteering website to registered Results Secretaries).
4.1.4 Organisers must carry forward competitors’ British Orienteering membership numbers with their results and must submit them as soon as possible after the event.
4.1.5 Other mandatory data is defined within the ‘User Guide’ available on the British Orienteering website.
5. Calculation of ranking points
5.1 Background
5.1.1 The full specification of the algorithm used to maintain the ranking list is given in a separate document.
5.1.2 All competitive runners at all qualifying events are awarded points on a scale in which 1000 represents the average performance of the average orienteer, with a standard deviation of 200 points.
5.1.3 Although all results may contribute to individual's ranking, the algorithm is devised to automatically exclude abnormal runs from the background statistical analysis.
5.1.4 If a course, after outliers (as defined by 5.2.3) have been discarded, has fewer than 8 counting runners remaining, it is deemed to be statistically unreliable and is not ranked.
5.2 Definitions
5.2.1 Scores: Each runner has a set of “scores”, one for each event that they complete. These are published in the event results and do not change subsequently.
5.2.2 Ranking scores: Each runner has a “ranking”, which is the sum of their best six scores in the previous 12-month period. This is published on the website, but is not used in any calculation.
5.2.3 Outlier: Outliers are regarded as performances that are inappropriate to include in the calculation of Ranking points and may arise for several reasons. These might include exceptional results, incorrect results, results attributed to the wrong BOF member etc. An “outlier” is any score which is more than 100 different from a runner’s current score, and is ignored in all calculations.
5.2.4 Current scores: Each runner has a current score (CS), which is the average of all their scores excluding “outliers”. It is the best predictor of their performance at an event in which they are included in the statistics. Current scores are not published. They are continually rebased to ensure that the mean of all current scores is 1000, and the standard deviation is 200. Only runners with non-zero current scores are used to determine new scores.
5.3 Calculation of scores for a course
5.3.1 For each qualifying course, at each ranking event, the following quantities are required:
SP is the standard deviation of current points of the non-outlier runners
MP is the mean of current points of the non-outlier runners
ST is the standard deviation of times of the non-outlier runners
MT is the mean time of the non-outlier runners
5.3.2 Each runner’s points are then given by MP+(SP(MT-RT)/ST
The only known implementation error is in 5.2.4, where the outliers are included in people's CS.
WOC2024 Edinburgh
Test races at SprintScotland (Alloa/Falkirk) and Euromeeting (near Stirling).
Test races at SprintScotland (Alloa/Falkirk) and Euromeeting (near Stirling).
-
graeme - god
- Posts: 4726
- Joined: Thu Feb 12, 2004 6:04 pm
- Location: struggling with an pɹɐɔ ʇıɯǝ
Re: BOF Updated Ranking Algorithm - effect on older vets.
LesS wrote:So it wasn't Heather's fault - it was Dave's! Another possibility about Dave on this course was what we've all done - coming back from injury or illness, we have a walk round a short course and this clearly has a big effect on low-competitor-numbers courses.
............
Yes, Dave was "running" down. A look at his splits shows that he "walked", or jogged
very slowly, through the first 2 controls and then started to jog a bit more quickly.
The splits suggest that he never actually got into a full run at any time over the course.
His "run", thus, grossly inflated everyone elses ranking scores and as a result of the system not performing multiple iterations he alone became "responsible" for the number of "outliers" getting to 6, or more. There were 15 finishers,of which 1 was non-BO and 1 did not have any current scores leaving 13 valid runners (as far as calculations are concerned).
Dave's "score" was around 300 points below his average.
- MIE
- green
- Posts: 370
- Joined: Fri Jan 11, 2008 10:05 pm
Re: BOF Updated Ranking Algorithm - effect on older vets.
MIE wrote:His "run", thus, grossly inflated everyone elses ranking scores and as a result of the system not performing multiple iterations he alone became "responsible" for the number of "outliers" getting to 6, or more.
Not quite: the effect is to spread out the points distribution - those at the front have their scores greatly increased, those at the back have their scores reduced. In the table below are the points as (not) awarded, and my estimate of what they would have been if say Dave Ryder had mp'd
Heather Smithard 1130 1044
Mary Nixon 1020 955
Denise Harper 1014 950
Sue Hands 979 922
Denise Mullins 976 919
Dave Ryder 863
Michael Gammon 802 779
Deborah Sullivan 757 742
Peter Bennett 745 732
Dianne O'Donoghue 726 717
Susan Parker 679 679
Yvonne Hodson 676 676
Ruth Rhodes 617 628
Ursula Oxburgh 455 497
I think there are still 3 competitors with scores more than 100 from their average, but that wouldn't have been enough to remove all the scores from the course.
NB I say "estimate" and "think" because I don't have access to the full event files, so the second column is probably not quite right.
Two things I noted: those WUV seem to get to a lot of events, and even those at the front of the field tend to have a much greater range of scores than younger runners...
And one more thing: if anyone from AIRE is reading, soemone from your club needs to remove the combined results of the Fat Rascal prologue & chase from the rankings
-
greywolf - addict
- Posts: 1416
- Joined: Sat Apr 07, 2007 12:45 pm
- Location: far far away
Re: BOF Updated Ranking Algorithm - effect on older vets.
Thanks to Greywolf for the detailed explanations, analysis and now concrete calculations showing what scores we might have expected had the points been awarded.
Yes, we WUVs do get around, and our results do vary, but we know roughly what we would expect one another to score. Thus it seems bizarre that one 'outsider' running down can have such a profound effect, causing such an upset. As I said in a previous post, our times were generally not anomalous, thus making it hard to understand why no points were awarded.
Is there no way the system can identify then remove the score of the initial outlier which has then caused so many of the 'normal' scores to become outliers and thus be discarded? (if I have understood the process correctly)
Yes, we WUVs do get around, and our results do vary, but we know roughly what we would expect one another to score. Thus it seems bizarre that one 'outsider' running down can have such a profound effect, causing such an upset. As I said in a previous post, our times were generally not anomalous, thus making it hard to understand why no points were awarded.
Is there no way the system can identify then remove the score of the initial outlier which has then caused so many of the 'normal' scores to become outliers and thus be discarded? (if I have understood the process correctly)
- xxx
- off string
- Posts: 26
- Joined: Tue Jan 03, 2006 9:38 pm
Re: BOF Updated Ranking Algorithm - effect on older vets.
Apparently notxxx wrote:.. Is there no way the system can identify then remove the score of the initial outlier which has then caused so many of the 'normal' scores to become outliers and thus be discarded?
Perhaps the more worrying aspect should be that this result "almost" got through? It looks as though people in the bottom half of the results generally have potential scores that will be close to their average. If there had been say 2-3 more competitors on the course, finishing roughly as expected with mid-course results, there might have been at least the 8 competitors within 100 points of their average. Then points would have been awarded, but probably still with a significant distortion at the front end (even if not quite as extreme as the figures above, given that the extra 2-3 would narrow the SD).greywolf wrote:... it would be preferable for the algorithm to work iteratively - identify the largest anomaly, remove it, and repeat - unfortunately as discussed previously this is apparently beyond the capacity of the developers
- Snail
- diehard
- Posts: 709
- Joined: Wed Nov 30, 2005 8:37 pm
Who is online
Users browsing this forum: No registered users and 161 guests