I would still like to see the official analysis for this course.
It appears that the points awarded to the winner were too high, but therein lies the problem, with the algorithm coming up with a score of 1130 for a winning time that was good but certainly not exceptional. Had the points been , for example, 1030, which is much more in line with what I would have expected, she would not have been classified as an outlier, and maybe some of the outliers at the low end would also have escaped their fate.
I know the original points were too high, as before they were totally removed by the new calculations using the updated algorithm, I had been awarded a score of 1040 or thereabouts, which was about 80 points more than any of my previous scores  and I had a somewhat less then perfect run, making approximately 3 4 minutes of mistakes.
So the question remains  why did the algorithm get it so wrong and deprive us all of our points?
BOF Updated Ranking Algorithm  effect on older vets.
Moderators: [nope] cartel, team nopesport
Re: BOF Updated Ranking Algorithm  effect on older vets.
xxx wrote:So the question remains  why did the algorithm get it so wrong and deprive us all of our points?
That's actually two questions...
It got it so wrong because there isn't enough reliable data to do a statistical analysis.
No points were awarded because it realised how bad the data was.
I would still like to see the official analysis for this course.
So would I... Even the development team has no access to see the data or the code. I think DJM wrote to Mike Hamilton to ask. I think the formula on the website is before the outliers are removed, it would be particularly interesting to see how the scores would have looked on the second pass.
Graham & Graeme's Masterplan Adventure Resolution Sprint. Jan 1st. Boness. Resolve to do it now!

graeme  god
 Posts: 4527
 Joined: Thu Feb 12, 2004 6:04 pm
 Location: struggling with an pɹɐɔ ʇıɯǝ
Re: BOF Updated Ranking Algorithm  effect on older vets.
I've just looked at my scores over the past year and they fluctuate between 850 and 1100 depending on whether I've run too fast for my brain to keep up and seriously messed up on one or two controls. Does this mean all my scores would be deemed anomalous? I'd love to be more consistent but I just haven't mastered it yet.

Miner  white
 Posts: 77
 Joined: Mon Oct 12, 2009 3:33 pm
 Location: In the pits
Re: BOF Updated Ranking Algorithm  effect on older vets.
I've just looked at my scores over the past year and they fluctuate between 850 and 1100 depending on whether I've run too fast for my brain to keep up and seriously messed up on one or two controls. Does this mean all my scores would be deemed anomalous?
No, it doesn't! Taking the current situation, your average score is probably around 1000 (I can't tell unless I have your complete portfolio of points).
Let's say it's exactly 1000. Then, only if you score 1100 or more or 900 or fewer in the next race will your score be treated as an outlier by the algorithm and will not be included in the points calculation algorithm. However, as graeme has been correctly stressing, you will still be awarded points (assuming the minimum of 8 criterion is met).
Going back in time, the variation in your points suggests that, on at least one occasion, your result will have been treated as an outlier by the algorithm. But you will still have got the correct points for your race time ...
 DJM
 diehard
 Posts: 896
 Joined: Sun Sep 30, 2007 8:19 pm
 Location: Wye Valley
Re: BOF Updated Ranking Algorithm  effect on older vets.
DJM wrote:
Going back in time, the variation in your points suggests that, on at least one occasion, your result will have been treated as an outlier by the algorithm. But you will still have got the correct points for your race time ...
I think the concern that a lot of people are having is precisely this, that the "anomalous" score will be much more frequent than is being suggested and when you add in the vagaries of say "less consistent" orienteers you'll end up with more and more cases where multiple anomalous scores mean noone gets points.
Please dont underestimate the negative impact this could have on the regular club orienteer.
Orienteering  its no walk in the park
 andypat
 god
 Posts: 2853
 Joined: Sat Apr 11, 2009 9:58 pm
 Location: Houston, we have a problem.
Re: BOF Updated Ranking Algorithm  effect on older vets.
xxx wrote: why did the algorithm get it so wrong ...?
I don't think it did "get it wrong"  it did what it was supposed to do with the data it was given. (However the outcome illustrates a vulnerability and potentially a flaw in the implementation). AFAICS the primary anomaly is that an M55 (Dave Ryder) with a previous average of 1188 points produced a performance worth 863 points  at least 20 minutes slower than might've been expected. Given that an M55 would normally be on course 3 not course 5 and he isn't on the preentry list (http://www.fabian4.co.uk/start/list.aspx?EventID=1814) the first task is probably to check that this isn't a competitor ID issue.
Why does this matter? Because (crudely) on any given course the algorithm uses the mean of runners' previous scores to work out the mean score for that course (so e.g. the average score on Men's Open is going to be much higher than on WSV) and it uses the standard deviation (a measure of how spread out the numbers are) of previous scores (and the SD of times achieved on the day) to calculate how widely to distribute points  on any given course the higher the standard deviation of runners' previous scores, the greater the spread of ranking points awarded. (Generally this is sensible  at an event with just one course for all ages and abilities it's reasonable to expect a wider range of performances than say, the Men's Elite final at the British sprints.)
In the formula shown on the results page:
824.06 = mean of runners' previous scores
186.18 = SD of runners' previous scores
3259.85 = mean time on the day (in seconds)
672.99 = SD of times on the day
T = time on the day of competitor
so achieving the mean time of 3260 seconds (54.20) would score 824 points (the mean of runners' previous scores) whist finishing in 2587 seconds (43.07) i.e. 1 SD (673 seconds) better than the mean time would score 1010 points (mean of runners' previous scores plus 186).
What's happened in this case is that the inclusion in the calculation of a competitor with a mean previous score of 1188 in a field where most of the others were clustered around 800  900 has greatly increased the value of "standard deviation of runner's previous scores" cf what it would have been otherwise  the "186" number in the equation. This is a relatively high figure and has the effect of ensuring that there will be a wide spread of points awarded: e.g. if the SD of runners' previous scores had been 120 then Heather S (whose performance was 1.645 sd better than the mean time) would have scored 824+120*1.645 points = 1021 rather than 824+186*1.645 =1130)
Had Mr Ryder performed as expected (<30 minutes) then the algorithm would have delivered "sensible" scores. If he had done exactly 30 mins then (the mean and SD times on the day would've have changed and) he would've got 1149 points, Heather S 1064, Mary Nixon 969, Denise Harper 963 etc, none of which would've been judged anomalous).
Two implementation issues:
it would be preferable for the algorithm to work iteratively  identify the largest anomaly, remove it, and repeat  unfortunately as discussed previously this is apparently beyond the capacity of the developers
also I noted when doing the above calculations that it appears that the final finisher(s) are no longer being excluded as outliers  not sure this a great idea.

greywolf  addict
 Posts: 1396
 Joined: Sat Apr 07, 2007 12:45 pm
 Location: far far away
Re: BOF Updated Ranking Algorithm  effect on older vets.
greywolf wrote: it would be preferable for the algorithm to work iteratively
Yes, and that would have solved the current problem.
100 points slower than gg wrote:I don't think it did "get it wrong"
Indeed, the algorithm flagged Heather's run as an anomaly. LesS confirmed that this was true  she's just back running properly and it could be wrong to compare fitHeather with injuredHeather.
100 points slower than gg wrote:
the final finisher(s) are no longer being excluded as outliers  not sure this a great idea.
That's right  they're only excluded if they had a disastrous run by their own standards.
And it is a good idea for several reasons.
1/ More data is generally better: including the last3 finishers makes it more likely the required 8 sensible score
2/ It give a direct connection between highranked and lowranked competitors.
3/ Being slow is no reason to exclude people. If someone runs as slowly as expected, it helps to include them.
4/ Notwithstanding #3, the final finishers are more like to have had a poor run than a good one. So the scores would be based on people having aboveaverage runs: a systematic bias which, over time, led to the wellknown problem of scores drifting downwards and older results being worth more than recent ones*.
Graham & Graeme's Masterplan Adventure Resolution Sprint. Jan 1st. Boness. Resolve to do it now!

graeme  god
 Posts: 4527
 Joined: Thu Feb 12, 2004 6:04 pm
 Location: struggling with an pɹɐɔ ʇıɯǝ
Re: BOF Updated Ranking Algorithm  effect on older vets.
graeme wrote:100 points slower than gg wrote:I don't think it did "get it wrong"
Indeed, the algorithm flagged Heather's run as an anomaly. LesS confirmed that this was true  she's just back running properly and it could be wrong to compare fitHeather with injuredHeather.
I'm still not getting why Heather running better than expected shouldnt have an adverse effect on my ranking points if she beat me?
Orienteering  its no walk in the park
 andypat
 god
 Posts: 2853
 Joined: Sat Apr 11, 2009 9:58 pm
 Location: Houston, we have a problem.
Re: BOF Updated Ranking Algorithm  effect on older vets.
graeme wrote:"100 points slower than gg"
An excellent illustration of how big a difference 100 points covers!
graeme wrote:Indeed, the algorithm flagged Heather's run as an anomaly. LesS confirmed that this was true  she's just back running properly and it could be wrong to compare fitHeather with injuredHeather.
Except that, whilst it may have been a "better than recent" run, it might not've been flagged as anomalous without the outliers in runners' previous scores which cause the algorithm to exaggerate the spread of points.

greywolf  addict
 Posts: 1396
 Joined: Sat Apr 07, 2007 12:45 pm
 Location: far far away
Re: BOF Updated Ranking Algorithm  effect on older vets.
So it wasn't Heather's fault  it was Dave's! Another possibility about Dave on this course was what we've all done  coming back from injury or illness, we have a walk round a short course and this clearly has a big effect on lowcompetitornumbers courses.
Can someone confirm  the Rules for the Ranking Scheme on the BOF website say 'Effective Jan 2014'  and given Greywolf's helpful analysis, it appears that the formula for calculating the points has remained the same but its implementation has been improved but has not yet been reflected in the Rules?
Can someone confirm  the Rules for the Ranking Scheme on the BOF website say 'Effective Jan 2014'  and given Greywolf's helpful analysis, it appears that the formula for calculating the points has remained the same but its implementation has been improved but has not yet been reflected in the Rules?
 LesS
 off string
 Posts: 27
 Joined: Tue Mar 24, 2009 3:49 pm
Re: BOF Updated Ranking Algorithm  effect on older vets.
LesS wrote: the Rules for the Ranking Scheme on the BOF website say 'Effective Jan 2014'
That is at least two iterations out of date. The final 2016 sent for implementation was
1. General information
1.1 Purpose
1.1.1 The aim of the British Orienteering Ranking Scheme is to rank regular orienteers in order of orienteering ability. The Ranking list may also be used for International selection purposes as well as for seeding competitors at Competitions.
2. Eligibility
2.1.1 All Events at levels A, B and C are part of the ranking scheme except for Relay, score, chasing start and other mass start events.
2.1.2 All British Orienteering members who gain points at ranked events will have their points stored in one ranking list.
2.1.3 Only competitors who are members of British Orienteering and will be 16 or over on 31st December of the year of the event will have their results included in the ranking calculations and score ranking points at Ranking events.
2.1.4 Membership numbers are used to link results from different events for a given runner. Only runners who provide their British Orienteering Membership number at the time of entry will have their results carried forward for ranking.
2.1.5 The total of a runner’s best six scores over the past 12 months determines a runner’s ranking.
3. Administration
3.1.1 The ranking list is accessible from the British Orienteering website and may be filtered.
3.1.2 The system is written in such a way that it can accept results data from both age based courses (e.g. Area Championships) and also “ageless” courses, e.g. colour coded courses and Urban series courses and produce valid rankings points.
3.1.3 When a competitor is to be ranked as a member of a new club or under a change of name, the member must inform British Orienteering so that the membership records can be updated. This will then update the information in the ranking lists.
4. Submission of results
4.1.1 Final results must be submitted electronically to the results page in the British Orienteering website within 7 days of the event. Results not uploaded within this period may be excluded from rankings calculations.
4.1.2 Rankings points will then be calculated and displayed automatically.
4.1.3 Results must be submitted in the British Orienteering file format (see ’User Guide’ available on the British Orienteering website to registered Results Secretaries).
4.1.4 Organisers must carry forward competitors’ British Orienteering membership numbers with their results and must submit them as soon as possible after the event.
4.1.5 Other mandatory data is defined within the ‘User Guide’ available on the British Orienteering website.
5. Calculation of ranking points
5.1 Background
5.1.1 The full specification of the algorithm used to maintain the ranking list is given in a separate document.
5.1.2 All competitive runners at all qualifying events are awarded points on a scale in which 1000 represents the average performance of the average orienteer, with a standard deviation of 200 points.
5.1.3 Although all results may contribute to individual's ranking, the algorithm is devised to automatically exclude abnormal runs from the background statistical analysis.
5.1.4 If a course, after outliers (as defined by 5.2.3) have been discarded, has fewer than 8 counting runners remaining, it is deemed to be statistically unreliable and is not ranked.
5.2 Definitions
5.2.1 Scores: Each runner has a set of “scores”, one for each event that they complete. These are published in the event results and do not change subsequently.
5.2.2 Ranking scores: Each runner has a “ranking”, which is the sum of their best six scores in the previous 12month period. This is published on the website, but is not used in any calculation.
5.2.3 Outlier: Outliers are regarded as performances that are inappropriate to include in the calculation of Ranking points and may arise for several reasons. These might include exceptional results, incorrect results, results attributed to the wrong BOF member etc. An “outlier” is any score which is more than 100 different from a runner’s current score, and is ignored in all calculations.
5.2.4 Current scores: Each runner has a current score (CS), which is the average of all their scores excluding “outliers”. It is the best predictor of their performance at an event in which they are included in the statistics. Current scores are not published. They are continually rebased to ensure that the mean of all current scores is 1000, and the standard deviation is 200. Only runners with nonzero current scores are used to determine new scores.
5.3 Calculation of scores for a course
5.3.1 For each qualifying course, at each ranking event, the following quantities are required:
SP is the standard deviation of current points of the nonoutlier runners
MP is the mean of current points of the nonoutlier runners
ST is the standard deviation of times of the nonoutlier runners
MT is the mean time of the nonoutlier runners
5.3.2 Each runner’s points are then given by MP+(SP(MTRT)/ST
The only known implementation error is in 5.2.4, where the outliers are included in people's CS.
Graham & Graeme's Masterplan Adventure Resolution Sprint. Jan 1st. Boness. Resolve to do it now!

graeme  god
 Posts: 4527
 Joined: Thu Feb 12, 2004 6:04 pm
 Location: struggling with an pɹɐɔ ʇıɯǝ
Re: BOF Updated Ranking Algorithm  effect on older vets.
LesS wrote:So it wasn't Heather's fault  it was Dave's! Another possibility about Dave on this course was what we've all done  coming back from injury or illness, we have a walk round a short course and this clearly has a big effect on lowcompetitornumbers courses.
............
Yes, Dave was "running" down. A look at his splits shows that he "walked", or jogged
very slowly, through the first 2 controls and then started to jog a bit more quickly.
The splits suggest that he never actually got into a full run at any time over the course.
His "run", thus, grossly inflated everyone elses ranking scores and as a result of the system not performing multiple iterations he alone became "responsible" for the number of "outliers" getting to 6, or more. There were 15 finishers,of which 1 was nonBO and 1 did not have any current scores leaving 13 valid runners (as far as calculations are concerned).
Dave's "score" was around 300 points below his average.
 MIE
 green
 Posts: 360
 Joined: Fri Jan 11, 2008 10:05 pm
Re: BOF Updated Ranking Algorithm  effect on older vets.
MIE wrote:His "run", thus, grossly inflated everyone elses ranking scores and as a result of the system not performing multiple iterations he alone became "responsible" for the number of "outliers" getting to 6, or more.
Not quite: the effect is to spread out the points distribution  those at the front have their scores greatly increased, those at the back have their scores reduced. In the table below are the points as (not) awarded, and my estimate of what they would have been if say Dave Ryder had mp'd
Heather Smithard 1130 1044
Mary Nixon 1020 955
Denise Harper 1014 950
Sue Hands 979 922
Denise Mullins 976 919
Dave Ryder 863
Michael Gammon 802 779
Deborah Sullivan 757 742
Peter Bennett 745 732
Dianne O'Donoghue 726 717
Susan Parker 679 679
Yvonne Hodson 676 676
Ruth Rhodes 617 628
Ursula Oxburgh 455 497
I think there are still 3 competitors with scores more than 100 from their average, but that wouldn't have been enough to remove all the scores from the course.
NB I say "estimate" and "think" because I don't have access to the full event files, so the second column is probably not quite right.
Two things I noted: those WUV seem to get to a lot of events, and even those at the front of the field tend to have a much greater range of scores than younger runners...
And one more thing: if anyone from AIRE is reading, soemone from your club needs to remove the combined results of the Fat Rascal prologue & chase from the rankings

greywolf  addict
 Posts: 1396
 Joined: Sat Apr 07, 2007 12:45 pm
 Location: far far away
Re: BOF Updated Ranking Algorithm  effect on older vets.
Thanks to Greywolf for the detailed explanations, analysis and now concrete calculations showing what scores we might have expected had the points been awarded.
Yes, we WUVs do get around, and our results do vary, but we know roughly what we would expect one another to score. Thus it seems bizarre that one 'outsider' running down can have such a profound effect, causing such an upset. As I said in a previous post, our times were generally not anomalous, thus making it hard to understand why no points were awarded.
Is there no way the system can identify then remove the score of the initial outlier which has then caused so many of the 'normal' scores to become outliers and thus be discarded? (if I have understood the process correctly)
Yes, we WUVs do get around, and our results do vary, but we know roughly what we would expect one another to score. Thus it seems bizarre that one 'outsider' running down can have such a profound effect, causing such an upset. As I said in a previous post, our times were generally not anomalous, thus making it hard to understand why no points were awarded.
Is there no way the system can identify then remove the score of the initial outlier which has then caused so many of the 'normal' scores to become outliers and thus be discarded? (if I have understood the process correctly)
 xxx
 off string
 Posts: 25
 Joined: Tue Jan 03, 2006 9:38 pm
Re: BOF Updated Ranking Algorithm  effect on older vets.
Apparently notxxx wrote:.. Is there no way the system can identify then remove the score of the initial outlier which has then caused so many of the 'normal' scores to become outliers and thus be discarded?
Perhaps the more worrying aspect should be that this result "almost" got through? It looks as though people in the bottom half of the results generally have potential scores that will be close to their average. If there had been say 23 more competitors on the course, finishing roughly as expected with midcourse results, there might have been at least the 8 competitors within 100 points of their average. Then points would have been awarded, but probably still with a significant distortion at the front end (even if not quite as extreme as the figures above, given that the extra 23 would narrow the SD).greywolf wrote:... it would be preferable for the algorithm to work iteratively  identify the largest anomaly, remove it, and repeat  unfortunately as discussed previously this is apparently beyond the capacity of the developers
 Snail
 brown
 Posts: 561
 Joined: Wed Nov 30, 2005 8:37 pm
Who is online
Users browsing this forum: No registered users and 5 guests