[computer-go] Re: ZG1-trunMCx-50k

[Łukasz Lew]

On 9/24/06, Don Dailey <drd at mit.edu> wrote:
> I don't understand why you expect them to have the same exact rating?
>
> ELO is a statistical rating system.   There is only a 44 rating point
> difference in the worst case, which is hardly significant.   44 rating
> points isn't nearly enough to say with serious confidence that you are a
> better player.

ELO system, to make calculation of rating simple, is making some
simplifications, which
make an explicit evaluation of standard error (intuitively
proportional to rating oscillation)
harder (hard?) (for given K, and with assumption that enough time was given).

This is easier for systems with more solid theoretical foundations.
For instance Bayesian TrueSkill, this parameter is explicitly maintained.

It is natural to evaluate standard error empirically.
So I set 4 programs, until they got to the lowest K possible.(couldn't
set more due to resource constraints).

3 (4 including one run earlier) programs ended with a rating very
close to each other,
which is strong evidence that standard error is really small.

One instance was far away, so maybe there was some kind of "thick error" or
"systematical error". Nothing came to my mind except high standard error.
So I asked.


>
> You also put up 4 players.  I don't understand why you chose 4, but if
> you put up enough players, you are going to find more fluctuations
> between the best and worst.

I put 4 due to limited resources.

>
> All 4 of your players are close to what they should be.   You obviously
> need to get a better understanding of probability and statistics and you
> would realize this isn't strange.


Łukasz Lew



>
> - Don
>
>
> On Sun, 2006-09-24 at 11:43 +0200, Łukasz Lew wrote:
> > Some time ago I put 4 identical bots on CGOS. Now they got stable rank:
> > ZG1-trunMC1-50k	1519*
> > ZG1-trunMC2-50k	1527*
> > ZG1-trunMC3-50k	1529*
> > ZG1-trunMC4-50k	1574*	
> >
> > They are identiacl to bot that played earlier:
> > ZG1-trunMC-100k	1530*
> >
> > I would be very happy to have this precision of strength measurement,
> > but ZG1-trunMC4-50k has oddly high rating.
> >
> > I double checked the binary and parameters of all the bots and they
> > are truly the same.
> >
> > So what may be the reason of this?
> >
> >
> > Lukasz Lew
> >
> >
> > On 9/21/06, Łukasz Lew <lukasz.lew at gmail.com> wrote:
> > > All CGOS bots
> > > ZG1-trunMCx-50k ( x = 1,2,3,4 )
> > > are the same program as
> > > ZG1-trunMC-100k
> > > (also play 100k games not 50k - my mistake)
> > >
> > > It is an experiment to check drift and accuracy of CGOS rating.
> > >
> > > Best regards,
> > > Lukasz
> > >
> > > PS
> > > Is there a possibility to extend the limit on bot name's length?
> > >
> > _______________________________________________
> > computer-go mailing list
> > computer-go at computer-go.org
> > http://www.computer-go.org/mailman/listinfo/computer-go/
>
> _______________________________________________
> computer-go mailing list
> computer-go at computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>