[computer-go] Re: ZG1-trunMCx-50k

[Łukasz Lew]

I got some private mail that stated that I suggest that there are some flaws in
CGOS rating algorithm or I suggest that there is some panacea that would make
CGOS more accurate.

I would like to say that:

1) I believe that CGOS may be single most beneficial thing for
computer go development in last years.

2) ELO may be close (enough) to optimum in measuring accuracy of go
programs performance. (How close? You may compare ratings evaluated by
bayeselo provided by Aloril. I believe that bayeselo do a slightly
better use of game results.)

3) Several times I mentioned TrueSkill system.
This was not because I believe that It would produce a significantly
more accurate ratings.
(probably fixed point rating algorithms like KGS system and bayeselo
do here a better job)
It was because it provided theoretically justified way to update "K"
constant, just like ELO system provided theoretically justified way to
update rating AND update equations easy enough to implement, so I've
seen no reason to stick to ad hoc methods.

I dare to say it again - it would probably bring no improvement in
accuracy of ratings.
It's just nice to have some good theory supporting. :)

4) I put 4 identical bots on CGOS to check the level of rating
accuracy I may expect
INCLUDING the luck of programs

On 9/25/06, Christoph Birk <birk at ociw.edu> wrote:
> >> ELO is a statistical rating system.   There is only a 44 rating point
> >> difference in the worst case, which is hardly significant.   44 rating
> >> points isn't nearly enough to say with serious confidence that you are a
> >> better player.
>
> I did a Monte-Carlo simulation of the (ELO) rating of identical
> programs with P(win)=0.5. (like the ZG1-trunMCxxx group on CGOS).
>
> After about 600 games the rating spread (best-worst) of the ZG1-group
> is (1505,1526,1530,1547,1566) 51 points.
> The likelyhood for a spread of more than 50 points is about 12% ...
> IMHO not too unlikely.

It's nice experiment, rendering all the processor cycles spent on ZGx bots
unnecessary :)

Maybe we could find this way a best decreasing speed of K for CGOS?

Lukasz Lew

>
> Christoph
> _______________________________________________
> computer-go mailing list
> computer-go at computer-go.org
> http://www.computer-go.org/mailman/listinfo/computer-go/
>