[Computer-go] Practical significance?
3-Hirn-Verlag at gmx.de
Mon Nov 26 01:05:48 PST 2012
One general comment:
Ratings are not transitive. For instance,
A1 may score 25 % against B,
and A2 may score 22 % against B.
Then it can not be concluded that A1 will score more than 50 %
in direct duel with A2.
It is rather easy it construct triples of "semi-simple" agents A, B, C
for some "normal" game where
A score 95+ percent against B,
B scores 95+ percent against C,
C scores 95+ percent against A.
-------- Original-Nachricht --------
> Datum: Sun, 25 Nov 2012 17:03:33 -0800
> Von: Leandro Marcolino <sorianom at usc.edu>
> An: computer-go at dvandva.org
> Betreff: [Computer-go] Practical significance?
> Hello all!..
> I am currently doing a research about Computer Go. I can't tell the
> about it yet, but I will post them here after (if) my paper is accepted...
> In my research I compare many systems (An), playing against a fixed strong
> adversary (B). So A1 would have a percentage of victory x1 against B,
> A2 would have a percentage of victory x2, etc... Then I compare the
> percentage of victories, and for most cases I can show that one system is
> better than another with 95% of confidence. However, my adviser is asking
> me about not only the STATISTICAL significance of the results, but also
> PRACTICAL significance of them. I mean, if one system is, for example only
> 1% better than another, with 99% of confidence, the result would have a
> statistical significance, but wouldn't really matter in a practical sense.
> In my case, the difference between the systems can range from about 4% to
> about 23%. Doesn't seem to be enough to argue that one system would be
> one-handicap stone better than another. But what would be the minimum
> difference for me to argue that one system is significantly better than
> another, in a practical sense? (or they are not, in the end?..) Would
> calculating ELO-ratings help me in answering this question?
> I think it gets even more complex if we think that, let's say, changing
> percentage of victory from 95% to 100% seems to be much more significant
> (in a practical sense) than changing from 30% to 35%, even though the
> difference between the two systems is still only 5%. In my case, I am
> dealing with percentages of victories that range from around 30% to
> What do you guys think?..
> Thanks for your help!..
More information about the Computer-go