[Computer-go] Practical significance?

Don Dailey dailey.don at gmail.com
Mon Nov 26 05:20:44 PST 2012


On Mon, Nov 26, 2012 at 12:58 AM, Darren Cook <darren at dcook.org> wrote:

> > better than another with 95% of confidence. However, my adviser is asking
> > me about not only the STATISTICAL significance of the results, but also
> the
> > PRACTICAL significance of them. I mean, if one system is, for example
> only
> > 1% better than another, with 99% of confidence, the result would have a
> > statistical significance, but wouldn't really matter in a practical
> sense.
>
> It all comes down to defining "practical significance", which is
> subjective; but still a useful line of thought.
>
> I have thought of this before too.   ELO is probably the right way and I
have always used 200 ELO (in my mind) to define what I consider a
"signficant superiority."   In chess a "class" is 200 ELO wide and
represents a "clear" superiority so it's almost like a standard anyway.
 In statistical terms a 200 ELO  superiority give you about 76% chance of
winning.      Another way to look at this is that with 200 ELO you might
not win every game, but you are quite likely to win  even a short match.

Of course it's arbitrary how you define a clear superiority.    From
another practical perspective I have noticed that it takes well over 100
ELO for most people to be able to judge that one player is better than
another simply from watching it play.    In fact people are so subjective
and so influenced by style that they will often judge the weaker player to
be the stronger one if there is something about how it plays that appeals
to them.

Another way to see this is to estimate how many games is required before
you can measure a given superiority with close to statistical certainty,
and for 200 ELO it does not take very many games.

Don









> I started thinking a definition could be: in any *short* series of games
> (e.g. best of 5) you'd be confident it would come out ahead. That would
> be my definition of clear superiority. 1% better isn't enough, as you
> need hundreds of games to be sure of seeing a difference.
>
> When you said a "23% difference" did you mean the win-ratio is
> 61.5:38.5? If so, what is the probability the stronger player will get
> 3+ wins from 5 games? If over 0.9 I think you can make the case that the
> strength difference is of practical significance.
>
> Or, yes, do it all in ELO ratings, and decide on how many ELO points
> feels significant.
>
> Darren
>
>
> --
> Darren Cook, Software Researcher/Developer
>
> http://dcook.org/work/ (About me and my work)
> http://dcook.org/blogs.html (My blogs and articles)
> _______________________________________________
> Computer-go mailing list
> Computer-go at dvandva.org
> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://computer-go.org/pipermail/computer-go/attachments/20121126/7116b504/attachment.html>


More information about the Computer-go mailing list