[Computer-go] A Regression test set for exploring some limitations of current MCTS programs in Go

Olivier Teytaud olivier.teytaud at lri.fr
Thu May 17 06:21:01 PDT 2012


If you run tests twice, you get nearly the same results ?
Aja: you'll publish results with varying numbers of simulations for MC bots
?
Olivier

2012/5/17 Rémi Coulom <Remi.Coulom at free.fr>

> Hi Aja,
>
> Thanks for this interesting test. This is Crazy Stone's output for
> seki_moves:
> http://www.grappa.univ-lille3.fr/~coulom/seki-128k.html
> Many correct answers are probably a bit lucky, because the evaluation is
> rarely correct.
>
> It is not easy for me to implement sg_compare_float. But I'll try if I
> find time.
>
> gnugo rules :-)
>
> Rémi
>
> On 15 mai 2012, at 04:39, Aja Huang wrote:
>
> > Dear all,
> >
> > Martin Mueller and I are writing a paper about exploring some
> limitations of current MCTS programs in Go. For this purpose we have
> carefully designed a regression test set which consists of 20 seki and 15
> two-safe-groups cases on 9x9 board. If you are interested, it is available
> at
> >
> >
> http://webdocs.cs.ualberta.ca/~mmueller/ps/seki-and-two-safe-groups-regression-test.zip
> >
> > We will appreciate if you would like to run your program over our
> regression test and send us the results for our publication.
> >
> > It's easy to run your program through these positions (.sgf). Mainly,
> the script run.sh under /utility is able to run a given program for a given
> regression test file (.tst) and produce the result in a related html file.
> For example, for the seki test you can simply type
> >
> > ./run.sh -p PATH_TO_PROGRAM -t g_seki_moves.tst
> >
> > Some notes:
> > 1. Your program must support the command sg_compare_float for the
> two-safe-groups test. If it doesn't support reg_genmove then the test file
> g_seki_moves.tst is good to use which performs genmove instead.
> >
> > 2. On Windows platform, you will be able to execute 'run.sh' directly at
> the command prompt after cygwin is installed.
> >
> > 3. If your program doesn't support the GTP command 'loadsgf',
> gogui-adapter is able to translate 'loadsgf' into a sequence of 'play'
> commands. The file gogui-adapter.jar under /utility is good to use because
> Markus has fixed some bugs for us, see
> >
> >
> https://sourceforge.net/tracker/?func=detail&aid=3522401&group_id=59117&atid=489964
> >
> https://sourceforge.net/tracker/?func=detail&aid=3519829&group_id=59117&atid=489964
> >
> > Under /experimental results, there are results of several programs such
> as Fuego (tilburg version), pachi, ManyFaces and GnuGo. We thank David for
> providing us the valuable results of ManyFaces. The test set is really not
> easy because these programs all failed in many cases.
> >
> > Questions are very welcome. If you find any error in the test set please
> inform us. Thanks.
> >
> > Best regards,
> > Aja
> > _______________________________________________
> > Computer-go mailing list
> > Computer-go at dvandva.org
> > http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
>
> _______________________________________________
> Computer-go mailing list
> Computer-go at dvandva.org
> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
>



-- 
=========================================================
Olivier Teytaud -- olivier.teytaud at inria.fr
TAO, LRI, UMR 8623(CNRS - Universite Paris-Sud),
bat 490 Universite Paris-Sud F-91405 Orsay Cedex France http://0z.fr/EJm0g
(one of the 56.5 % of french who did not vote for Sarkozy in 2007)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://computer-go.org/pipermail/computer-go/attachments/20120517/efe40295/attachment.html>


More information about the Computer-go mailing list