[Computer-go] A Regression test set for exploring some limitations of current MCTS programs in Go

Rémi Coulom Remi.Coulom at free.fr
Thu May 17 04:41:27 PDT 2012


Hi Aja,

Thanks for this interesting test. This is Crazy Stone's output for seki_moves:
http://www.grappa.univ-lille3.fr/~coulom/seki-128k.html
Many correct answers are probably a bit lucky, because the evaluation is rarely correct.

It is not easy for me to implement sg_compare_float. But I'll try if I find time.

gnugo rules :-)

Rémi

On 15 mai 2012, at 04:39, Aja Huang wrote:

> Dear all,
> 
> Martin Mueller and I are writing a paper about exploring some limitations of current MCTS programs in Go. For this purpose we have carefully designed a regression test set which consists of 20 seki and 15 two-safe-groups cases on 9x9 board. If you are interested, it is available at
> 
> http://webdocs.cs.ualberta.ca/~mmueller/ps/seki-and-two-safe-groups-regression-test.zip
> 
> We will appreciate if you would like to run your program over our regression test and send us the results for our publication. 
> 
> It's easy to run your program through these positions (.sgf). Mainly, the script run.sh under /utility is able to run a given program for a given regression test file (.tst) and produce the result in a related html file. For example, for the seki test you can simply type
> 
> ./run.sh -p PATH_TO_PROGRAM -t g_seki_moves.tst
> 
> Some notes:
> 1. Your program must support the command sg_compare_float for the two-safe-groups test. If it doesn't support reg_genmove then the test file g_seki_moves.tst is good to use which performs genmove instead.
> 
> 2. On Windows platform, you will be able to execute 'run.sh' directly at the command prompt after cygwin is installed.
> 
> 3. If your program doesn't support the GTP command 'loadsgf', gogui-adapter is able to translate 'loadsgf' into a sequence of 'play' commands. The file gogui-adapter.jar under /utility is good to use because Markus has fixed some bugs for us, see
> 
> https://sourceforge.net/tracker/?func=detail&aid=3522401&group_id=59117&atid=489964
> https://sourceforge.net/tracker/?func=detail&aid=3519829&group_id=59117&atid=489964
> 
> Under /experimental results, there are results of several programs such as Fuego (tilburg version), pachi, ManyFaces and GnuGo. We thank David for providing us the valuable results of ManyFaces. The test set is really not easy because these programs all failed in many cases.
> 
> Questions are very welcome. If you find any error in the test set please inform us. Thanks.
> 
> Best regards,
> Aja
> _______________________________________________
> Computer-go mailing list
> Computer-go at dvandva.org
> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go




More information about the Computer-go mailing list