[Computer-go] A Regression test set for exploring some limitations of current MCTS programs in Go

Aja Huang ajahuang at gmail.com
Mon May 14 19:39:31 PDT 2012


Dear all,

Martin Mueller and I are writing a paper about exploring some limitations
of current MCTS programs in Go. For this purpose we have carefully designed
a regression test set which consists of 20 seki and 15 two-safe-groups
cases on 9x9 board. If you are interested, it is available at

http://webdocs.cs.ualberta.ca/~mmueller/ps/seki-and-two-safe-groups-regression-test.zip

We will appreciate if you would like to run your program over our
regression test and send us the results for our publication.

It's easy to run your program through these positions (.sgf). Mainly, the
script run.sh under /utility is able to run a given program for a given
regression test file (.tst) and produce the result in a related html file.
For example, for the seki test you can simply type

./run.sh -p PATH_TO_PROGRAM -t g_seki_moves.tst

Some notes:
1. Your program must support the command sg_compare_float for the
two-safe-groups test. If it doesn't support reg_genmove then the test file
g_seki_moves.tst is good to use which performs genmove instead.

2. On Windows platform, you will be able to execute 'run.sh' directly at
the command prompt after cygwin is installed.

3. If your program doesn't support the GTP command 'loadsgf', gogui-adapter
is able to translate 'loadsgf' into a sequence of 'play' commands. The file
gogui-adapter.jar under /utility is good to use because Markus has fixed
some bugs for us, see

https://sourceforge.net/tracker/?func=detail&aid=3522401&group_id=59117&atid=489964
https://sourceforge.net/tracker/?func=detail&aid=3519829&group_id=59117&atid=489964

Under /experimental results, there are results of several programs such as
Fuego (tilburg version), pachi, ManyFaces and GnuGo. We thank David for
providing us the valuable results of ManyFaces. The test set is really not
easy because these programs all failed in many cases.

Questions are very welcome. If you find any error in the test set please
inform us. Thanks.

Best regards,
Aja
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://computer-go.org/pipermail/computer-go/attachments/20120514/859e8231/attachment.html>


More information about the Computer-go mailing list