[Computer-go] A Regression test set for exploring some limitations of current MCTS programs in Go

Rémi Coulom Remi.Coulom at free.fr
Thu May 17 07:09:04 PDT 2012

Sorry, I have just figured out that my loadsgf command did not set the komi correctly. Now that it is fixed, the result is much better:

So only 3 errors at slow time control. I can't tell for sure they are really errors.

The evaluation for ID 111 & 115 is losing. The evaluation for ID 191 is jigo.

All the other (the correct ones) are either jigo or winning, except for case17 (ID 172), that is a very  funny kind of seki, and that is evaluated as bad for W, although CS plays the game correctly.


On 17 mai 2012, at 13:41, Rémi Coulom wrote:

> Hi Aja,
> Thanks for this interesting test. This is Crazy Stone's output for seki_moves:
> http://www.grappa.univ-lille3.fr/~coulom/seki-128k.html
> Many correct answers are probably a bit lucky, because the evaluation is rarely correct.
> It is not easy for me to implement sg_compare_float. But I'll try if I find time.
> gnugo rules :-)
> Rémi
> On 15 mai 2012, at 04:39, Aja Huang wrote:
>> Dear all,
>> Martin Mueller and I are writing a paper about exploring some limitations of current MCTS programs in Go. For this purpose we have carefully designed a regression test set which consists of 20 seki and 15 two-safe-groups cases on 9x9 board. If you are interested, it is available at
>> http://webdocs.cs.ualberta.ca/~mmueller/ps/seki-and-two-safe-groups-regression-test.zip
>> We will appreciate if you would like to run your program over our regression test and send us the results for our publication. 
>> It's easy to run your program through these positions (.sgf). Mainly, the script run.sh under /utility is able to run a given program for a given regression test file (.tst) and produce the result in a related html file. For example, for the seki test you can simply type
>> ./run.sh -p PATH_TO_PROGRAM -t g_seki_moves.tst
>> Some notes:
>> 1. Your program must support the command sg_compare_float for the two-safe-groups test. If it doesn't support reg_genmove then the test file g_seki_moves.tst is good to use which performs genmove instead.
>> 2. On Windows platform, you will be able to execute 'run.sh' directly at the command prompt after cygwin is installed.
>> 3. If your program doesn't support the GTP command 'loadsgf', gogui-adapter is able to translate 'loadsgf' into a sequence of 'play' commands. The file gogui-adapter.jar under /utility is good to use because Markus has fixed some bugs for us, see
>> https://sourceforge.net/tracker/?func=detail&aid=3522401&group_id=59117&atid=489964
>> https://sourceforge.net/tracker/?func=detail&aid=3519829&group_id=59117&atid=489964
>> Under /experimental results, there are results of several programs such as Fuego (tilburg version), pachi, ManyFaces and GnuGo. We thank David for providing us the valuable results of ManyFaces. The test set is really not easy because these programs all failed in many cases.
>> Questions are very welcome. If you find any error in the test set please inform us. Thanks.
>> Best regards,
>> Aja
>> _______________________________________________
>> Computer-go mailing list
>> Computer-go at dvandva.org
>> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
> _______________________________________________
> Computer-go mailing list
> Computer-go at dvandva.org
> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go

More information about the Computer-go mailing list