This post is a little different, in that it has nothing to do with Data Science or Web Services - still involves Python, though!

This project involved building a Pythonic wrapper around the Stockfish Universal Chess Interface (UCI) engine, implemented in C++. To accomplish that task, thw wonderful cppyy was used.

Requirements

The requirements came from a client on Upwork and were nailed down after a longish series of message exchanges. The basic requirement was to provide a Stockfish Python API that would

(1) Allow for the setting of a board position

(2) Generate a list of all legal moves given the set position

(3) Display the list of all legal moves

Also, based on the client’s previous attempts with other developers, the API needed to be

(1) Performant - One of the earlier attempts was unusably slow

(2) Free of memory leaks - Another attempt was reasonably performant but had a bad memory leak and made his machine with 64GB of RAM to crash after a million invocations.

Python Wrapper

There were several steps involved in building the Python wrapper and the following subsections provide details for each of them, All the work was done in the .../src directory of the Stockfish repo. The wrapper is implemented in the file stockfish.py

Preparatory work - the mise en place

Rhe preparatory work for this project involved 2 pieces of work.

1.Forking the offiicial Stockfish repo

A fork Stockfish-python was created from the official-stockfish repo.

2.Installing cppyy

Detailed information on installing cppyy is available in the cppyy Installation Page The cppyy package(version 2.4.0) was installed on my Fedora release 34 (Thirty Four) VM using the pip install --user cppyy command.

Exposing symbol declaearions to cppyy

This task is accomplished by using the include function from the cppyy package.

In this case

cppyy.include('./stockfish.h')

Where stockfish.h contains

#include <iostream>

#include "bitboard.h"
#include "endgame.h"
#include "position.h"
#include "psqt.h"
#include "search.h"
#include "syzygy/tbprobe.h"
#include "thread.h"
#include "tt.h"
#include "uci.h"

Exposing sumbol definitions to cppyy

This task is accomplished by using the load_library function from the cppyy package.

cppyy.load_library('./libstockfish.so')

The Makefile was modified as follows to allow for the generation of the libstockfish.so DSO.

diff --git a/src/Makefile b/src/Makefile
index ff2452d6..02fad01b 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -40,10 +40,13 @@ endif
 ### Executable name
 ifeq ($(target_windows),yes)
 	EXE = stockfish.exe
+	DSO = stockfish.dll
 else
 	EXE = stockfish
+	DSO = libstockfish.so
 endif
 
+
 ### Installation dir definitions
 PREFIX = /usr/local
 BINDIR = $(PREFIX)/bin
@@ -56,11 +59,14 @@ else
 endif
 
 ### Source and object files
-SRCS = benchmark.cpp bitbase.cpp bitboard.cpp endgame.cpp evaluate.cpp main.cpp \
+LIBSRCS = benchmark.cpp bitbase.cpp bitboard.cpp endgame.cpp evaluate.cpp \
 	material.cpp misc.cpp movegen.cpp movepick.cpp pawns.cpp position.cpp psqt.cpp \
 	search.cpp thread.cpp timeman.cpp tt.cpp uci.cpp ucioption.cpp tune.cpp syzygy/tbprobe.cpp \
 	nnue/evaluate_nnue.cpp nnue/features/half_ka_v2_hm.cpp
 
+SRCS = $(LIBSRCS) main.cpp
+
+LIBOBJS = $(notdir $(LIBSRCS:.cpp=.o))
 OBJS = $(notdir $(SRCS:.cpp=.o))
 
 VPATH = syzygy:nnue:nnue/features
@@ -362,7 +368,7 @@ endif
 ifeq ($(COMP),gcc)
 	comp=gcc
 	CXX=g++
-	CXXFLAGS += -pedantic -Wextra -Wshadow
+	CXXFLAGS += -pedantic -Wextra -Wshadow -fPIC
 
 	ifeq ($(arch),$(filter $(arch),armv7 armv8))
 		ifeq ($(OS),Android)
@@ -868,7 +874,7 @@ default:
 ### Section 5. Private Targets
 ### ==========================================================================
 
-all: $(EXE) .depend
+all: $(EXE) $(DSO) .depend
 
 config-sanity: net
 	@echo ""
@@ -926,6 +932,13 @@ config-sanity: net
 	@test "$(comp)" = "gcc" || test "$(comp)" = "icc" || test "$(comp)" = "mingw" || test "$(comp)" = "clang" \
 	|| test "$(comp)" = "armv7a-linux-androideabi16-clang"  || test "$(comp)" = "aarch64-linux-android21-clang"
 
+
+libstockfish.so: $(LIBOBJS)
+	$(CXX) -o $@ -shared -Wl,-flto -Wl,-soname,$@ $(OBJS) $(LDFLAGS)
+
+stockfish.dll:$(LIBOBJS)
+	$(CXX) -o $@ -shared -Wl,--out-implib,libstockfish.a -Wl,-no-undefined -Wl,--enable-runtime-pseudo-reloc
+
 $(EXE): $(OBJS)
 	+$(CXX) -o $@ $(OBJS) $(LDFLAGS)

Defining python stmbols for the Stockfish Namespaces, Classes and Objects required

The various C++ symbols are now available through the gbl object from the cppyy package.

The various namespaces defined in the Stockfish code are first imported into Python

#Namespaces
StockfishNS = cppyy.gbl.Stockfish
UCI = StockfishNS.UCI
PSQT = StockfishNS.PSQT
Bitboards = StockfishNS.Bitboards
Bitbases = StockfishNS.Bitbases
Endgames = StockfishNS.Endgames
Search = StockfishNS.Search
Eval = StockfishNS.Eval
NNUE = Eval.NNUE

This is followd by importing the classes required

#Classes
Tune = StockfishNS.Tune
Position = StockfishNS.Position
MoveList_LEGAL = StockfishNS.MoveList_LEGAL

Finally, the objects required are imported

#Objects
Options = StockfishNS.Options
Threads = StockfishNS.Threads

The Options object has the following default values for the various options

#Clear Hash 0
#Debug Log File 0
#EvalFile 0
#Hash 16
#Move Overhead 10
#MultiPV 1
#nodestime 0
#Ponder 0
#Skill Level 20
#Slow Mover 100
#Syzygy50MoveRule 1
#SyzygyPath 0
#SyzygyProbeDepth 1
#SyzygyProbeLimit 7
#Threads 1
#UCI_AnalyseMode 0
#UCI_Chess960 0
#UCI_Elo 1350
#UCI_LimitStrength 0
#UCI_ShowWDL 0
#Use NNUE 1
#LEGAL Iteration 0

The Python Stockfish API

The Python Stockfish API is expressed via 2 classes - the main Stockfish class and the auxiliary Moves iterator.

The Stockfish class

The major functionality of this class is performed by 3 methods

1. __init__

This method initializes the stockfish engine and instantiates a Position object that is stored in the pos attribute.

    def __init__(self):
        UCI.init(Options)
        Tune.init()
        PSQT.init()
        Bitboards.init()
        Position.init()
        Bitbases.init()
        Endgames.init()
        # The default value for the 'Threads' option is 1.
        Threads.set(int(Options['Threads'].__float__()))
        Search.clear()
        NNUE.init()

        self.pos = Position()
2. position

This method sets the pos attribute.

    def position(self, position_str):
        UCI.set_position(self.pos, position_str)

The set_position function definition was added to the UCI namespace in uci.cpp

diff --git a/src/uci.cpp b/src/uci.cpp
index c0bacfaf..d0c7d364 100644
--- a/src/uci.cpp
+++ b/src/uci.cpp
@@ -221,6 +221,13 @@ namespace {
 
 } // namespace
 
+void UCI::set_position(Position& pos, const string& strpos) {
+  static StateListPtr states;
+  
+  states = StateListPtr(new std::deque<StateInfo>(1)); // Drop old and create a new one
+  pos.set(strpos, Options["UCI_Chess960"], &states->back(), Threads.main());
+}
+
 
 /// UCI::loop() waits for a command from the stdin, parses it and then calls the appropriate
 /// function. It also intercepts an end-of-file (EOF) indication from the stdin to ensure a

and the declaration to uci.h

diff --git a/src/uci.h b/src/uci.h
index 76a893f9..d5c7f532 100644
--- a/src/uci.h
+++ b/src/uci.h
@@ -76,6 +76,7 @@ std::string pv(const Position& pos, Depth depth, Value alpha, Value beta);
 std::string wdl(Value v, int ply);
 Move to_move(const Position& pos, std::string& str);
 
+void set_position(Position& pos, const std::string& strpos);
 } // namespace UCI
 
 extern UCI::OptionsMap Options;

This method instantiates a MoveList_LEGAL object to generate the list of legal moves possible for the position set in pos, wraps an iterator object around it using Moves and returns it.

    def legal_moves(self):
        return Moves(MoveList_LEGAL(self.pos))


diff --git a/src/movegen.h b/src/movegen.h
index bbb35b39..276e40c4 100644
--- a/src/movegen.h
+++ b/src/movegen.h
@@ -63,6 +63,7 @@ struct MoveList {
   explicit MoveList(const Position& pos) : last(generate<T>(pos, moveList)) {}
   const ExtMove* begin() const { return moveList; }
   const ExtMove* end() const { return last; }
+  const ExtMove* item(int i) {return moveList+i;}
   size_t size() const { return last - moveList; }
   bool contains(Move move) const {
     return std::find(begin(), end(), move) != end();
@@ -72,6 +73,7 @@ private:
   ExtMove moveList[MAX_MOVES], *last;
 };
 
+using MoveList_LEGAL = MoveList<LEGAL>;
 } // namespace Stockfish
 
 #endif // #ifndef MOVEGEN_H_INCLUDED

In the C++ code, MoveList is a template struct and MoveLisr_LEGAL is just MoveList<LEGAL> - ideally, we should have been able to define the MoveList_LEGAL symbol in python using cppyy syntax as

...
LEGAL = StockfishNS.GenType.LEGAL
...
MoveList_LEGAL = StockfishNS.MoveList(LEGAL)
...

Unfortunately, this fails with the following message

TypeError: 'Stockfish::MoveList<LEGAL>' is not a known C++ class

To work around ths issue, the MoveList_LEGAL specialized type is defined in the C++ code as

using MoveList_LEGAL = MoveList<LEGAL>;

and used in the python code via

MoveList_LEGAL = StockfishNS.MoveList_LEGAL

The Moves iterator

The Moves iterator uses the MoveList object’s size method

 size_t size() const { return last - moveList; }

to determine the length of the iterable

    def __iter__(self):
        self.idx = -1
        self.end = self.movelist.size()
        return self

and uses the item method(added to the fork)

+  const ExtMove* item(int i) {return moveList+i;}

to access the item at the specified index

   def __next__(self):
        self.idx += 1
        if self.idx not in range(self.end):
            raise StopIteration

        return self.movelist.item(self.idx)

Sample usage and test script - move_generator.py

Here’s the output from a run of the script

Clear Hash 0
Debug Log File 0
EvalFile 0
Hash 16
Move Overhead 10
MultiPV 1
nodestime 0
Ponder 0
Skill Level 20
Slow Mover 100
Syzygy50MoveRule 1
SyzygyPath 0
SyzygyProbeDepth 1
SyzygyProbeLimit 7
Threads 1
UCI_AnalyseMode 0
UCI_Chess960 0
UCI_Elo 1350
UCI_LimitStrength 0
UCI_ShowWDL 0
Use NNUE 1
LEGAL Iteration 0
LEGAL : [b'a2a3', b'b2b3', b'c2c3', b'd2d3', b'e2e3', b'f2f3', b'g2g3', b'h2h3', b'a2a4', b'b2b4', b'c2c4', b'd2d4', b'e2e4', b'f2f4', b'g2g4', b'h2h4', b'b1a3', b'b1c3', b'g1f3', b'g1h3']
...
LEGAL Iteration 999999
LEGAL : [b'a2a3', b'b2b3', b'c2c3', b'd2d3', b'e2e3', b'f2f3', b'g2g3', b'h2h3', b'a2a4', b'b2b4', b'c2c4', b'd2d4', b'e2e4', b'f2f4', b'g2g4', b'h2h4', b'b1a3', b'b1c3', b'g1f3', b'g1h3']
List of legal moves for position(rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1) : [b'a2a3', b'b2b3', b'c2c3', b'd2d3', b'e2e3', b'f2f3', b'g2g3', b'h2h3', b'a2a4', b'b2b4', b'c2c4', b'd2d4', b'e2e4', b'f2f4', b'g2g4', b'h2h4', b'b1a3', b'b1c3', b'g1f3', b'g1h3']
Time taken for 1000000 iterations: 0:00:46.035899
Change in process memory usage: 389120

The script essentially

1. Instantiates a Stockfish object

...
from stockfish import UCI, Stockfish
...
s = Stockfish()

2. Stores the start timestamp and process resident memory

from datetime import datetime
...
from os import getpid
from psutil import Process

process = Process(getpid())
...
start = datetime.now()
start_mem = process.memory_info().rss

3. Loop a million times

Loop a million times over invocations of the Stockfish API - setting the board position and generating the list of legal moves.

LOOP_COUNT = 1000000
POSITION = 'rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1'

...
for i in range(LOOP_COUNT):
    print(f'LEGAL Iteration {i}')
    s.position(POSITION)

    #for m in s.legal_moves():
    #    print(UCI.move(m.move, s.is_chess960()), end=' ')

    print(f'LEGAL : {s.legal_moves_str()}')

4. Compute and print time and memory data

end = datetime.now()
end_mem = process.memory_info().rss
...
print(f'Time taken for {LOOP_COUNT} iterations: {end-start}')
print(f'Change in process memory usage: {end_mem-start_mem}')