Answers.Body: <p>not quite the solution I was looking for but everything is working with the following Dockerfile</p>
  <pre><code>FROM nvidia/cuda:10.0-cudnn7-devel
  
  RUN apt-get update &amp;&amp; apt-get dist-upgrade -y &amp;&amp; apt clean all
  RUN apt-get update &amp;&amp; apt-get install -y curl wget supervisor git clang-6.0 ninja-build protobuf-compiler libprotobuf-dev python3-pip &amp;&amp; apt-get clean all
  RUN pip3 install meson
  RUN pip3 install python-chess
  
  RUN mkdir src
  WORKDIR src/
  COPY . .
  RUN pip3 install jupyter
  WORKDIR /src/notebooks
  ENV TINI_VERSION v0.6.0
  ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
  RUN chmod +x /usr/bin/tini
  ENTRYPOINT [&quot;/usr/bin/tini&quot;, &quot;--&quot;]
  
  RUN mkdir /lczero &amp;&amp; \
      cd ~ &amp;&amp; \
      git clone -b release/0.26 --recurse-submodules https://github.com/LeelaChessZero/lc0.git &amp;&amp; \
      cd lc0 &amp;&amp;\
      ./build.sh
  
  RUN cp /root/lc0/build/release/lc0 /lczero &amp;&amp; \
      file /lczero/lc0 &amp;&amp; \
      chmod +x /lczero/lc0
  
  RUN cd /lczero &amp;&amp; \
      wget https://github.com/LeelaChessZero/lczero-client/releases/download/v29/lc0-training-client-linux &amp;&amp; \
      chmod +x lc0-training-client-linux &amp;&amp; \
      wget https://storage.lczero.org/files/networks-contrib/J92-280
  
  RUN git clone https://github.com/official-stockfish/Stockfish.git &amp;&amp;\
      cd Stockfish &amp;&amp;\
      cd src &amp;&amp;\
      make help &amp;&amp;\
      make net &amp;&amp;\
      make build ARCH=x86-64-modern
  
  COPY supervisord.conf /etc/supervisor/conf.d/
  COPY settings.json /lczero/settings.json
  RUN service supervisor start
  
  CMD [&quot;jupyter&quot;, &quot;notebook&quot;, &quot;--port=8888&quot;, &quot;--no-browser&quot;, &quot;--ip=0.0.0.0&quot;, &quot;--allow-root&quot;]
  
  # sudo docker exec -it $(sudo docker run --runtime=nvidia --name leela -it -d -p 8888:8888 leela:latest ) service supervisor start
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-27T22:11:10.980Z
Answers.Text: What was your `build` command?
Answers.UserDisplayName: Mobeus Zoom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-28T23:09:48.480Z
Answers.Text: probably "sudo docker-compose -t leela ." I havnt worked on this project in a few months but managed to get leela and stockfish running in a jupyter notebook which was nice. Ill try to check and update this thread in the near future and please remind me if I forget.
Answers.UserDisplayName: nak3c
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-30T18:28:29.927Z
Answers.Text: Upvoted already. I think this is an important idea, would be nice if you could supply all the surrounding documentation :)
Answers.UserDisplayName: Mobeus Zoom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-01T23:50:34.310Z
Answers.Text: agreed, I took a look at my scripts this afternoon. something broke with the python meson package but I should be able to fix it. Ill set up a public git repo and link here. may take a day or two.
Answers.UserDisplayName: nak3c
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T17:54:47.887Z
Answers.Text: ok, so I created an initial commit here: https://github.com/nkinney06/LeelaAndStockfishInJupyter but a few things broke since I was working on this project so I had to simplify in a few spots. Some of the background jobs are not working like they used to. Nevertheless, this is a start and Ill try to update the project somewhat regularly. You may have trouble on your machine stemming from the supervisord.conf file. If so you need to regenerate with the createSuperuser.py script. Ill have to add some more documentation for portability in the near future.
Answers.UserDisplayName: nak3c
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T18:01:15.310Z
Answers.Text: incidentally I have some other projects at the intersection of chess and data science. See for example, https://www.thechesscite.com/. Let me know if you are interested in this sort of thing and maybe we can collab.
Answers.UserDisplayName: nak3c
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-24T00:03:47.683Z
Answers.LastActivityDate: 2020-10-24T00:03:47.683Z
Answers.UserDisplayName: nak3c
Body: <p>I have access to a powerful computer at work so naturally I want to run a chess engine on it (stockfish for now and later leela). I am trying to use docker-compose to link a stockfish image and a jupyter notebook image; then use the python-chess library to interact with the uci version of the stockfish engine. I am having difficulty linking the images with docker-compose (I am new to docker). Here is what I have tried:</p>
  <p>I set up my directory structure as below:</p>
  <pre><code>├── docker-compose.yml
  ├── jupyter
  │   └── Dockerfile
  └── stockfish
      └── Dockerfile
  </code></pre>
  <p>my Docker file for stockfish is from <a href="https://github.com/jessegersensonchess/stockfish-nnue-docker/blob/master/Dockerfile" rel="nofollow noreferrer">https://github.com/jessegersensonchess/stockfish-nnue-docker/blob/master/Dockerfile</a>:</p>
  <pre><code>FROM ubuntu:20.04 as builder
  RUN apt update &amp;&amp; \
      apt upgrade -y &amp;&amp; \
      apt-get install -y git make g++ wget &amp;&amp; \
      git clone https://github.com/official-stockfish/Stockfish &amp;&amp; \
      cd Stockfish/src &amp;&amp; \
      CXXFLAGS='-march=native' make -j2 profile-build ARCH=x86-64-bmi2
  WORKDIR /Stockfish/src
  FROM ubuntu:20.04
  RUN apt-get update &amp;&amp; apt-get install -y libgomp1 wget &amp;&amp; apt-get autoremove -y &amp;&amp; apt-get autoclean
  COPY --from=0 /Stockfish/src/stockfish /app/
  COPY --from=0 /Stockfish/src/*.nnue /app/
  WORKDIR /app
  CMD [&quot;./stockfish&quot;]
  </code></pre>
  <p>My dockerfile for jupyter:</p>
  <pre><code>FROM jupyter/scipy-notebook
  USER root
  RUN [&quot;bash&quot;,&quot;-c&quot;,&quot;source activate root &amp;&amp; pip install python-chess&quot;]
  USER jovyan
  </code></pre>
  <p>from the stockfish directory I can build the stockfish image and confirm that the uci executable is in the WORKDIR:</p>
  <pre><code>sudo docker build -t stockfish .
  sudo docker run -it stockfish bash
  root@mycontainerid:/app# ls -1
  nn-eba324f53044.nnue
  stockfish    &lt;-- the uci engine executable
  </code></pre>
  <p>from the jupyter directory I can build the jupyter image and confirm that I can import the python-chess library:</p>
  <pre><code>sudo docker build -t jupyter/scipy-notebook .
  sudo docker run -d -P jupyter/scipy-notebook
  in [1]: import chess.engine  &lt;-- the library to connect to uci engines
  </code></pre>
  <p>Now I try to link the images using docker-compose. My yml file look like this:</p>
  <pre><code>version: '3'
  services:
     this_jupyter:
        build: jupyter
        ports:
           - &quot;8888:8888&quot;
        volumes:
           - .:/home/joyvan
     this_stockfish:
        build: stockfish
        volumes:
           - .:/home/joyvan
  </code></pre>
  <p>Ive tried a few variations of the yml file without success:</p>
  <pre><code>sudo docker-compose ps
           Name                      Command              State            Ports
  ---------------------------------------------------------------------------------------
  stockfish_this_jupyter_1     tini -g -- start-notebook.sh   Up       0.0.0.0:8888-&gt;8888/tcp
  stockfish_this_stockfish_1   ./stockfish                    Exit 0
  </code></pre>
  <p>my question is (a) why is the stockfish container not running, and (b) where will the executable be located once I successfully link to the jupyter notebook container?</p>
  <p>UPDATE when I start the containers with docker compose (sudo docker-compose up -d) I get the following log files,</p>
  <p>log file for jupyter:</p>
  <pre><code>sudo docker-compose logs this_jupyter
  Attaching to leela_this_jupyter_1
  this_jupyter_1    | Executing the command: jupyter notebook
  this_jupyter_1    | [I 23:33:10.836 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
  this_jupyter_1    | [I 23:33:11.782 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
  this_jupyter_1    | [I 23:33:11.783 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] Serving notebooks from local directory: /home/jovyan
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] Jupyter Notebook 6.1.4 is running at:
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] http://648f6d3c8eca:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  this_jupyter_1    | [I 23:33:11.791 NotebookApp]  or http://127.0.0.1:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
  this_jupyter_1    | [C 23:33:11.800 NotebookApp]
  this_jupyter_1    |
  this_jupyter_1    |     To access the notebook, open this file in a browser:
  this_jupyter_1    |         file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
  this_jupyter_1    |     Or copy and paste one of these URLs:
  this_jupyter_1    |         http://648f6d3c8eca:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  this_jupyter_1    |      or http://127.0.0.1:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  </code></pre>
  <p>and the log file for stockfish:</p>
  <pre><code>sudo docker-compose logs this_stockfish
  Attaching to leela_this_stockfish_1
  this_stockfish_1  | Stockfish 181020 by the Stockfish developers (see AUTHORS file)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-19T18:20:28.787Z
Comments.Text: Without a log it is difficult to answer your question, run the services with docker compose and then post the log you read running this command: docker-compose logs this_jupyter or docker-compose logs this_stockfish depending which service you want to read logs
Comments.UserDisplayName: g3k0
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-19T23:42:44.097Z
Comments.Text: @Christian_g3k0 appreciate it. I added an update. The log files don't appear to be particularly informative. Docker takes some getting used to.
Comments.UserDisplayName: nak3c
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-20T16:07:50.703Z
Comments.Text: I ran the docker compose locally on my machine and replicated your situation. The stockfish container exit with status code 0 it means there are no errors, I think there is something wrong with the stockfish dockerfile...it is not written to maintain the container running....
Comments.UserDisplayName: g3k0
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-20T23:26:08.170Z
Comments.Text: @Christian_g3k0 much appreciated. I may play around with some other dockerfiles. Im still a bit puzzled that I can run the stockfish image by itselt (sudo docker run -it stockfish bash) and run the stockfish binary in the container's shell (still getting used to the jargon). Ill post updates if I figure anything out. Eventually want to get leela working as well; my work computer has 4x tesla v100's.
Comments.UserDisplayName: nak3c
Tags: <software><stockfish><programming><python>
Title: running stockfish with docker and docker-compose
ThreadText: # running stockfish with docker and docker-compose
  
  (Asked by: username_0 on 2020-10-18)
  
  I have access to a powerful computer at work so naturally I want to run a chess engine on it (stockfish for now and later leela). I am trying to use docker-compose to link a stockfish image and a jupyter notebook image; then use the python-chess library to interact with the uci version of the stockfish engine. I am having difficulty linking the images with docker-compose (I am new to docker). Here is what I have tried:
  
  I set up my directory structure as below:
  
  ```
  ├── docker-compose.yml
  ├── jupyter
  │   └── Dockerfile
  └── stockfish
      └── Dockerfile
  
  ```
  
  my Docker file for stockfish is from <https://github.com/jessegersensonchess/stockfish-nnue-docker/blob/master/Dockerfile>:
  
  ```
  FROM ubuntu:20.04 as builder
  RUN apt update && \
      apt upgrade -y && \
      apt-get install -y git make g++ wget && \
      git clone https://github.com/official-stockfish/Stockfish && \
      cd Stockfish/src && \
      CXXFLAGS='-march=native' make -j2 profile-build ARCH=x86-64-bmi2
  WORKDIR /Stockfish/src
  FROM ubuntu:20.04
  RUN apt-get update && apt-get install -y libgomp1 wget && apt-get autoremove -y && apt-get autoclean
  COPY --from=0 /Stockfish/src/stockfish /app/
  COPY --from=0 /Stockfish/src/*.nnue /app/
  WORKDIR /app
  CMD ["./stockfish"]
  
  ```
  
  My dockerfile for jupyter:
  
  ```
  FROM jupyter/scipy-notebook
  USER root
  RUN ["bash","-c","source activate root && pip install python-chess"]
  USER jovyan
  
  ```
  
  from the stockfish directory I can build the stockfish image and confirm that the uci executable is in the WORKDIR:
  
  ```
  sudo docker build -t stockfish .
  sudo docker run -it stockfish bash
  root@mycontainerid:/app# ls -1
  nn-eba324f53044.nnue
  stockfish    <-- the uci engine executable
  
  ```
  
  from the jupyter directory I can build the jupyter image and confirm that I can import the python-chess library:
  
  ```
  sudo docker build -t jupyter/scipy-notebook .
  sudo docker run -d -P jupyter/scipy-notebook
  in [1]: import chess.engine  <-- the library to connect to uci engines
  
  ```
  
  Now I try to link the images using docker-compose. My yml file look like this:
  
  ```
  version: '3'
  services:
     this_jupyter:
        build: jupyter
        ports:
           - "8888:8888"
        volumes:
           - .:/home/joyvan
     this_stockfish:
        build: stockfish
        volumes:
           - .:/home/joyvan
  
  ```
  
  Ive tried a few variations of the yml file without success:
  
  ```
  sudo docker-compose ps
           Name                      Command              State            Ports
  ---------------------------------------------------------------------------------------
  stockfish_this_jupyter_1     tini -g -- start-notebook.sh   Up       0.0.0.0:8888->8888/tcp
  stockfish_this_stockfish_1   ./stockfish                    Exit 0
  
  ```
  
  my question is (a) why is the stockfish container not running, and (b) where will the executable be located once I successfully link to the jupyter notebook container?
  
  UPDATE when I start the containers with docker compose (sudo docker-compose up -d) I get the following log files,
  
  log file for jupyter:
  
  ```
  sudo docker-compose logs this_jupyter
  Attaching to leela_this_jupyter_1
  this_jupyter_1    | Executing the command: jupyter notebook
  this_jupyter_1    | [I 23:33:10.836 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
  this_jupyter_1    | [I 23:33:11.782 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
  this_jupyter_1    | [I 23:33:11.783 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] Serving notebooks from local directory: /home/jovyan
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] Jupyter Notebook 6.1.4 is running at:
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] http://648f6d3c8eca:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  this_jupyter_1    | [I 23:33:11.791 NotebookApp]  or http://127.0.0.1:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  this_jupyter_1    | [I 23:33:11.791 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
  this_jupyter_1    | [C 23:33:11.800 NotebookApp]
  this_jupyter_1    |
  this_jupyter_1    |     To access the notebook, open this file in a browser:
  this_jupyter_1    |         file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
  this_jupyter_1    |     Or copy and paste one of these URLs:
  this_jupyter_1    |         http://648f6d3c8eca:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  this_jupyter_1    |      or http://127.0.0.1:8888/?token=f2d986295231a8adf0a3b3263d1177512090e2309972ca2e
  
  ```
  
  and the log file for stockfish:
  
  ```
  sudo docker-compose logs this_stockfish
  Attaching to leela_this_stockfish_1
  this_stockfish_1  | Stockfish 181020 by the Stockfish developers (see AUTHORS file)
  
  ```
  
  **Tags:** software, stockfish, programming, python
  
  > **Comments:**
  > **username_1**: Without a log it is difficult to answer your question, run the services with docker compose and then post the log you read running this command: docker-compose logs this_jupyter or docker-compose logs this_stockfish depending which service you want to read logs  
  > **username_0**: @Christian_g3k0 appreciate it. I added an update. The log files don't appear to be particularly informative. Docker takes some getting used to.  
  > **username_1**: I ran the docker compose locally on my machine and replicated your situation. The stockfish container exit with status code 0 it means there are no errors, I think there is something wrong with the stockfish dockerfile...it is not written to maintain the container running....  
  > **username_0**: @Christian_g3k0 much appreciated. I may play around with some other dockerfiles. Im still a bit puzzled that I can run the stockfish image by itselt (sudo docker run -it stockfish bash) and run the stockfish binary in the container's shell (still getting used to the jargon). Ill post updates if I figure anything out. Eventually want to get leela working as well; my work computer has 4x tesla v100's.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-10-24. Score: 2
  
  not quite the solution I was looking for but everything is working with the following Dockerfile
  
  ```
  FROM nvidia/cuda:10.0-cudnn7-devel
  
  RUN apt-get update && apt-get dist-upgrade -y && apt clean all
  RUN apt-get update && apt-get install -y curl wget supervisor git clang-6.0 ninja-build protobuf-compiler libprotobuf-dev python3-pip && apt-get clean all
  RUN pip3 install meson
  RUN pip3 install python-chess
  
  RUN mkdir src
  WORKDIR src/
  COPY . .
  RUN pip3 install jupyter
  WORKDIR /src/notebooks
  ENV TINI_VERSION v0.6.0
  ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
  RUN chmod +x /usr/bin/tini
  ENTRYPOINT ["/usr/bin/tini", "--"]
  
  RUN mkdir /lczero && \
      cd ~ && \
      git clone -b release/0.26 --recurse-submodules https://github.com/LeelaChessZero/lc0.git && \
      cd lc0 &&\
      ./build.sh
  
  RUN cp /root/lc0/build/release/lc0 /lczero && \
      file /lczero/lc0 && \
      chmod +x /lczero/lc0
  
  RUN cd /lczero && \
      wget https://github.com/LeelaChessZero/lczero-client/releases/download/v29/lc0-training-client-linux && \
      chmod +x lc0-training-client-linux && \
      wget https://storage.lczero.org/files/networks-contrib/J92-280
  
  RUN git clone https://github.com/official-stockfish/Stockfish.git &&\
      cd Stockfish &&\
      cd src &&\
      make help &&\
      make net &&\
      make build ARCH=x86-64-modern
  
  COPY supervisord.conf /etc/supervisor/conf.d/
  COPY settings.json /lczero/settings.json
  RUN service supervisor start
  
  CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]
  
  # sudo docker exec -it $(sudo docker run --runtime=nvidia --name leela -it -d -p 8888:8888 leela:latest ) service supervisor start
  
  ```
  
  > **Comments:**
  > **username_2**: What was your `build` command?  
  > **username_0**: probably "sudo docker-compose -t leela ." I havnt worked on this project in a few months but managed to get leela and stockfish running in a jupyter notebook which was nice. Ill try to check and update this thread in the near future and please remind me if I forget.  
  > **username_2**: Upvoted already. I think this is an important idea, would be nice if you could supply all the surrounding documentation :)  
  > **username_0**: agreed, I took a look at my scripts this afternoon. something broke with the python meson package but I should be able to fix it. Ill set up a public git repo and link here. may take a day or two.  
  > **username_0**: ok, so I created an initial commit here: https://github.com/nkinney06/LeelaAndStockfishInJupyter but a few things broke since I was working on this project so I had to simplify in a few spots. Some of the background jobs are not working like they used to. Nevertheless, this is a start and Ill try to update the project somewhat regularly. You may have trouble on your machine stemming from the supervisord.conf file. If so you need to regenerate with the createSuperuser.py script. Ill have to add some more documentation for portability in the near future.  
  > **username_0**: incidentally I have some other projects at the intersection of chess and data science. See for example, https://www.thechesscite.com/. Let me know if you are interested in this sort of thing and maybe we can collab.  
  
  ---
  
  