Answers: <p>As far as I am aware, <code>commit</code> and <code>run</code> are the best options here to give you full access to the container as it was when it died.</p>
  
  <p>Ideally, your container would spit out some more useful information when it fails, but that is another topic altogether.</p>
  
  <p>Edit: to expand my answer, if the container is dying right on start, you can also use <code>docker run</code> to specify an alternative <code>--entrypoint</code> and <code>CMD</code>. Generally I'll set this to a loop or something that won't exit on its own. Once you're in the container, you can manually run the steps that are failing, and then inspect the outcome without having to worry about the container exiting.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-09T02:49:33.433Z
Answers: 2017-03-11T16:46:04.713Z
Answers: 2017-03-11T16:46:04.713Z
Answers: tayworm
Answers: <p>General ways to track why a process in Linux failed are good. One such way is to run a process using <a href="http://man7.org/linux/man-pages/man1/strace.1.html" rel="nofollow noreferrer"><code>strace</code></a> which will tell you the system calls process did and usually point to the reason for a failure.</p>
  
  <p>You can create a <code>Dockerfile</code> that looks something like this:</p>
  
  <pre><code>FROM original_image
  
  RUN apt-get -y update &amp;&amp; apt-get install -y strace
  
  # build with `docker build -t debug_version`
  </code></pre>
  
  <p>Then run your new image using <code>docker run debug_version strace original_cmd</code>.</p>
  
  <p>For processes that fork children (and then die) you want to run <code>strace</code> with the <code>-ff</code> option. You can also map some file using <a href="https://docs.docker.com/engine/tutorials/dockervolumes/" rel="nofollow noreferrer">Docker data volumes</a> and use the <code>-o</code> option from <code>strace</code> to write to it. But in general <code>strace</code> will leave output on stdout, which is readable using <code>docker log</code>.</p>
  
  <p>Related <a href="https://serverfault.com/questions/656760/linux-process-terminates-mysteriously">Q: Linux process terminates mysteriously</a></p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-07T15:20:09.213Z
Answers: This means I still have to `docker commit` my stopped container first to have an image to start from
Answers: SztupY
Answers: CC BY-SA 3.0
Answers: 2017-03-07T15:23:26.047Z
Answers: You said it dies on start. I assume you have an image then. For those that are stopped, yes a commit is required.
Answers: Evgeny Zislis
Answers: CC BY-SA 3.0
Answers: 2017-03-07T15:25:09.273Z
Answers: That's only one of the scenarios to get a stopped container
Answers: SztupY
Answers: CC BY-SA 3.0
Answers: 2017-03-08T10:43:48.347Z
Answers: There is also a package for `strace` in Alpine Linux, https://pkgs.alpinelinux.org/package/edge/main/x86_64/strace. Use the Alpine package manager to install it, `apk install strace`.
Answers: Evgeny Zislis
Answers: CC BY-SA 3.0
Answers: 2017-03-07T15:05:58.583Z
Answers: 2017-03-07T15:12:14.087Z
Answers: 2017-04-13T12:14:55.217Z
Answers: Evgeny Zislis
Body: <p>Sometimes you have to investigate a container, which is stopped, or a container which after starting up dies very quickly and stops.</p>
  
  <p><code>docker exec -ti &lt;id&gt; bash</code> only works on running containers, once it finishes, the bash prompt terminates as well.</p>
  
  <p>With <code>docker start</code> you cannot supply a different command, and if the container dies abruptly again you won't have enough time to get into the container and do your investigations.</p>
  
  <p>We can do <code>docker commit</code>, then <code>docker run</code> on the new image with a different command, but I'm wondering if there are any other alternatives.</p>
  
  <p><strong>Note</strong>: <code>docker logs</code> just returns whatever the apps printed to stdout/stderr. That might not be enough to figure out what the problem was.</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-07T22:15:28.143Z
Comments: After a while thinking about it: Docker **main** process ???? As a container aim at running only one process , either  the term 'main' is to be removed, or you're doing something weird (like running an init process) , or you're taking threads as processes... I assume it's option one, but I had to say it because it bothers me
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-08T09:04:44.713Z
Comments: @Tensibai sometimes you have to run something like dumb-init, to handle the pid 1 / signalling issue in containers, if your main command cannot handle it itself. There might be also other cases where a docker container runs more than one processes
Comments: SztupY
Comments: CC BY-SA 3.0
Comments: 2017-03-08T10:23:01.977Z
Comments: Yes, that's what I call weird, mainly because containers have been made to isolate a process. Sometimes containers are not the solution for an app an willing to put everything inside a container is more a path to headaches than anything else.
Comments: Tensibai
Tags: <docker><containers><post-mortem>
Title: How to investigate a main process that has died in a docker container?
UserDisplayName: SztupY
ThreadText: # How to investigate a main process that has died in a docker container?
  
  (Asked by: username_0 on 2017-03-07)
  
  Sometimes you have to investigate a container, which is stopped, or a container which after starting up dies very quickly and stops.
  
  `docker exec -ti <id> bash` only works on running containers, once it finishes, the bash prompt terminates as well.
  
  With `docker start` you cannot supply a different command, and if the container dies abruptly again you won't have enough time to get into the container and do your investigations.
  
  We can do `docker commit`, then `docker run` on the new image with a different command, but I'm wondering if there are any other alternatives.
  
  **Note**: `docker logs` just returns whatever the apps printed to stdout/stderr. That might not be enough to figure out what the problem was.
  
  **Tags:** docker, containers, post-mortem
  
  > **Comments:**
  > **username_1**: After a while thinking about it: Docker **main** process ???? As a container aim at running only one process , either  the term 'main' is to be removed, or you're doing something weird (like running an init process) , or you're taking threads as processes... I assume it's option one, but I had to say it because it bothers me  
  > **username_0**: @username_1 sometimes you have to run something like dumb-init, to handle the pid 1 / signalling issue in containers, if your main command cannot handle it itself. There might be also other cases where a docker container runs more than one processes  
  > **username_1**: Yes, that's what I call weird, mainly because containers have been made to isolate a process. Sometimes containers are not the solution for an app an willing to put everything inside a container is more a path to headaches than anything else.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-03-07. Score: 9
  
  General ways to track why a process in Linux failed are good. One such way is to run a process using [`strace`](http://man7.org/linux/man-pages/man1/strace.1.html) which will tell you the system calls process did and usually point to the reason for a failure.
  
  You can create a `Dockerfile` that looks something like this:
  
  ```
  FROM original_image
  
  RUN apt-get -y update && apt-get install -y strace
  
  # build with `docker build -t debug_version`
  
  ```
  
  Then run your new image using `docker run debug_version strace original_cmd`.
  
  For processes that fork children (and then die) you want to run `strace` with the `-ff` option. You can also map some file using [Docker data volumes](https://docs.docker.com/engine/tutorials/dockervolumes/) and use the `-o` option from `strace` to write to it. But in general `strace` will leave output on stdout, which is readable using `docker log`.
  
  Related [Q: Linux process terminates mysteriously](https://serverfault.com/questions/656760/linux-process-terminates-mysteriously)
  
  > **Comments:**
  > **username_0**: This means I still have to `docker commit` my stopped container first to have an image to start from  
  > **username_3**: You said it dies on start. I assume you have an image then. For those that are stopped, yes a commit is required.  
  > **username_0**: That's only one of the scenarios to get a stopped container  
  > **username_3**: There is also a package for `strace` in Alpine Linux, https://pkgs.alpinelinux.org/package/edge/main/x86_64/strace. Use the Alpine package manager to install it, `apk install strace`.  
  
  ---
  
  ### Answer by username_2 on 2017-03-09. Score: 3
  
  As far as I am aware, `commit` and `run` are the best options here to give you full access to the container as it was when it died.
  
  Ideally, your container would spit out some more useful information when it fails, but that is another topic altogether.
  
  Edit: to expand my answer, if the container is dying right on start, you can also use `docker run` to specify an alternative `--entrypoint` and `CMD`. Generally I'll set this to a loop or something that won't exit on its own. Once you're in the container, you can manually run the steps that are failing, and then inspect the outcome without having to worry about the container exiting.
  
  
  ---
  
  