Answers: <p>One thing that pops my mind is using either of <a href="https://www.wireshark.org/" rel="nofollow noreferrer">Wireshark</a>, <a href="https://tshark.dev/setup/install/" rel="nofollow noreferrer">tshark</a>, or <a href="https://www.tcpdump.org/" rel="nofollow noreferrer">tcpdump</a> for capturing network packets originating at docker daemon while docker build is running.</p>
  <p>After recording the traffic, need to filter &amp; process the captured data.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-02-05T12:22:33.377Z
Answers: 2023-03-15T18:20:33.243Z
Answers: 2023-03-15T18:20:33.243Z
Answers: rokpoto.com
Answers: <p>You could start with something like this (<code>tcpdump</code>):</p>
  <pre><code>tcpdump -q ip and ip6 and dst port 80 or 443 &gt; tcpdump.log
  </code></pre>
  <p>This should capture all outgoing ipv4 and ipv6 traffic, to destiation ports 80 and 443 (http and https).
  You can add <code>-i eth1</code> to capture data on a specific interface only. You can play with filters.
  You should get something like this (results from curling google and facebook) - notice that you won't get actual <strong>google.com</strong> domain, but the results could still help to see which domains are blocked by your firewall:</p>
  <pre><code>19:07:58.915115 IP pmsbuild02.4752 &gt; bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.915125 IP pmsbuild02.4752 &gt; bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.947639 IP pmsbuild02.4752 &gt; bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.947869 IP pmsbuild02.4752 &gt; bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.948406 IP pmsbuild02.4752 &gt; bud02s38-in-f4.1e100.net.http: tcp 0
  19:08:04.608537 IP pmsbuild02.10146 &gt; edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.610415 IP pmsbuild02.10146 &gt; edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.610494 IP pmsbuild02.10146 &gt; edge-star-mini-shv-02-frt3.facebook.com.http: tcp 80
  19:08:04.656677 IP pmsbuild02.10146 &gt; edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.656887 IP pmsbuild02.10146 &gt; edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.657372 IP pmsbuild02.10146 &gt; edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  </code></pre>
  <p>You can then easily filter out the domains with awk (get the 5th section, delimited by spaces) - note that there's still the http/https appended to the end of the domain, and the colon sign:</p>
  <pre><code>awk '{print $5}' tcpdump.log
  </code></pre>
  <p>Result:</p>
  <pre><code>edge-star-mini-shv-01-vie1.facebook.com.https:
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2023-03-19T18:18:51.603Z
Answers: 2023-03-19T18:18:51.603Z
Answers: GChuf
Answers: <p>Maybe you could use some local redirecting proxy service and patch your <em>localhosts</em> file to get all requests caught there?</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-03-16T16:01:43.893Z
Answers: 2023-03-16T16:01:43.893Z
Answers: Ta Mu
Body: <p>Is there a straight forward way to generate a <code>.txt</code> file containing each URL (or corresponding IP) visited during a docker build process?</p>
  <h3>Example</h3>
  <p>An example list might look like this:</p>
  <pre><code># urls.txt
  https://hub.docker.com/
  https://www.npmjs.com/
  </code></pre>
  <p>(but a real case using an extensive Dockerfile may be more complex and contain many more URLs/IPs).</p>
  <h3>Things I tried so far</h3>
  <p><strong>1. Installing and using an intrusion detection engine</strong></p>
  <p>IDSs I attempted to install and use were feature rich and probably more complex than required for what I'm attempting. Configuring IDS engines is <a href="https://www.youtube.com/watch?v=UXKbh0jPPpg" rel="nofollow noreferrer">not trivial</a> (especially for someone with little network knowledge), so I'd like to explore simpler solutions, if possible.</p>
  <p><strong>2. Collecting and parsing docker build logs</strong></p>
  <p>I captured build logs with <a href="https://forums.docker.com/t/capture-ouput-of-docker-build-into-a-log-file/123178" rel="nofollow noreferrer">this</a>:</p>
  <pre class="lang-none prettyprint-override"><code>docker build --no-cache --progress=plain  . &amp;&gt; build.log
  </code></pre>
  <p>and extracted lines containing a URL with <a href="https://stackoverflow.com/a/6041965/5783745">this</a> regular expression:</p>
  <pre class="lang-none prettyprint-override"><code>(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&amp;:\/~+#-]*[\w@?^=%&amp;\/~+#-])
  </code></pre>
  <p>A lot of URLs are found, but... i) it doesn't find URLs which aren't explicitly stated in the build logs, and ii) it finds URLs which aren't actually accessed but which are simply stated in logs (i.e. logs along the lines of &quot;For more info, visit https://www.example.com&quot;.</p>
  <h3>Question</h3>
  <p>How can I generate a nice clean list of each external URL visited during a <code>docker build</code> process?</p>
  
Comments: CC BY-SA 4.0
Comments: 2023-02-05T02:22:27.590Z
Comments: Note: wasn't sure if this is best here on or on SO. Glad for it to be moved if it's more appropriate elsewhere.
Comments: stevec
Tags: <docker><networking><network>
Title: How to generate a list of every external URL accessed during docker build?
UserDisplayName: stevec
ThreadText: # How to generate a list of every external URL accessed during docker build?
  
  (Asked by: username_0 on 2023-02-05)
  
  Is there a straight forward way to generate a `.txt` file containing each URL (or corresponding IP) visited during a docker build process?
  
  ### Example
  
  An example list might look like this:
  
  ```
  # urls.txt
  https://hub.docker.com/
  https://www.npmjs.com/
  
  ```
  
  (but a real case using an extensive Dockerfile may be more complex and contain many more URLs/IPs).
  
  ### Things I tried so far
  
  **1. Installing and using an intrusion detection engine**
  
  IDSs I attempted to install and use were feature rich and probably more complex than required for what I'm attempting. Configuring IDS engines is [not trivial](https://www.youtube.com/watch?v=UXKbh0jPPpg) (especially for someone with little network knowledge), so I'd like to explore simpler solutions, if possible.
  
  **2. Collecting and parsing docker build logs**
  
  I captured build logs with [this](https://forums.docker.com/t/capture-ouput-of-docker-build-into-a-log-file/123178):
  
  ```none
  docker build --no-cache --progress=plain  . &> build.log
  
  ```
  
  and extracted lines containing a URL with [this](https://stackoverflow.com/a/6041965/5783745) regular expression:
  
  ```none
  (http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])
  
  ```
  
  A lot of URLs are found, but... i) it doesn't find URLs which aren't explicitly stated in the build logs, and ii) it finds URLs which aren't actually accessed but which are simply stated in logs (i.e. logs along the lines of "For more info, visit https://www.example.com".
  
  ### Question
  
  How can I generate a nice clean list of each external URL visited during a `docker build` process?
  
  **Tags:** docker, networking, network
  
  > **Comments:**
  > **username_0**: Note: wasn't sure if this is best here on or on SO. Glad for it to be moved if it's more appropriate elsewhere.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-02-05. Score: 3
  
  One thing that pops my mind is using either of [Wireshark](https://www.wireshark.org/), [tshark](https://tshark.dev/setup/install/), or [tcpdump](https://www.tcpdump.org/) for capturing network packets originating at docker daemon while docker build is running.
  
  After recording the traffic, need to filter & process the captured data.
  
  
  ---
  
  ### Answer by username_2 on 2023-03-19. Score: 1
  
  You could start with something like this (`tcpdump`):
  
  ```
  tcpdump -q ip and ip6 and dst port 80 or 443 > tcpdump.log
  
  ```
  
  This should capture all outgoing ipv4 and ipv6 traffic, to destiation ports 80 and 443 (http and https).
  You can add `-i eth1` to capture data on a specific interface only. You can play with filters.
  You should get something like this (results from curling google and facebook) - notice that you won't get actual **google.com** domain, but the results could still help to see which domains are blocked by your firewall:
  
  ```
  19:07:58.915115 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.915125 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.947639 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.947869 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
  19:07:58.948406 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
  19:08:04.608537 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.610415 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.610494 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 80
  19:08:04.656677 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.656887 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  19:08:04.657372 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
  
  ```
  
  You can then easily filter out the domains with awk (get the 5th section, delimited by spaces) - note that there's still the http/https appended to the end of the domain, and the colon sign:
  
  ```
  awk '{print $5}' tcpdump.log
  
  ```
  
  Result:
  
  ```
  edge-star-mini-shv-01-vie1.facebook.com.https:
  
  ```
  
  
  ---
  
  ### Answer by username_3 on 2023-03-16. Score: 1
  
  Maybe you could use some local redirecting proxy service and patch your *localhosts* file to get all requests caught there?
  
  
  ---
  
  