Answers: <p>I've created a couple crude methods of grabbing the latest major versions of releases, as well as absolute latest.</p>
  
  <pre><code># Grab Latest Tomcat 7
  curl -i https://www-us.apache.org/dist/tomcat/tomcat-7/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'
  
  # Grab Latest Tomcat 8
  curl -i https://www-us.apache.org/dist/tomcat/tomcat-8/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'
  
  # Grab Latest Tomcat 9
  curl -i https://www-us.apache.org/dist/tomcat/tomcat-9/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'
  
  # Grab Latest Tomcat
  curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"
  </code></pre>
  
  <p>It's a bit of a hack where I'm looking for text in-between two values on a web page to grab the version number. It's crude and I'm sure there's a more elegant way to grab this, but this what I was able to figure out with my skillset! Maybe it can help someone in the future, but if anyone has better ideas I'm all ears.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-03-21T21:13:24.143Z
Answers: 2019-03-21T21:13:24.143Z
Answers: Keegan Jacobson
Answers: <p>In general, I would not recommend scraping version numbers and download URLs from websites.  You will end up having to modify your scripts when they inevitably end up breaking — I have learned this the hard way time and time again.</p>
  <p>For example, take the following script I wrote in 2016 to scrape the latest Tomcat version:</p>
  <pre class="lang-sh prettyprint-override"><code>curl https://tomcat.apache.org/whichversion.html | awk -n '
  /class=&quot;detail-table&quot;/ {
      phase = 1
  }
  phase == 1 &amp;&amp; /&lt;th&gt;/ {
      target_column++
      if (/Actual release revision/) phase = 2
  }
  phase == 2 &amp;&amp; /&lt;tr&gt;/ {
      current_column = 0
  }
  phase == 2 &amp;&amp; /&lt;td&gt;/ {
      current_column++
  }
  phase == 2 &amp;&amp; current_column == target_column &amp;&amp; !/alpha|beta|superceded/ {
      print substr($0, match($0, /[0-9]+\.[0-9]+\.[0-9]+/), RLENGTH)
      exit
  }'
  </code></pre>
  <p>Unfortunately, that script <strong>no longer works</strong>, and I do not know when it broke.  No amount of fancy scripting will save you from unexpected structural changes to a website you are scraping.</p>
  <p>Sure, maybe a little bit of finagling will fix this one particular script, but if you try to download all your dependencies this way, you'll be playing catch-up till the end of time.  Aim for a solution that you only need to code once.</p>
  <p><strong>EDIT:</strong> There's a good chance that I've been successfully nerd-sniped by @Bill_Stewart, but a working solution today (October 18th, 2024) is to simply replace <code>/Actual release revision/</code> with <code>/Latest Released Version/</code> in the above <code>awk</code> script. My original sentiments still stand and I would never use this in production unless I literally had no other choice.</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-10-17T13:52:16.750Z
Answers: This is sound advice, but these comments don't constitute an actual answer to the question.
Answers: Bill_Stewart
Answers: CC BY-SA 4.0
Answers: 2019-04-01T18:09:24.193Z
Answers: 2024-10-18T23:48:51.777Z
Answers: 2024-10-18T23:48:51.777Z
Answers: eeowaa
Body: <p>Looking for some assistance or ideas on how to grab the latest version of a Tomcat release from their website or other website.</p>
  
  <p>I came across these Stack Overflow/Stack Exchange links that were relevant below:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/22510705/get-the-latest-download-link-programmatically">https://stackoverflow.com/questions/22510705/get-the-latest-download-link-programmatically</a> (Nginx, but same idea)</li>
  <li><a href="https://devops.stackexchange.com/questions/3262/how-to-get-the-latest-tomcat-version">How to get the latest Tomcat version?</a></li>
  </ul>
  
  <p>..But unfortunately both led to some dead-ends. It did give me some ideas though.</p>
  
  <p>I have some installation scripts where the download URL is predictable based on the version number of the software, so if there's a consistent way of scraping for the latest version of the software I simply add a variable into the script for it to curl / grep for that version on the text and use that variable for download links, untaring, moving, etc.</p>
  
  <p>Hoping to do something with Tomcat in the same vein. A installation script I won't need to keep updating (unless they change their site).</p>
  
  <p>Any sort of thoughts, ideas, are appreciated.</p>
  
  <p>I've also looked at <a href="http://tomcat.apache.org/whichversion.html" rel="nofollow noreferrer">http://tomcat.apache.org/whichversion.html</a> which lists the "Latest Released Version" and this seems like maybe the best, more reliable location to grab the version, but I had trouble getting that with Curl / Grep because of the table structure. I am only just scratching the surface with learning those commands, so maybe someone more well versed could get that working, otherwise I'm definitely open to other thoughts!</p>
  
  <pre><code>curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"
  </code></pre>
  
  <p>It seems to work but I am not sure if I am entirely happy with it. An additional use-case that I thought of, is it would be nice to grab the latest version of each major version instead of just the absolute latest.</p>
  
  <p>Do you have suggestions on how you would parse the highest number with the link you provided, and/or how to start in one of those directories and parse the highest there? That way for example I could create a latest version install script for Tomcat 7, 8, 9, etc.</p>
  
  <p>Thank you!! (Like I said I am just getting started, so please my apologies with the lack of knowledge but I do appreciate digging in and understanding how the commands work or reached their conclusion)</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-03-21T15:01:05.667Z
Comments: You should not upgrade tomcat without prior reading of the changelog, even minor security fix may break your application.
Comments: Tensibai
Comments: CC BY-SA 4.0
Comments: 2019-03-21T15:06:24.047Z
Comments: Hi @Tensibai thanks for the advise. The goal here is to create a version agnostic installation script that will always grab the latest version. So that when we do approve the latest version, I can simply run the script that will already be present on the machines or systems.
  
  Assume here that we already taking the necessary precautions surrounding the version, and just care to have a script to grab the latest for a version (Tomcat 7, Tomcat 8, Tomcat 9, etc).
  
  Appreciate your concern and insight!
Comments: Keegan Jacobson
Comments: CC BY-SA 4.0
Comments: 2019-03-21T15:44:16.040Z
Comments: Best course of action would be to pass as a parameter the version to install, specially as there's no "latest" tomcat, but 3 latest depending on major version, so taking the latest by date may be 7, 8 or 9 and you'll have to at least pass this major as parameter, better pass the whole desired version directly.
Comments: Tensibai
Comments: CC BY-SA 4.0
Comments: 2019-03-21T20:19:18.737Z
Comments: What I'm actually going to do is have three different installation scripts, one for 7, 8, and 9 (and then in the future one for each of the future versions) but I want to be able to grab the latest release for a particular major version.
Comments: Keegan Jacobson
Comments: CC BY-SA 4.0
Comments: 2019-03-22T08:45:32.303Z
Comments: And you'll have 3 scripts to maintain after that, that's just a bad idea.
Comments: Tensibai
Comments: CC BY-SA 4.0
Comments: 2019-03-22T16:27:52.273Z
Comments: @Tensibai I'm not sure how to pass a parameter so this is probably the best I'm going to get otherwise. I'm open to suggestions if you know how to do that, it seems more like a preference than a bad idea to me though.
Comments: Keegan Jacobson
Comments: CC BY-SA 4.0
Comments: 2019-03-22T21:40:45.313Z
Comments: OK, you need a sheel tutorial then, exemple: `./command parameter1` in the script named command, parameter1 will be available by using the variable $1, just pass the version. And no, it's not a preference, but hopefully your future self will remember this exchange when a bunch of nearly identical scripts will all fail at the same time and you'll understand why Dry (Don't repeat yourself) is something to do while coding.
Comments: Tensibai
Comments: CC BY-SA 4.0
Comments: 2019-03-22T22:12:52.220Z
Comments: Thank you for the reply! Ultimately I'm trying to solve the solution at hand, and also understand if there's a better or more efficient way to solve problems. I do see the benefit of that for a lot of scenarios not just this one. I'll need to do some research and testing at a shell when I get an opportunity.
Comments: Keegan Jacobson
Tags: <shell-script><tomcat>
Title: Programmatically scrape the latest version of Tomcat for an installation script
UserDisplayName: Keegan Jacobson
ThreadText: # Programmatically scrape the latest version of Tomcat for an installation script
  
  (Asked by: username_0 on 2019-03-21)
  
  Looking for some assistance or ideas on how to grab the latest version of a Tomcat release from their website or other website.
  
  I came across these Stack Overflow/Stack Exchange links that were relevant below:
  
  * <https://stackoverflow.com/questions/22510705/get-the-latest-download-link-programmatically> (Nginx, but same idea)
  * [How to get the latest Tomcat version?](https://devops.stackexchange.com/questions/3262/how-to-get-the-latest-tomcat-version)
  
  ..But unfortunately both led to some dead-ends. It did give me some ideas though.
  
  I have some installation scripts where the download URL is predictable based on the version number of the software, so if there's a consistent way of scraping for the latest version of the software I simply add a variable into the script for it to curl / grep for that version on the text and use that variable for download links, untaring, moving, etc.
  
  Hoping to do something with Tomcat in the same vein. A installation script I won't need to keep updating (unless they change their site).
  
  Any sort of thoughts, ideas, are appreciated.
  
  I've also looked at <http://tomcat.apache.org/whichversion.html> which lists the "Latest Released Version" and this seems like maybe the best, more reliable location to grab the version, but I had trouble getting that with Curl / Grep because of the table structure. I am only just scratching the surface with learning those commands, so maybe someone more well versed could get that working, otherwise I'm definitely open to other thoughts!
  
  ```
  curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"
  
  ```
  
  It seems to work but I am not sure if I am entirely happy with it. An additional use-case that I thought of, is it would be nice to grab the latest version of each major version instead of just the absolute latest.
  
  Do you have suggestions on how you would parse the highest number with the link you provided, and/or how to start in one of those directories and parse the highest there? That way for example I could create a latest version install script for Tomcat 7, 8, 9, etc.
  
  Thank you!! (Like I said I am just getting started, so please my apologies with the lack of knowledge but I do appreciate digging in and understanding how the commands work or reached their conclusion)
  
  **Tags:** shell-script, tomcat
  
  > **Comments:**
  > **username_1**: You should not upgrade tomcat without prior reading of the changelog, even minor security fix may break your application.  
  > **username_0**: Hi @username_1 thanks for the advise. The goal here is to create a version agnostic installation script that will always grab the latest version. So that when we do approve the latest version, I can simply run the script that will already be present on the machines or systems.  Assume here that we already taking the necessary precautions surrounding the version, and just care to have a script to grab the latest for a version (Tomcat 7, Tomcat 8, Tomcat 9, etc).  Appreciate your concern and insight!  
  > **username_1**: Best course of action would be to pass as a parameter the version to install, specially as there's no "latest" tomcat, but 3 latest depending on major version, so taking the latest by date may be 7, 8 or 9 and you'll have to at least pass this major as parameter, better pass the whole desired version directly.  
  > **username_0**: What I'm actually going to do is have three different installation scripts, one for 7, 8, and 9 (and then in the future one for each of the future versions) but I want to be able to grab the latest release for a particular major version.  
  > **username_1**: And you'll have 3 scripts to maintain after that, that's just a bad idea.  
  > **username_0**: @username_1 I'm not sure how to pass a parameter so this is probably the best I'm going to get otherwise. I'm open to suggestions if you know how to do that, it seems more like a preference than a bad idea to me though.  
  > **username_1**: OK, you need a sheel tutorial then, exemple: `./command parameter1` in the script named command, parameter1 will be available by using the variable $1, just pass the version. And no, it's not a preference, but hopefully your future self will remember this exchange when a bunch of nearly identical scripts will all fail at the same time and you'll understand why Dry (Don't repeat yourself) is something to do while coding.  
  > **username_0**: Thank you for the reply! Ultimately I'm trying to solve the solution at hand, and also understand if there's a better or more efficient way to solve problems. I do see the benefit of that for a lot of scenarios not just this one. I'll need to do some research and testing at a shell when I get an opportunity.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2019-03-21. Score: 0 | Accepted answer
  
  I've created a couple crude methods of grabbing the latest major versions of releases, as well as absolute latest.
  
  ```
  # Grab Latest Tomcat 7
  curl -i https://www-us.apache.org/dist/tomcat/tomcat-7/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'
  
  # Grab Latest Tomcat 8
  curl -i https://www-us.apache.org/dist/tomcat/tomcat-8/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'
  
  # Grab Latest Tomcat 9
  curl -i https://www-us.apache.org/dist/tomcat/tomcat-9/ | grep -Po '(?<=(<a href="v)).*(?=/">v)'
  
  # Grab Latest Tomcat
  curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"
  
  ```
  
  It's a bit of a hack where I'm looking for text in-between two values on a web page to grab the version number. It's crude and I'm sure there's a more elegant way to grab this, but this what I was able to figure out with my skillset! Maybe it can help someone in the future, but if anyone has better ideas I'm all ears.
  
  
  ---
  
  ### Answer by username_2 on 2019-04-01. Score: 0
  
  In general, I would not recommend scraping version numbers and download URLs from websites. You will end up having to modify your scripts when they inevitably end up breaking — I have learned this the hard way time and time again.
  
  For example, take the following script I wrote in 2016 to scrape the latest Tomcat version:
  
  ```sh
  curl https://tomcat.apache.org/whichversion.html | awk -n '
  /class="detail-table"/ {
      phase = 1
  }
  phase == 1 && /<th>/ {
      target_column++
      if (/Actual release revision/) phase = 2
  }
  phase == 2 && /<tr>/ {
      current_column = 0
  }
  phase == 2 && /<td>/ {
      current_column++
  }
  phase == 2 && current_column == target_column && !/alpha|beta|superceded/ {
      print substr($0, match($0, /[0-9]+\.[0-9]+\.[0-9]+/), RLENGTH)
      exit
  }'
  
  ```
  
  Unfortunately, that script **no longer works**, and I do not know when it broke. No amount of fancy scripting will save you from unexpected structural changes to a website you are scraping.
  
  Sure, maybe a little bit of finagling will fix this one particular script, but if you try to download all your dependencies this way, you'll be playing catch-up till the end of time. Aim for a solution that you only need to code once.
  
  **EDIT:** There's a good chance that I've been successfully nerd-sniped by @Bill\_Stewart, but a working solution today (October 18th, 2024) is to simply replace `/Actual release revision/` with `/Latest Released Version/` in the above `awk` script. My original sentiments still stand and I would never use this in production unless I literally had no other choice.
  
  > **Comments:**
  > **username_3**: This is sound advice, but these comments don't constitute an actual answer to the question.  
  
  ---
  
  