Answers: <p>This functionality doesn't yet exist in the UI, though internally job logs are upload as a job artifact. And you can search them with a little bit of work. The backing store is configured in your, <code>/etc/gitlab/gitlab.rb</code>.</p>
  <ul>
  <li><p>If <code>artifacts_object_store_enabled</code> is enabled and you're using the <a href="https://docs.gitlab.com/ee/administration/object_storage.html" rel="nofollow noreferrer">Object Storage</a>, the artifacts will store in whatever the value is of <code>artifacts_object_store_remote_directory</code> under a file called <code>job.log</code>. Assuming that value is <code>artifacts</code>, and you're using something like <a href="https://min.io/" rel="nofollow noreferrer">minio</a>, you can search across jobs for a specific failure with</p>
  <ul>
  <li><p><code>mc find</code> and <code>mc cat</code>,</p>
  <pre class="lang-sh prettyprint-override"><code># In the example below, I'm looking for failures that contain Client.Timeout
  mc find minio/artifacts \
    --name 'job.log'      \
    --exec 'sh -c &quot;mc cat {} | grep Client\.Timeout | xargs -r echo {time} {}; exit 0&quot;' \
    | tee /tmp/client_timeout.log
  </code></pre>
  <p>In my experience, this is just too slow as <code>mc find</code> isn't indexed it's just looking for any bucket with the name <code>job.log</code>.</p>
  </li>
  <li><p>A faster method is to lean on the database. Gitlab ships a PostgreSQL client wrapper called <code>gitlab-psql</code> which allows you to connect to underlying PostgreSQL datastore. After you connect with <code>gitlab-psql</code> can query for all <code>file.log</code> and <a href="https://devops.stackexchange.com/q/17298/18965">get the relative path to the bucket from the result</a>. You can easily tool this by creating a batch file of commands for <a href="https://github.com/peak/s5cmd" rel="nofollow noreferrer">s5cmd</a> log,</p>
  <pre><code>\t
  \a
  \o /tmp/pull_logs.s5cmd
  SELECT FORMAT(
    'cp s3://artifacts/%s /tmp/job_%s.log',
    to_bucket_path(ci_job_artifacts),
    id
  )
  FROM ci_job_artifacts
  WHERE file = 'job.log'
  AND expire_at IS NULL
  AND file_store = 2;
  </code></pre>
  <p>Which will generate a file you can run with something like <a href="https://github.com/peak/s5cmd" rel="nofollow noreferrer">s5cmd</a> to pull down all of your logs from the Gitlab Object store. You'd run it by setting up your <code>~/.aws/credentials</code> and running (example with minio).</p>
  <pre><code>s5cmd --endpoint-url https://minio.acme.net run /tmp/pull_logs.s5cmd
  </code></pre>
  </li>
  </ul>
  </li>
  <li><p>You can also use <a href="https://docs.gitlab.com/ee/administration/nfs.html" rel="nofollow noreferrer">NFS</a>. If you're using NFS for artifact storage, it's as easy as going to the mount point and searching with ripgrep.</p>
  </li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2023-01-19T22:39:40.567Z
Answers: 2023-01-26T20:08:48.770Z
Answers: 2023-01-26T20:08:48.770Z
Answers: Evan Carroll
Body: <p>I need to find all jobs that failed in a specific fashion, a few methods are viable</p>
  <ul>
  <li>One method that would be viable is finding where GitLab stores its log files, and searching them with something like Grep. <a href="https://devops.stackexchange.com/q/17254/18965">I asked that question here</a>.</li>
  <li>Another method would be using native GitLab search functionality to search the logs. Does such functionality exist?</li>
  </ul>
  
Comments: CC BY-SA 4.0
Comments: 2023-11-01T11:45:42.843Z
Comments: The functionality seems not implemented but it was raised here https://gitlab.com/gitlab-org/gitlab/-/issues/363640 .
Comments: YAZ84
Tags: <gitlab><logging>
Title: Does GitLab support searching logs across jobs for specific strings?
UserDisplayName: Evan Carroll
ThreadText: # Does GitLab support searching logs across jobs for specific strings?
  
  (Asked by: username_0 on 2023-01-17)
  
  I need to find all jobs that failed in a specific fashion, a few methods are viable
  
  * One method that would be viable is finding where GitLab stores its log files, and searching them with something like Grep. [I asked that question here](https://devops.stackexchange.com/q/17254/18965).
  * Another method would be using native GitLab search functionality to search the logs. Does such functionality exist?
  
  **Tags:** gitlab, logging
  
  > **Comments:**
  > **username_1**: The functionality seems not implemented but it was raised here https://gitlab.com/gitlab-org/gitlab/-/issues/363640 .  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-01-19. Score: 0 | Accepted answer
  
  This functionality doesn't yet exist in the UI, though internally job logs are upload as a job artifact. And you can search them with a little bit of work. The backing store is configured in your, `/etc/gitlab/gitlab.rb`.
  
  * If `artifacts_object_store_enabled` is enabled and you're using the [Object Storage](https://docs.gitlab.com/ee/administration/object_storage.html), the artifacts will store in whatever the value is of `artifacts_object_store_remote_directory` under a file called `job.log`. Assuming that value is `artifacts`, and you're using something like [minio](https://min.io/), you can search across jobs for a specific failure with
  
    + `mc find` and `mc cat`,
  
      ```sh
      # In the example below, I'm looking for failures that contain Client.Timeout
      mc find minio/artifacts \
        --name 'job.log'      \
        --exec 'sh -c "mc cat {} | grep Client\.Timeout | xargs -r echo {time} {}; exit 0"' \
        | tee /tmp/client_timeout.log
  
      ```
  
      In my experience, this is just too slow as `mc find` isn't indexed it's just looking for any bucket with the name `job.log`.
    + A faster method is to lean on the database. Gitlab ships a PostgreSQL client wrapper called `gitlab-psql` which allows you to connect to underlying PostgreSQL datastore. After you connect with `gitlab-psql` can query for all `file.log` and [get the relative path to the bucket from the result](https://devops.stackexchange.com/q/17298/18965). You can easily tool this by creating a batch file of commands for [s5cmd](https://github.com/peak/s5cmd) log,
  
      ```
      \t
      \a
      \o /tmp/pull_logs.s5cmd
      SELECT FORMAT(
        'cp s3://artifacts/%s /tmp/job_%s.log',
        to_bucket_path(ci_job_artifacts),
        id
      )
      FROM ci_job_artifacts
      WHERE file = 'job.log'
      AND expire_at IS NULL
      AND file_store = 2;
  
      ```
  
      Which will generate a file you can run with something like [s5cmd](https://github.com/peak/s5cmd) to pull down all of your logs from the Gitlab Object store. You'd run it by setting up your `~/.aws/credentials` and running (example with minio).
  
      ```
      s5cmd --endpoint-url https://minio.acme.net run /tmp/pull_logs.s5cmd
  
      ```
  * You can also use [NFS](https://docs.gitlab.com/ee/administration/nfs.html). If you're using NFS for artifact storage, it's as easy as going to the mount point and searching with ripgrep.
  
  
  ---
  
  