Answers: <p>Doing something with <code>collectd</code> feeding into Prometheus or datadog is definitely a good idea for gathering metrics, but it doesn't help you with things like the kernel version.  Handling process monitoring might be better handled with nagios or datadog as well.  But for the bulk of what you're doing I'd say that it makes the most sense to stick with ansible.</p>
  <p>What you're doing sounds very similar to the <a href="https://github.com/fboender/ansible-cmdb" rel="nofollow noreferrer">ansible-cmdb project</a>:</p>
  <blockquote>
  <p>Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page (and other things) containing system configuration information.</p>
  <p>It supports multiple types of output (html, csv, sql, etc) and extending information gathered by Ansible with custom data. For each host it also shows the groups, host variables, custom variables and machine-local facts.</p>
  </blockquote>
  <p><img src="https://raw.githubusercontent.com/fboender/ansible-cmdb/master/contrib/screenshot-overview.png" alt="ansible-cmdb example" /></p>
  <p>I've been using this for the last couple of years and it continues to evolve and improve.  I hope you'll find it useful as well.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-08-28T01:30:38.113Z
Answers: 2018-08-28T01:30:38.113Z
Answers: 2020-06-18T08:28:10.803Z
Answers: chicks
Body: <p>Recently, I use <code>ansible</code> to gather information periodically from thousands of servers(which already installed <code>collectd</code> to gather CPU, memory, disks, netowrking info) in our company.</p>
  
  <p>For example: I need to get those information daily</p>
  
  <ul>
  <li>kernel version</li>
  <li>installed packages versions(JDK, docker, etc)</li>
  <li>running process info</li>
  <li>some commands output (lldp, docker ps, etc)</li>
  <li>some files content (cat /etc/some/config | grep word)</li>
  </ul>
  
  <p>I use ansible <code>shell</code> module to execute those variant commands and parse the output into our inventory  NOSQL.</p>
  
  <p>By using ansible which use SSH/WINRM to access thousands servers seems inefficient.(which fine tune with more <code>forks</code>, enabled <code>ControlMaster</code> <code>pipeline</code>, etc).</p>
  
  <p>I are trying to find a better way to replace ansible for those information gathering and parsing.</p>
  
  <p>Below are some solutions which I am surveying.</p>
  
  <p><code>facebook/osquery</code> is flexible and powerful, but maybe not easy(comparing to bash script) to run a customized commands and parsing the output by writing an extension. </p>
  
  <p><code>collectd with exec plugin</code> to run customized scripts seems straightforward. in this way I need to deploy new scripts to all servers from day to day. </p>
  
  <p>Are there some tools which is more suitable for such variant commands output gathering? and easy to <code>deploy</code>, <code>reporting</code>, or may with some <code>dashboard</code> and <code>API</code> for operation.</p>
  
Tags: <ansible><monitoring><systems>
Title: tools to gather variant information from thousands of servers and update into database
UserDisplayName: CSJ
ThreadText: # tools to gather variant information from thousands of servers and update into database
  
  (Asked by: username_0 on 2018-08-27)
  
  Recently, I use `ansible` to gather information periodically from thousands of servers(which already installed `collectd` to gather CPU, memory, disks, netowrking info) in our company.
  
  For example: I need to get those information daily
  
  * kernel version
  * installed packages versions(JDK, docker, etc)
  * running process info
  * some commands output (lldp, docker ps, etc)
  * some files content (cat /etc/some/config | grep word)
  
  I use ansible `shell` module to execute those variant commands and parse the output into our inventory NOSQL.
  
  By using ansible which use SSH/WINRM to access thousands servers seems inefficient.(which fine tune with more `forks`, enabled `ControlMaster` `pipeline`, etc).
  
  I are trying to find a better way to replace ansible for those information gathering and parsing.
  
  Below are some solutions which I am surveying.
  
  `facebook/osquery` is flexible and powerful, but maybe not easy(comparing to bash script) to run a customized commands and parsing the output by writing an extension.
  
  `collectd with exec plugin` to run customized scripts seems straightforward. in this way I need to deploy new scripts to all servers from day to day.
  
  Are there some tools which is more suitable for such variant commands output gathering? and easy to `deploy`, `reporting`, or may with some `dashboard` and `API` for operation.
  
  **Tags:** ansible, monitoring, systems
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-28. Score: 1
  
  Doing something with `collectd` feeding into Prometheus or datadog is definitely a good idea for gathering metrics, but it doesn't help you with things like the kernel version. Handling process monitoring might be better handled with nagios or datadog as well. But for the bulk of what you're doing I'd say that it makes the most sense to stick with ansible.
  
  What you're doing sounds very similar to the [ansible-cmdb project](https://github.com/fboender/ansible-cmdb):
  
  > Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page (and other things) containing system configuration information.
  >
  > It supports multiple types of output (html, csv, sql, etc) and extending information gathered by Ansible with custom data. For each host it also shows the groups, host variables, custom variables and machine-local facts.
  
  ![ansible-cmdb example](https://raw.githubusercontent.com/fboender/ansible-cmdb/master/contrib/screenshot-overview.png)
  
  I've been using this for the last couple of years and it continues to evolve and improve. I hope you'll find it useful as well.
  
  
  ---
  
  