Answers: <p>Is it a hard requirement to use only <code>sar</code> and <code>sysstat</code>? If not you might want to look at <a href="http://collectl.sourceforge.net" rel="nofollow noreferrer">collectl</a> or <a href="https://collectd.org" rel="nofollow noreferrer">collectd</a>. These will enable you to study memory usage over time on a per-process granularity. It's really not worth wasting your time writing your own scripts to parse <code>/proc</code> as the other answer suggests; this is a solved problem.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-04T18:45:08.233Z
Answers: 2017-09-04T18:45:08.233Z
Answers: Gaius
Answers: <p>As Tensibai mentioned, you can extract this info from <a href="http://man7.org/linux/man-pages/man5/proc.5.html" rel="noreferrer">the <code>/proc</code> filesystem</a>, but in most cases you need to determine the trending yourself. There are several places which could be of interest:</p>
  
  <ul>
  <li><strong>/proc/[pid]/statm</strong></li>
  </ul>
  
  <blockquote>
  <pre><code>          Provides information about memory usage, measured in pages.
            The columns are:
  
                size       (1) total program size
                           (same as VmSize in /proc/[pid]/status)
                resident   (2) resident set size
                           (same as VmRSS in /proc/[pid]/status)
                shared     (3) number of resident shared pages (i.e., backed by a file)
                           (same as RssFile+RssShmem in /proc/[pid]/status)
                text       (4) text (code)
                lib        (5) library (unused since Linux 2.6; always 0)
                data       (6) data + stack
                dt         (7) dirty pages (unused since Linux 2.6; always 0)
  </code></pre>
  </blockquote>
  
  <pre><code>cat /proc/31520/statm
  1217567 835883 84912 29 0 955887 0
  </code></pre>
  
  <ul>
  <li>memory-related fields in <strong>/proc/[pid]/status</strong> (notably <code>Vm*</code> and <code>Rss*</code>), might be preferable if you also collect other info from this file</li>
  </ul>
  
  <blockquote>
  <pre><code>          * VmPeak: Peak virtual memory size.
  
            * VmSize: Virtual memory size.
  
            * VmLck: Locked memory size (see mlock(3)).
  
            * VmPin: Pinned memory size (since Linux 3.2).  These are
              pages that can't be moved because something needs to
              directly access physical memory.
  
            * VmHWM: Peak resident set size ("high water mark").
  
            * VmRSS: Resident set size.  Note that the value here is the
              sum of RssAnon, RssFile, and RssShmem.
  
            * RssAnon: Size of resident anonymous memory.  (since Linux
              4.5).
  
            * RssFile: Size of resident file mappings.  (since Linux 4.5).
  
            * RssShmem: Size of resident shared memory (includes System V
              shared memory, mappings from tmpfs(5), and shared anonymous
              mappings).  (since Linux 4.5).
  
            * VmData, VmStk, VmExe: Size of data, stack, and text
              segments.
  
            * VmLib: Shared library code size.
  
            * VmPTE: Page table entries size (since Linux 2.6.10).
  
            * VmPMD: Size of second-level page tables (since Linux 4.0).
  
            * VmSwap: Swapped-out virtual memory size by anonymous private
              pages; shmem swap usage is not included (since Linux
              2.6.34).
  </code></pre>
  </blockquote>
  
  <pre><code>server:/&gt; egrep '^(Vm|Rss)' /proc/31520/status
  VmPeak:  6315376 kB
  VmSize:  4870332 kB
  VmLck:         0 kB
  VmPin:         0 kB
  VmHWM:   5009608 kB
  VmRSS:   3344300 kB
  VmData:  3822572 kB
  VmStk:      1040 kB
  VmExe:       116 kB
  VmLib:    146736 kB
  VmPTE:      8952 kB
  VmSwap:        0 kB
  </code></pre>
  
  <p>Some processes can, through their behaviour and not through their actual memory footprint, contribute to the overall system memory starvation and eventual demise. So it might also be of interest to look at the <a href="https://linux-mm.org/OOM_Killer" rel="noreferrer">OOM Killer</a> related information, which already takes into account some trending information:</p>
  
  <ul>
  <li><strong>/proc/[pid]/oom_score</strong></li>
  </ul>
  
  <blockquote>
  <pre><code>          This file displays the current score that the kernel gives to
            this process for the purpose of selecting a process for the
            OOM-killer.  A higher score means that the process is more
            likely to be selected by the OOM-killer.  The basis for this
            score is the amount of memory used by the process, with
            increases (+) or decreases (-) for factors including:
  
            * whether the process creates a lot of children using fork(2)
              (+);
  
            * whether the process has been running a long time, or has
              used a lot of CPU time (-);
  
            * whether the process has a low nice value (i.e., &gt; 0) (+);
  
            * whether the process is privileged (-); and
  
            * whether the process is making direct hardware access (-).
  
            The oom_score also reflects the adjustment specified by the
            oom_score_adj or oom_adj setting for the process.
  </code></pre>
  </blockquote>
  
  <pre><code>server:/&gt; cat proc/31520/oom_score
  103
  </code></pre>
  
  <ul>
  <li><strong>/proc/[pid]/oom_score_adj</strong> (or its deprecated predecessor <strong>/proc/[pid]/oom_adj</strong>, if need be)</li>
  </ul>
  
  <blockquote>
  <pre><code>          This file can be used to adjust the badness heuristic used to
            select which process gets killed in out-of-memory conditions.
  
            The badness heuristic assigns a value to each candidate task
            ranging from 0 (never kill) to 1000 (always kill) to determine
            which process is targeted.  The units are roughly a proportion
            along that range of allowed memory the process may allocate
            from, based on an estimation of its current memory and swap
            use.  For example, if a task is using all allowed memory, its
            badness score will be 1000.  If it is using half of its
            allowed memory, its score will be 500.
  
            There is an additional factor included in the badness score:
            root processes are given 3% extra memory over other tasks.
  
            The amount of "allowed" memory depends on the context in which
            the OOM-killer was called.  If it is due to the memory
            assigned to the allocating task's cpuset being exhausted, the
            allowed memory represents the set of mems assigned to that
            cpuset (see cpuset(7)).  If it is due to a mempolicy's node(s)
            being exhausted, the allowed memory represents the set of
            mempolicy nodes.  If it is due to a memory limit (or swap
            limit) being reached, the allowed memory is that configured
            limit.  Finally, if it is due to the entire system being out
            of memory, the allowed memory represents all allocatable
            resources.
  
            The value of oom_score_adj is added to the badness score
            before it is used to determine which task to kill.  Acceptable
            values range from -1000 (OOM_SCORE_ADJ_MIN) to +1000
            (OOM_SCORE_ADJ_MAX).  This allows user space to control the
            preference for OOM-killing, ranging from always preferring a
            certain task or completely disabling it from OOM killing.  The
            lowest possible value, -1000, is equivalent to disabling OOM-
            killing entirely for that task, since it will always report a
            badness score of 0.
  
            Consequently, it is very simple for user space to define the
            amount of memory to consider for each task.  Setting an
            oom_score_adj value of +500, for example, is roughly
            equivalent to allowing the remainder of tasks sharing the same
            system, cpuset, mempolicy, or memory controller resources to
            use at least 50% more memory.  A value of -500, on the other
            hand, would be roughly equivalent to discounting 50% of the
            task's allowed memory from being considered as scoring against
            the task.
  
            For backward compatibility with previous kernels,
            /proc/[pid]/oom_adj can still be used to tune the badness
            score.  Its value is scaled linearly with oom_score_adj.
  
            Writing to /proc/[pid]/oom_score_adj or /proc/[pid]/oom_adj
            will change the other with its scaled value.
  </code></pre>
  </blockquote>
  
  <pre><code>server:/&gt; cat proc/31520/oom_score_adj 
  0
  </code></pre>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-02T00:28:59.373Z
Answers: this is an amazing response, thanks a lot. we were starting to have a bunch of oom but weren't sure which process(es) was causing it since they didn't happen that often before. the purpose of this question was to look for an "easy" way to log the memory usage of the system and its evolution and correlate it with operations before and after an oom. anyhoo learn a lot with your reply, so thanks again.
Answers: kokito
Answers: CC BY-SA 3.0
Answers: 2017-09-01T15:42:57.807Z
Answers: 2017-09-01T15:50:11.330Z
Answers: 2017-09-01T15:50:11.330Z
Answers: Dan Cornilescu
Body: <p>Can I get memory usage per process with Linux? we monitor our servers with <code>sysstat</code>/<code>sar</code>. But besides seeing that memory went off the roof at some point, we can't pinpoint which process was getting bigger and bigger. is there a way with <code>sar</code> (or other tools) to get memory usage per process? and look at it, later on?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-04-21T19:02:26.003Z
Comments: You can parse /proc for this,  there's an answer of me about swap easily adaptable for any other memory metric. Can't get the link right now
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-04-21T20:12:28.657Z
Comments: Starting point: http://stackoverflow.com/a/32327705/3627607
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-09-01T13:26:34.793Z
Comments: @kokito were you able to solve the issue?
Comments: 030
Tags: <monitoring><linux><logging>
Title: How to get memory usage per process with sar, sysstat?
UserDisplayName: kokito
ThreadText: # How to get memory usage per process with sar, sysstat?
  
  (Asked by: username_0 on 2017-04-21)
  
  Can I get memory usage per process with Linux? we monitor our servers with `sysstat`/`sar`. But besides seeing that memory went off the roof at some point, we can't pinpoint which process was getting bigger and bigger. is there a way with `sar` (or other tools) to get memory usage per process? and look at it, later on?
  
  **Tags:** monitoring, linux, logging
  
  > **Comments:**
  > **username_1**: You can parse /proc for this,  there's an answer of me about swap easily adaptable for any other memory metric. Can't get the link right now  
  > **username_1**: Starting point: http://stackoverflow.com/a/32327705/3627607  
  > **username_2**: @username_0 were you able to solve the issue?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2017-09-01. Score: 5 | Accepted answer
  
  As username_1 mentioned, you can extract this info from [the `/proc` filesystem](http://man7.org/linux/man-pages/man5/proc.5.html), but in most cases you need to determine the trending yourself. There are several places which could be of interest:
  
  * **/proc/[pid]/statm**
  
  > ```
  >           Provides information about memory usage, measured in pages.
  >           The columns are:
  >
  >               size       (1) total program size
  >                          (same as VmSize in /proc/[pid]/status)
  >               resident   (2) resident set size
  >                          (same as VmRSS in /proc/[pid]/status)
  >               shared     (3) number of resident shared pages (i.e., backed by a file)
  >                          (same as RssFile+RssShmem in /proc/[pid]/status)
  >               text       (4) text (code)
  >               lib        (5) library (unused since Linux 2.6; always 0)
  >               data       (6) data + stack
  >               dt         (7) dirty pages (unused since Linux 2.6; always 0)
  >
  > ```
  
  ```
  cat /proc/31520/statm
  1217567 835883 84912 29 0 955887 0
  
  ```
  
  * memory-related fields in **/proc/[pid]/status** (notably `Vm*` and `Rss*`), might be preferable if you also collect other info from this file
  
  > ```
  >           * VmPeak: Peak virtual memory size.
  >
  >           * VmSize: Virtual memory size.
  >
  >           * VmLck: Locked memory size (see mlock(3)).
  >
  >           * VmPin: Pinned memory size (since Linux 3.2).  These are
  >             pages that can't be moved because something needs to
  >             directly access physical memory.
  >
  >           * VmHWM: Peak resident set size ("high water mark").
  >
  >           * VmRSS: Resident set size.  Note that the value here is the
  >             sum of RssAnon, RssFile, and RssShmem.
  >
  >           * RssAnon: Size of resident anonymous memory.  (since Linux
  >             4.5).
  >
  >           * RssFile: Size of resident file mappings.  (since Linux 4.5).
  >
  >           * RssShmem: Size of resident shared memory (includes System V
  >             shared memory, mappings from tmpfs(5), and shared anonymous
  >             mappings).  (since Linux 4.5).
  >
  >           * VmData, VmStk, VmExe: Size of data, stack, and text
  >             segments.
  >
  >           * VmLib: Shared library code size.
  >
  >           * VmPTE: Page table entries size (since Linux 2.6.10).
  >
  >           * VmPMD: Size of second-level page tables (since Linux 4.0).
  >
  >           * VmSwap: Swapped-out virtual memory size by anonymous private
  >             pages; shmem swap usage is not included (since Linux
  >             2.6.34).
  >
  > ```
  
  ```
  server:/> egrep '^(Vm|Rss)' /proc/31520/status
  VmPeak:  6315376 kB
  VmSize:  4870332 kB
  VmLck:         0 kB
  VmPin:         0 kB
  VmHWM:   5009608 kB
  VmRSS:   3344300 kB
  VmData:  3822572 kB
  VmStk:      1040 kB
  VmExe:       116 kB
  VmLib:    146736 kB
  VmPTE:      8952 kB
  VmSwap:        0 kB
  
  ```
  
  Some processes can, through their behaviour and not through their actual memory footprint, contribute to the overall system memory starvation and eventual demise. So it might also be of interest to look at the [OOM Killer](https://linux-mm.org/OOM_Killer) related information, which already takes into account some trending information:
  
  * **/proc/[pid]/oom\_score**
  
  > ```
  >           This file displays the current score that the kernel gives to
  >           this process for the purpose of selecting a process for the
  >           OOM-killer.  A higher score means that the process is more
  >           likely to be selected by the OOM-killer.  The basis for this
  >           score is the amount of memory used by the process, with
  >           increases (+) or decreases (-) for factors including:
  >
  >           * whether the process creates a lot of children using fork(2)
  >             (+);
  >
  >           * whether the process has been running a long time, or has
  >             used a lot of CPU time (-);
  >
  >           * whether the process has a low nice value (i.e., > 0) (+);
  >
  >           * whether the process is privileged (-); and
  >
  >           * whether the process is making direct hardware access (-).
  >
  >           The oom_score also reflects the adjustment specified by the
  >           oom_score_adj or oom_adj setting for the process.
  >
  > ```
  
  ```
  server:/> cat proc/31520/oom_score
  103
  
  ```
  
  * **/proc/[pid]/oom\_score\_adj** (or its deprecated predecessor **/proc/[pid]/oom\_adj**, if need be)
  
  > ```
  >           This file can be used to adjust the badness heuristic used to
  >           select which process gets killed in out-of-memory conditions.
  >
  >           The badness heuristic assigns a value to each candidate task
  >           ranging from 0 (never kill) to 1000 (always kill) to determine
  >           which process is targeted.  The units are roughly a proportion
  >           along that range of allowed memory the process may allocate
  >           from, based on an estimation of its current memory and swap
  >           use.  For example, if a task is using all allowed memory, its
  >           badness score will be 1000.  If it is using half of its
  >           allowed memory, its score will be 500.
  >
  >           There is an additional factor included in the badness score:
  >           root processes are given 3% extra memory over other tasks.
  >
  >           The amount of "allowed" memory depends on the context in which
  >           the OOM-killer was called.  If it is due to the memory
  >           assigned to the allocating task's cpuset being exhausted, the
  >           allowed memory represents the set of mems assigned to that
  >           cpuset (see cpuset(7)).  If it is due to a mempolicy's node(s)
  >           being exhausted, the allowed memory represents the set of
  >           mempolicy nodes.  If it is due to a memory limit (or swap
  >           limit) being reached, the allowed memory is that configured
  >           limit.  Finally, if it is due to the entire system being out
  >           of memory, the allowed memory represents all allocatable
  >           resources.
  >
  >           The value of oom_score_adj is added to the badness score
  >           before it is used to determine which task to kill.  Acceptable
  >           values range from -1000 (OOM_SCORE_ADJ_MIN) to +1000
  >           (OOM_SCORE_ADJ_MAX).  This allows user space to control the
  >           preference for OOM-killing, ranging from always preferring a
  >           certain task or completely disabling it from OOM killing.  The
  >           lowest possible value, -1000, is equivalent to disabling OOM-
  >           killing entirely for that task, since it will always report a
  >           badness score of 0.
  >
  >           Consequently, it is very simple for user space to define the
  >           amount of memory to consider for each task.  Setting an
  >           oom_score_adj value of +500, for example, is roughly
  >           equivalent to allowing the remainder of tasks sharing the same
  >           system, cpuset, mempolicy, or memory controller resources to
  >           use at least 50% more memory.  A value of -500, on the other
  >           hand, would be roughly equivalent to discounting 50% of the
  >           task's allowed memory from being considered as scoring against
  >           the task.
  >
  >           For backward compatibility with previous kernels,
  >           /proc/[pid]/oom_adj can still be used to tune the badness
  >           score.  Its value is scaled linearly with oom_score_adj.
  >
  >           Writing to /proc/[pid]/oom_score_adj or /proc/[pid]/oom_adj
  >           will change the other with its scaled value.
  >
  > ```
  
  ```
  server:/> cat proc/31520/oom_score_adj 
  0
  
  ```
  
  > **Comments:**
  > **username_0**: this is an amazing response, thanks a lot. we were starting to have a bunch of oom but weren't sure which process(es) was causing it since they didn't happen that often before. the purpose of this question was to look for an "easy" way to log the memory usage of the system and its evolution and correlate it with operations before and after an oom. anyhoo learn a lot with your reply, so thanks again.  
  
  ---
  
  ### Answer by username_3 on 2017-09-04. Score: 4
  
  Is it a hard requirement to use only `sar` and `sysstat`? If not you might want to look at [collectl](http://collectl.sourceforge.net) or [collectd](https://collectd.org). These will enable you to study memory usage over time on a per-process granularity. It's really not worth wasting your time writing your own scripts to parse `/proc` as the other answer suggests; this is a solved problem.
  
  
  ---
  
  