title: How to use java memory histogram "jmap"
body: <p>We have java streaming server running in production and it requires ~10GB of RAM for operation so we have 32GB installed. Memory is gradually increased until limit is reached and out-of-memory exceptions pop up.</p>
  
  <p>I have trouble pinpointing what objects are accumulating over time because histogram and memory dump numbers do not match the system reported memory usage, namely java process occupies a little more than max 20GB (so out-of-memory exceptions are justified) but histogram and memory dump shows total of 6.4GB used.</p>
  
  <pre><code>process : 19.8G
  Java
  reported:  6.4G
  ---------------
  unknown 
  occupied
  segment : 13.4G
  </code></pre>
  
  <p>How to get information about memory in unknown occupied segment that is not shown in histogram ?</p>
  
  <p>I used <code>jmap -J-d64 -histo &lt;pid&gt; &gt; &lt;file&gt;</code> command to generate histogram. </p>
  
  <p>Process has following memory segments mapped, shown sorted by size</p>
  
  <pre><code>0x2DE000000: 13333.5MB
  0x61F580000: 6666.5MB
  0x7C0340000: 1020.8MB
  0x7FBFF33C9000: 716.2MB
  0x7FC086A75000: 196.9MB
  0x7FB85C000000: 64.0MB
  0x7FBAC0000000: 64.0MB
  ...
  </code></pre>
  
  <p>Total size of all java objects reported by jmap fits in <code>0x61F580000: 6666.5MB</code> segment.  </p>
  
  <p>My guess is that larger segment the <code>0x2DE000000: 13333.5MB</code> holds the leaked objects because histogram shows normal memory usage for this application.  </p>
  
  <p>Is there a way to find out what is occupying other memory not included in histogram ?</p>
  
  <p>How can I detect If the closed-source part of server is using native extensions to allocate system memory instead of java memory ? In that case we would not see out-of-memory exceptions right ?</p>
  
  <p>Here is htop output:</p>
  
  <pre><code>  Mem[|||||||||||||||||||||31670/31988MB]     Tasks: 87; 35 running
    Swp[||||||||||||||||||   16361/32579MB]     Load average: 39.33 36.00 34.72
                                                Uptime: 44 days, 15:08:19
  
    PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
   3498 root       20   0 51.5G 19.8G  4516 S 151. 63.4     176h java -server -Xmx20000M -
  </code></pre>
  
  <p>Here is partial histogram output:</p>
  
  <pre><code> num     #instances         #bytes  class name
  ----------------------------------------------
     1:       1134597     5834904800  [B
     2:        407694      144032664  [Ljava.lang.Object;
     3:       2018132      111547480  [C
     4:        100217       71842520  [I
     5:        581934       55865664  com.wowza.wms.httpstreamer.cupertinostreaming.livestreampacketizer.CupertinoRepeaterHolder
     6:        568535       36386240  com.wowza.wms.httpstreamer.cupertinostreaming.livestreampacketizer.CupertinoTSHolder
     7:        975220       23405280  java.lang.String
     8:        967713       23225112  com.wowza.wms.amf.AMFObjChunk
     9:        621660       14919840  com.wowza.wms.httpstreamer.cupertinostreaming.livestreampacketizer.LiveStreamingCupertinoBlock
    10:        369892       11836544  java.util.ArrayList$Itr
    11:        184502       11808128  com.wowza.wms.amf.AMFPacket
    12:        329055        7897320  java.util.ArrayList
    13:         55882        6705840  com.wowza.wms.server.RtmpRequestMessage
    14:        200263        6408416  java.util.HashMap$Node
    15:         86784        6248448  com.wowza.wms.httpstreamer.cupertinostreaming.livestreampacketizer.CupertinoPacketHolder
    16:         24815        5360040  com.wowza.wms.media.h264.H264CodecConfigInfo
    17:        209398        5025552  java.lang.StringBuilder
    18:        168061        4033464  com.wowza.util.PacketFragment
    19:        119160        3813120  java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
    20:         93849        3753960  java.util.TreeMap$Entry
    21:        155756        3738144  java.lang.Long
    22:         55881        3576384  com.wowza.wms.server.RtmpResponseMessage
    23:         55760        3568640  com.wowza.util.FasterByteArrayOutputStream
    24:        130452        3130848  java.util.concurrent.LinkedBlockingQueue$Node
    25:         63172        3032256  java.util.HashMap
    26:         58747        2819856  java.nio.HeapByteBuffer
    27:         34830        2800568  [J
    28:         49076        2355648  java.util.TreeMap$AscendingSubMap
    29:         70567        2258144  com.wowza.wms.stream.livepacketizer.LiveStreamPacketizerBase$PacketizerEventHolder
    30:         55721        2228840  org.apache.mina.util.Queue
    31:         54990        2199600  java.util.HashMap$KeyIterator
    32:         58583        1874656  org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
    33:        112743        1803888  java.lang.Integer
    34:         55509        1776288  com.wowza.wms.server.ServerHandlerEvent
  ...
  2089:             1             16  sun.util.resources.LocaleData$LocaleDataResourceBundleControl
  Total      11078054     6454934408
  </code></pre>
  
  <p>The java version is:</p>
  
  <pre><code># java -version
  java version "1.8.0_05"
  Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
  Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
  </code></pre>