title: Go 1.13 RSS keeps on increasing, suspected scavenging issue
body: <p>We are experiencing RSS that keeps on increasing in one of our Go service.
  We are suspecting it's due to scavenger not returning memory to OS properly (or OS not taking back the memory due to use of MADV_FREE). Checked via pprof, no memory leak detected.</p>
  
  <p>We tried some experiment with the following simple Go program:
  Go version: go1.13.4 linux/amd64 (tried with go1.13.1 too)</p>
  
  <pre><code>package main
  import (
    "fmt"
    "time"
  )
  
  func allocate(s int) {
    a := make([]byte, s * 1024 * 1024)
    for i := 0;i &lt; len(a); i += 4096 {
      a[i] = 'x'
    }
    fmt.Printf("%c %d\n", a[0], s)
  }
  
  func main() {
    allocate(100)
    allocate(200)
    allocate(500)
    time.Sleep(300 * time.Second)
  }
  
  </code></pre>
  
  <p>We run with</p>
  
  <pre><code>strace -q -e trace=memory go run main.go
  </code></pre>
  
  <p>and the results are not consistent:</p>
  
  <ul>
  <li><a href="https://pastebin.com/sGw3dp9E" rel="nofollow noreferrer">https://pastebin.com/sGw3dp9E</a> (no MADV_FREE reported)</li>
  <li><a href="https://pastebin.com/d6CmRfD4" rel="nofollow noreferrer">https://pastebin.com/d6CmRfD4</a> (MADV_FREE reported, but why the values freed are so small?)</li>
  </ul>
  
  <p>running with gctrace, we got: <a href="https://pastebin.com/6JaC2r85" rel="nofollow noreferrer">https://pastebin.com/6JaC2r85</a></p>
  
  <p>we confirmed that RSS remains high via top and pmap:</p>
  
  <pre><code>$ pmap -x 22712
  22712:   /tmp/go-build073210395/b001/exe/main
  Address           Kbytes     RSS   Dirty Mode  Mapping
  0000000000400000     576     428       0 r-x-- main
  0000000000490000     756     320       0 r---- main
  000000000054d000      84      56      32 rw--- main
  0000000000562000     120      36      36 rw---   [ anon ]
  000000c000000000  983040  819440  397680 rw---   [ anon ]
  00007fcdfb2e1000   66684   26780   26780 rw---   [ anon ]
  00007fffa3ba2000     132      20      20 rw---   [ stack ]
  00007fffa3bdf000       8       4       0 r-x--   [ anon ]
  ffffffffff600000       4       0       0 r-x--   [ anon ]
  ---------------- ------- ------- -------
  total kB         1051404  847084  424548
  </code></pre>
  
  <p>Appreciate if someone could help to clarify the following:</p>
  
  <ol>
  <li>According to <a href="https://github.com/golang/go/issues/30333" rel="nofollow noreferrer">https://github.com/golang/go/issues/30333</a>, in go 1.13 scavenging should be performed periodically instead of waiting for the 5 minutes interval, but there is no MADV_FREE logged by strace sometimes. gctrace does print logs about scavenging, but seems like it's not really calling MADV_FREE? Am I missing something regarding this scavenging logic in go 1.13?</li>
  <li>Tried with GODEBUG=madvdontneed=1, result is better, but RSS still hovering at around 500MB, only when we combine madvdontneed with debug.FreeOSMemory() then we got RSS &lt; 30MB. Is this the only way to ensure Go is returning memory back to OS in Linux?</li>
  </ol>
  
  <p>Additional notes: running the same program on windows with go 1.13 seems to have the desired effect, i.e. memory is gradually released back to OS.</p>