<p dir="auto">I'm very sorry about the quality of this issue report, I've yet to be able to resolve it to a smaller reproduction.</p>
<p dir="auto">The background is we have an API server which instruments types and exposes their methods over an RPC interface. This stack trace is from the sever side, attempting to dispatch to <code class="notranslate">agent.(*AgentAPIV1).StateServingInfo</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="unexpected fault address 0xf8010260f8010260
fatal error: fault
[signal 0xb code=0x1 addr=0xf8010260f8010260 pc=0x1e56c]

goroutine 229 [running]:
runtime.throw(0xf45d00, 0x5)
        /home/ubuntu/go/src/runtime/panic.go:543 +0x8c fp=0xc2083ff688 sp=0xc2083ff670
runtime.sigpanic()
        /home/ubuntu/go/src/runtime/sigpanic_unix.go:27 +0x2f0 fp=0xc2083ff6d8 sp=0xc2083ff688
runtime.convI2E(0xf8010260f8010258, 0x7c030378e8410070, 0x0, 0x0)
        /home/ubuntu/go/src/runtime/iface.go:256 +0x74 fp=0xc2083ff708 sp=0xc2083ff6e0
github.com/juju/juju/apiserver/agent.(*AgentAPIV0).StateServingInfo(0x31f5c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93 +0xa0 fp=0xc2083ff830 sp=0xc2083ff708
github.com/juju/juju/apiserver/agent.(*AgentAPIV1).StateServingInfo(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        &lt;autogenerated&gt;:22 +0x40 fp=0xc2083ff838 sp=0xc2083ff830
runtime.call128(0xc208205bd0, 0xe85098, 0xc208398380)
        /home/ubuntu/go/src/runtime/asm_ppc64x.s:413 +0x88 fp=0xc2083ff8c0 sp=0xc2083ff838
reflect.Value.call(0xe84f40, 0xc20802c5f0, 0x913, 0xf30060, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/go/src/reflect/value.go:432 +0xe30 fp=0xc2083ffc00 sp=0xc2083ff8c0
reflect.Value.Call(0xe84f40, 0xc20802c5f0, 0x913, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/ubuntu/go/src/reflect/value.go:300 +0x94 fp=0xc2083ffc60 sp=0xc2083ffc00
github.com/juju/juju/rpc/rpcreflect.newMethod.func6(0xe84f40, 0xc20802c5f0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/rpc/rpcreflect/type.go:326 +0x1a0 fp=0xc2083ffd60 sp=0xc2083ffc60
github.com/juju/juju/apiserver.(*srvCaller).Call(0xc208581500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/apiserver/root.go:131 +0xf0 fp=0xc2083ffde8 sp=0xc2083ffd60
github.com/juju/juju/rpc.(*Conn).runRequest(0xc208472be0, 0x3fff7fc0ee38, 0xc208581500, 0x1286728, 0x3, 0xc2080a8460, 0x5, 0x1, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/rpc/server.go:552 +0x114 fp=0xc2083fff28 sp=0xc2083ffde8
runtime.goexit()
        /home/ubuntu/go/src/runtime/asm_ppc64x.s:1132 +0x4 fp=0xc2083fff28 sp=0xc2083fff28
created by github.com/juju/juju/rpc.(*Conn).handleRequest
        /home/ubuntu/src/github.com/juju/juju/rpc/server.go:481 +0x5f4"><pre class="notranslate"><code class="notranslate">unexpected fault address 0xf8010260f8010260
fatal error: fault
[signal 0xb code=0x1 addr=0xf8010260f8010260 pc=0x1e56c]

goroutine 229 [running]:
runtime.throw(0xf45d00, 0x5)
        /home/ubuntu/go/src/runtime/panic.go:543 +0x8c fp=0xc2083ff688 sp=0xc2083ff670
runtime.sigpanic()
        /home/ubuntu/go/src/runtime/sigpanic_unix.go:27 +0x2f0 fp=0xc2083ff6d8 sp=0xc2083ff688
runtime.convI2E(0xf8010260f8010258, 0x7c030378e8410070, 0x0, 0x0)
        /home/ubuntu/go/src/runtime/iface.go:256 +0x74 fp=0xc2083ff708 sp=0xc2083ff6e0
github.com/juju/juju/apiserver/agent.(*AgentAPIV0).StateServingInfo(0x31f5c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93 +0xa0 fp=0xc2083ff830 sp=0xc2083ff708
github.com/juju/juju/apiserver/agent.(*AgentAPIV1).StateServingInfo(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        &lt;autogenerated&gt;:22 +0x40 fp=0xc2083ff838 sp=0xc2083ff830
runtime.call128(0xc208205bd0, 0xe85098, 0xc208398380)
        /home/ubuntu/go/src/runtime/asm_ppc64x.s:413 +0x88 fp=0xc2083ff8c0 sp=0xc2083ff838
reflect.Value.call(0xe84f40, 0xc20802c5f0, 0x913, 0xf30060, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/go/src/reflect/value.go:432 +0xe30 fp=0xc2083ffc00 sp=0xc2083ff8c0
reflect.Value.Call(0xe84f40, 0xc20802c5f0, 0x913, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/ubuntu/go/src/reflect/value.go:300 +0x94 fp=0xc2083ffc60 sp=0xc2083ffc00
github.com/juju/juju/rpc/rpcreflect.newMethod.func6(0xe84f40, 0xc20802c5f0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/rpc/rpcreflect/type.go:326 +0x1a0 fp=0xc2083ffd60 sp=0xc2083ffc60
github.com/juju/juju/apiserver.(*srvCaller).Call(0xc208581500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/apiserver/root.go:131 +0xf0 fp=0xc2083ffde8 sp=0xc2083ffd60
github.com/juju/juju/rpc.(*Conn).runRequest(0xc208472be0, 0x3fff7fc0ee38, 0xc208581500, 0x1286728, 0x3, 0xc2080a8460, 0x5, 0x1, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/rpc/server.go:552 +0x114 fp=0xc2083fff28 sp=0xc2083ffde8
runtime.goexit()
        /home/ubuntu/go/src/runtime/asm_ppc64x.s:1132 +0x4 fp=0xc2083fff28 sp=0xc2083fff28
created by github.com/juju/juju/rpc.(*Conn).handleRequest
        /home/ubuntu/src/github.com/juju/juju/rpc/server.go:481 +0x5f4
</code></pre></div>
<p dir="auto">however <code class="notranslate">AgentAPIV1</code> is defined as</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="// AgentAPIV1 implements the version 1 of the API provided to an agent.
type AgentAPIV1 struct {
        *AgentAPIV0
}"><pre class="notranslate"><code class="notranslate">// AgentAPIV1 implements the version 1 of the API provided to an agent.
type AgentAPIV1 struct {
        *AgentAPIV0
}
</code></pre></div>
<p dir="auto">So the compiler has autogenerated <code class="notranslate">AgentAPIV1.StateServingInfo</code> and forwarded it to <code class="notranslate">AgentAPIV0.StateServingInfo</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="github.com/juju/juju/apiserver/agent.(*AgentAPIV0).StateServingInfo(0x31f5c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93 +0xa0 fp=0xc2083ff830 sp=0xc2083ff708
github.com/juju/juju/apiserver/agent.(*AgentAPIV1).StateServingInfo(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        &lt;autogenerated&gt;:22 +0x40 fp=0xc2083ff838 sp=0xc2083ff830"><pre class="notranslate"><code class="notranslate">github.com/juju/juju/apiserver/agent.(*AgentAPIV0).StateServingInfo(0x31f5c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93 +0xa0 fp=0xc2083ff830 sp=0xc2083ff708
github.com/juju/juju/apiserver/agent.(*AgentAPIV1).StateServingInfo(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        &lt;autogenerated&gt;:22 +0x40 fp=0xc2083ff838 sp=0xc2083ff830
</code></pre></div>
<p dir="auto">However, looking at the receiver value for <code class="notranslate">agent.(*AgentAPIV1).StateServingInfo</code> it is <code class="notranslate">0x00</code>, but in the next frame the receiver is <code class="notranslate">0x31f5c8</code>, this may be related to the problem.</p>
<p dir="auto">The faulting line,  /home/ubuntu/src/github.com/juju/juju/apiserver/agent/agent_v0.go:93, is</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="func (api *AgentAPIV0) StateServingInfo() (result state.StateServingInfo, err error) {
        auth := api.auth
        fmt.Printf(&quot;%#v\n&quot;, auth) // line 93"><pre class="notranslate"><code class="notranslate">func (api *AgentAPIV0) StateServingInfo() (result state.StateServingInfo, err error) {
        auth := api.auth
        fmt.Printf("%#v\n", auth) // line 93
</code></pre></div>
<p dir="auto">and at this point the value of <code class="notranslate">auth</code> is garbage, its interface type and value are random junk, <code class="notranslate">runtime.convI2E(0xf8010260f8010258, 0x7c030378e8410070, 0x0, 0x0)</code></p>
<p dir="auto">So far I have been able to determine that manually adding the forwarding method</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="func (api *AgentAPIV1) StateServingInfo() (result state.StateServingInfo, err error) {
        return api.AgentAPIV0.StateServingInfo()
}"><pre class="notranslate"><code class="notranslate">func (api *AgentAPIV1) StateServingInfo() (result state.StateServingInfo, err error) {
        return api.AgentAPIV0.StateServingInfo()
}
</code></pre></div>
<p dir="auto">Stops the panic, but I have not yet been able to construct a stand alone reproduction.</p>
<p dir="auto">Sadly this only happens on our ppc64le system, which is not accessible to others.</p>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/randall77/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/randall77">@randall77</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/minux/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/minux">@minux</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aclements/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aclements">@aclements</a></p>
<pre class="notranslate">What steps will reproduce the problem?

I'm starting a gotest of some code that uses lots of goroutines, but the process very
early (before even running any useful code).

I find this issue reproduces more readily on a specific machine x86_64, 8 cores, Fedora
14, 2.6.35.11-83.fc14.x86_64 and if I set GOMAXPROCS=500 and ulimit -u 256 although
neither of these should have any impact on anything.

What do you see instead?

Process hangs until I terminate it with SIGQUIT.

Which compiler are you using (5g, 6g, 8g, gccgo)?

6g

Which operating system are you using?

linux

Which revision are you using?  (hg identify)

tip

Please provide any additional information below.

I've attached the full stacktrace and strace output from a run that hangs.

SIGQUIT: quit
PC=0x414b15

runtime.futex+0x23 /home/alberts/go.hg/src/pkg/runtime/linux/amd64/sys.s:147
        runtime.futex()
futexsleep+0x50 /home/alberts/go.hg/src/pkg/runtime/linux/thread.c:52
        futexsleep(0x6698fc, 0x300000003, 0x414e85, 0x40ed9f)
futexlock+0x85 /home/alberts/go.hg/src/pkg/runtime/linux/thread.c:120
        futexlock(0x6698fc, 0x0)
runtime.notesleep+0x25 /home/alberts/go.hg/src/pkg/runtime/linux/thread.c:205
        runtime.notesleep(0x6698fc, 0x4da724fd)
runtime.stoptheworld+0x68 /home/alberts/go.hg/src/pkg/runtime/proc.c:426
        runtime.stoptheworld()
runtime.gc+0x15c /home/alberts/go.hg/src/pkg/runtime/mgc0.c:561
        runtime.gc(0xf800000000, 0x8)
runtime.mallocgc+0x1c3 /home/alberts/go.hg/src/pkg/runtime/malloc.c:90
        runtime.mallocgc(0x8, 0x100000000, 0x1, 0x267f900, 0x409a20, ...)</pre>
<p dir="auto">Attachments:</p>
<ol dir="auto">
<li><a href="https://storage.googleapis.com/go-attachment/1701/0/strace.txt" rel="nofollow">strace.txt</a> (28310 bytes)</li>
<li><a href="https://storage.googleapis.com/go-attachment/1701/0/stacktrace.txt" rel="nofollow">stacktrace.txt</a> (2804 bytes)</li>
</ol>