x/debug/cmd/viewcore: viewcore panics: bad int32 type uint32

[jordanlewis]

What version of Go are you using (go version)?

1.16.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
ubuntu@jordan-test-0001:~$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1750380533=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I attempted to use viewcore on a core dump from CockroachDB. I ran the CockroachDB binary, killed it with SIGSEGV, and collected the resultant dump. Then, I ran viewcore with the overview argument.

./go/bin/viewcore /mnt/data1/cores/core.cockroach.13188.jordan-test-0001.1620268383 --exe ./cockroach overview

What did you expect to see?

Viewcore output, which could hopefully help me track down some issues with memory usage. 😄

What did you see instead?

A panic:

ubuntu@jordan-test-0001:~$ ./go/bin/viewcore /mnt/data1/cores/core.cockroach.13188.jordan-test-0001.1620268383 --exe ./cockroach overview
panic: bad int32 type uint32

goroutine 1 [running]:
golang.org/x/debug/internal/gocore.region.Int32(0xc0004fa000, 0x777bdec, 0xc000182550, 0x86201a)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/region.go:82 +0xc6
golang.org/x/debug/internal/gocore.(*module).readFunc(0xc00f4cee70, 0xc0004fa000, 0x777bdd8, 0xc000aa86e0, 0xc0004fa000, 0x80ee0e8, 0xc000183d60, 0xc000182280)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/module.go:62 +0x22e
golang.org/x/debug/internal/gocore.(*Process).readModule(0xc0004fa000, 0xc0004fa000, 0x80ee080, 0xc000a58a50, 0x80ee080)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/module.go:45 +0x4cc
golang.org/x/debug/internal/gocore.(*Process).readModules(0xc0004fa000)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/module.go:26 +0x1a7
golang.org/x/debug/internal/gocore.Core(0xc000162000, 0x41, 0x0, 0x0)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/process.go:156 +0x2c8
main.readCore(0x78f037, 0xc00015c700, 0xc00011fd80, 0xc00011fda8)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/cmd/viewcore/main.go:266 +0xdb
main.runOverview(0xb05b80, 0xc000034260, 0x0, 0x2)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/cmd/viewcore/main.go:378 +0x34
github.com/spf13/cobra.(*Command).execute(0xb05b80, 0xc000034200, 0x2, 0x2, 0xb05b80, 0xc000034200)
        /home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xb05200, 0x0, 0xac76e0, 0xc00007a058)
        /home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2fe
github.com/spf13/cobra.(*Command).Execute(...)
        /home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/cmd/viewcore/main.go:244 +0x125

Here is a link to the core dump (334 MB): https://drive.google.com/file/d/1KlwSvP-5xxghBRHprxsmtFFQtsK9zAmg/view?usp=sharing
Here is a link to the binary used (178 MB): https://drive.google.com/file/d/147h4wTi7WloFWPjIXm4kZoIGPNvu6QNz/view?usp=sharing

I know this is a bit of a heavy repro, but I'm not quite sure how else to track down the issue. Any ideas would be appreciated for how to produce a smaller case for you to look at.

@randall77, would you have an idea about this issue? I saw #38638, but this appears to be a different issue.

Thank you!

[cagedmantis]

/cc @hyangah @randall77

[jordanlewis]

This happens even with a trivial program, which I should have checked first.

package main

func main() {
    panic("no")
}

If I run this with GOTRACEBACK=crash, the resultant core crashes:

[22:24:2]% viewcore core
panic: bad int32 type uint32

goroutine 1 [running]:
golang.org/x/debug/internal/gocore.region.Int32(0xc00016a580, 0x4b50fc, 0xc00014a2d0, 0x86201a)
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/region.go:82 +0xc6
golang.org/x/debug/internal/gocore.(*module).readFunc(0xc0003cdb00, 0xc00016a580, 0x4b50e8, 0xc000260aa0, 0xc00016a580, 0x4c8888, 0xc0000c5810, 0xc00014a0a0)
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/module.go:62 +0x22e
golang.org/x/debug/internal/gocore.(*Process).readModule(0xc00016a580, 0xc00016a580, 0x4c8820, 0xc0000c5ea0, 0x4c8820)
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/module.go:45 +0x4cc
golang.org/x/debug/internal/gocore.(*Process).readModules(0xc00016a580)
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/module.go:26 +0x1a7
golang.org/x/debug/internal/gocore.Core(0xc000192000, 0x4, 0x0, 0x0)
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/internal/gocore/process.go:156 +0x2c8
main.readCore(0x747c25, 0x7f3820, 0xc00011afc0, 0x862026)
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/cmd/viewcore/main.go:266 +0xdb
main.runRoot(0xb05200, 0xb3c0d0, 0x0, 0x0)
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/cmd/viewcore/main.go:288 +0x55
github.com/spf13/cobra.(*Command).execute(0xb05200, 0xc000138010, 0x0, 0x0, 0xb05200, 0xc000138010)
        /home/jordan/go/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xb05200, 0x0, 0xac76e0, 0xc00010e058)
        /home/jordan/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2fe
github.com/spf13/cobra.(*Command).Execute(...)
        /home/jordan/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
        /home/jordan/go/pkg/mod/golang.org/x/[email protected]/cmd/viewcore/main.go:244 +0x125

[jordanlewis]

The problem here seems to be that the types of npcdata and pscp were changed in _func to be uint32.

https://github.com/golang/go/blame/3b304ce7fe35b9d1e8cf0b0518ed2550c361a010/src/runtime/runtime2.go#L876

Fixing that locally seems to improve matters! But, there are still some things broken even after patching debug to expect uint32 instead of int32 on those slots.

diff --git a/internal/gocore/module.go b/internal/gocore/module.go
index 2f5e632..95c10b7 100644
--- a/internal/gocore/module.go
+++ b/internal/gocore/module.go
@@ -59,23 +59,23 @@ func (m *module) readFunc(r region, pcln region) *Func {
        f := &Func{module: m, r: r}
        f.entry = core.Address(r.Field("entry").Uintptr())
        f.name = r.p.proc.ReadCString(pcln.SliceIndex(int64(r.Field("nameoff").Int32())).a)
-       f.frameSize.read(r.p.proc, pcln.SliceIndex(int64(r.Field("pcsp").Int32())).a)
+       f.frameSize.read(r.p.proc, pcln.SliceIndex(int64(r.Field("pcsp").Uint32())).a)

        // Parse pcdata and funcdata, which are laid out beyond the end of the _func.
        a := r.a.Add(int64(r.p.findType("runtime._func").Size))
-       n := r.Field("npcdata").Int32()
-       for i := int32(0); i < n; i++ {
+       n := r.Field("npcdata").Uint32()
+       for i := uint32(0); i < n; i++ {
                f.pcdata = append(f.pcdata, r.p.proc.ReadInt32(a))
                a = a.Add(4)
        }
        a = a.Align(r.p.proc.PtrSize())

        if nfd := r.Field("nfuncdata"); nfd.typ.Size == 1 { // go 1.12 and beyond, this is a uint8
-               n = int32(nfd.Uint8())
+               n = uint32(nfd.Uint8())
        } else { // go 1.11 and earlier, this is an int32
-               n = nfd.Int32()
+               n = uint32(nfd.Int32())
        }
-       for i := int32(0); i < n; i++ {
+       for i := uint32(0); i < n; i++ {
                f.funcdata = append(f.funcdata, r.p.proc.ReadPtr(a))
                a = a.Add(r.p.proc.PtrSize())

Some cores give messages like this:

warning: giving up on backtrace: cannot read frame size at pc=0x465661: can't find pctab entry for offset 0x21
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1

Some cores fail to load altogether:

warning: giving up on backtrace: cannot find func for pc=0x99e0ffffffffff
panic: address 0 is not mapped in the core file

goroutine 1 [running]:
golang.org/x/debug/internal/core.(*Process).ReadUint32(0xc0001b2000, 0x0, 0xc0003c5020)
        /home/jordan/go/src/github.com/golang/debug/internal/core/read.go:63 +0x24c
golang.org/x/debug/internal/core.(*Process).ReadInt32(...)
        /home/jordan/go/src/github.com/golang/debug/internal/core/read.go:107
golang.org/x/debug/internal/gocore.region.Int32(0xc00018e840, 0x0, 0xc0000c7a90, 0x861ab7)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/region.go:84 +0x4e
golang.org/x/debug/internal/gocore.(*Process).readFrame(0xc00018e840, 0xc000084f90, 0x430705, 0x430705, 0x2, 0xc00018e840)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:662 +0x7f6
golang.org/x/debug/internal/gocore.(*Process).readG(0xc00018e840, 0xc00018e840, 0xc000000600, 0xc0000c7e50, 0xc000000600)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:603 +0x595
golang.org/x/debug/internal/gocore.(*Process).readGs(0xc00018e840)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:550 +0x12d
golang.org/x/debug/internal/gocore.Core(0xc0001b2000, 0x32, 0x0, 0x0)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:158 +0x2e5
main.readCore(0x747c25, 0x7f3820, 0xc0000d2fc0, 0x862026)
        /home/jordan/go/src/github.com/golang/debug/cmd/viewcore/main.go:266 +0xdb
main.runRoot(0xb04200, 0xb3b0d0, 0x0, 0x0)
        /home/jordan/go/src/github.com/golang/debug/cmd/viewcore/main.go:288 +0x55
github.com/spf13/cobra.(*Command).execute(0xb04200, 0xc000020050, 0x0, 0x0, 0xb04200, 0xc000020050)
        /home/jordan/go/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xb04200, 0x0, 0xac66e0, 0xc0000c8058)
        /home/jordan/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2fe
github.com/spf13/cobra.(*Command).Execute(...)
        /home/jordan/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
        /home/jordan/go/src/github.com/golang/debug/cmd/viewcore/main.go:244 +0x125

I'm wondering, perhaps there are some more updates that need to be applied for a more recent version of Go here?

[jordanlewis]

I have submitted a PR to fix this problem and another. Please take a look: golang/debug#7

[aktau]

Should this be closed, then?