Project

General

Profile

Bug #3106

go from pkg install hanging when running tests

Added by driusan about 2 months ago. Updated about 2 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Vendor software
Target version:
Start date:
11/29/2017
Due date:
% Done:

0%


Description

I reinstalled Dragonfly 5.0.1 and after re-installing Go with "pkg install go" (which installed go version 1.9) and running "go test" now seems to randomly freeze blocking for IO. It worked reliably on this hardware previously to this reinstall (it was running HAMMER2 when it was working, now it's running HAMMER. I don't think that's relevant because as far as I know Go uses /tmp which is tmpfs for its tests anyways. Unfortunately, I can't recall what version of Go was on it..)

I tried compiling 1.9.2 from source to see if it's something that's been fixed but just hasn't made it to dports yet, but there's enough tests in the go compiler that whenever I try at least one of them freezes and after a few minutes fails dumping a stack trace like the following:

goroutine 53 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x800788f20, 0x72, 0x1)
/home/driusan/go1.9.2/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42040bb98, 0x72, 0xffffffffffffff01, 0x725b40, 0x7241e0)
/home/driusan/go1.9.2/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42040bb98, 0xc4204b4201, 0x200, 0x200)
/home/driusan/go1.9.2/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42040bb80, 0xc4204b4200, 0x200, 0x200, 0x0, 0x0, 0x0)
/home/driusan/go1.9.2/src/internal/poll/fd_unix.go:126 +0x18a
os.(*File).read(0xc4203c4df0, 0xc4204b4200, 0x200, 0x200, 0x0, 0xc42047cde8, 0x4ce4cb)
/home/driusan/go1.9.2/src/os/file_unix.go:216 +0x4e
os.(*File).Read(0xc4203c4df0, 0xc4204b4200, 0x200, 0x200, 0x0, 0x0, 0x0)
/home/driusan/go1.9.2/src/os/file.go:103 +0x6d
bytes.(*Buffer).ReadFrom(0xc4200c4380, 0x725640, 0xc4203c4df0, 0x8007890c8, 0xc4200c4380, 0xc42047cf01)
/home/driusan/go1.9.2/src/bytes/buffer.go:209 +0x177
io.copyBuffer(0x725400, 0xc4200c4380, 0x725640, 0xc4203c4df0, 0x0, 0x0, 0x0, 0xc4200d00f0, 0x0, 0x0)
/home/driusan/go1.9.2/src/io/io.go:386 +0x2bb
io.Copy(0x725400, 0xc4200c4380, 0x725640, 0xc4203c4df0, 0x403663, 0xc4200aa000, 0xc42047cfb0)
/home/driusan/go1.9.2/src/io/io.go:362 +0x68
os/exec.(*Cmd).writerDescriptor.func1(0xc4200aa000, 0xc42047cfb0)
/home/driusan/go1.9.2/src/os/exec/exec.go:264 +0x4d
os/exec.(*Cmd).Start.func1(0xc4204f0580, 0xc42019d820)
/home/driusan/go1.9.2/src/os/exec/exec.go:380 +0x27
created by os/exec.(*Cmd).Start
/home/driusan/go1.9.2/src/os/exec/exec.go:379 +0x646
FAIL go/internal/gcimporter 180.018s

Which test freezes and eventually dies isn't deterministic as far as I can tell, but a stack trace like that with an IO timing out is pretty consistent. (Other threads also panic with timeouts but I've only included one here..)

I'm not sure if this bug report belongs with DragonFly or Go (or what details would be useful to help reproduce it), but since it's a (relatively) clean install of DragonFly with a go version installed from pkg, I thought I'd start here..

stderr - more stack traces of threads that seem to be deadlocked or somehow blocked (7.43 KB) driusan, 11/29/2017 07:38 PM

ktrace.out.xz - ktrace of a failing "go test" run. (619 KB) driusan, 11/30/2017 04:59 AM

History

#1 Updated by sepherosa about 2 months ago

Possible to get a ktrace?

On Thu, Nov 30, 2017 at 10:16 AM,
<> wrote:
> Issue #3106 has been reported by driusan.
>
> ----------------------------------------
> Bug #3106: go from pkg install hanging when running tests
> http://bugs.dragonflybsd.org/issues/3106
>
> * Author: driusan
> * Status: New
> * Priority: Normal
> * Assignee:
> * Category: Vendor software
> * Target version: Latest stable
> ----------------------------------------
> I reinstalled Dragonfly 5.0.1 and after re-installing Go with "pkg install go" (which installed go version 1.9) and running "go test" now seems to randomly freeze blocking for IO. It worked reliably on this hardware previously to this reinstall (it was running HAMMER2 when it was working, now it's running HAMMER. I don't think that's relevant because as far as I know Go uses /tmp which is tmpfs for its tests anyways. Unfortunately, I can't recall what version of Go was on it..)
>
> I tried compiling 1.9.2 from source to see if it's something that's been fixed but just hasn't made it to dports yet, but there's enough tests in the go compiler that whenever I try at least one of them freezes and after a few minutes fails dumping a stack trace like the following:
>
> goroutine 53 [IO wait, 2 minutes]:
> internal/poll.runtime_pollWait(0x800788f20, 0x72, 0x1)
> /home/driusan/go1.9.2/src/runtime/netpoll.go:173 +0x57
> internal/poll.(*pollDesc).wait(0xc42040bb98, 0x72, 0xffffffffffffff01, 0x725b40, 0x7241e0)
> /home/driusan/go1.9.2/src/internal/poll/fd_poll_runtime.go:85 +0xae
> internal/poll.(*pollDesc).waitRead(0xc42040bb98, 0xc4204b4201, 0x200, 0x200)
> /home/driusan/go1.9.2/src/internal/poll/fd_poll_runtime.go:90 +0x3d
> internal/poll.(*FD).Read(0xc42040bb80, 0xc4204b4200, 0x200, 0x200, 0x0, 0x0, 0x0)
> /home/driusan/go1.9.2/src/internal/poll/fd_unix.go:126 +0x18a
> os.(*File).read(0xc4203c4df0, 0xc4204b4200, 0x200, 0x200, 0x0, 0xc42047cde8, 0x4ce4cb)
> /home/driusan/go1.9.2/src/os/file_unix.go:216 +0x4e
> os.(*File).Read(0xc4203c4df0, 0xc4204b4200, 0x200, 0x200, 0x0, 0x0, 0x0)
> /home/driusan/go1.9.2/src/os/file.go:103 +0x6d
> bytes.(*Buffer).ReadFrom(0xc4200c4380, 0x725640, 0xc4203c4df0, 0x8007890c8, 0xc4200c4380, 0xc42047cf01)
> /home/driusan/go1.9.2/src/bytes/buffer.go:209 +0x177
> io.copyBuffer(0x725400, 0xc4200c4380, 0x725640, 0xc4203c4df0, 0x0, 0x0, 0x0, 0xc4200d00f0, 0x0, 0x0)
> /home/driusan/go1.9.2/src/io/io.go:386 +0x2bb
> io.Copy(0x725400, 0xc4200c4380, 0x725640, 0xc4203c4df0, 0x403663, 0xc4200aa000, 0xc42047cfb0)
> /home/driusan/go1.9.2/src/io/io.go:362 +0x68
> os/exec.(*Cmd).writerDescriptor.func1(0xc4200aa000, 0xc42047cfb0)
> /home/driusan/go1.9.2/src/os/exec/exec.go:264 +0x4d
> os/exec.(*Cmd).Start.func1(0xc4204f0580, 0xc42019d820)
> /home/driusan/go1.9.2/src/os/exec/exec.go:380 +0x27
> created by os/exec.(*Cmd).Start
> /home/driusan/go1.9.2/src/os/exec/exec.go:379 +0x646
> FAIL go/internal/gcimporter 180.018s
>
> Which test freezes and eventually dies isn't deterministic as far as I can tell, but a stack trace like that with an IO timing out is pretty consistent. (Other threads also panic with timeouts but I've only included one here..)
>
> I'm not sure if this bug report belongs with DragonFly or Go (or what details would be useful to help reproduce it), but since it's a (relatively) clean install of DragonFly with a go version installed from pkg, I thought I'd start here..
>
>
>
> --
> You have received this notification because you have either subscribed to it, or are involved in it.
> To change your notification preferences, please click here: http://bugs.dragonflybsd.org/my/account

--
Tomorrow Will Never Die

#2 Updated by driusan about 2 months ago

I can't attach a ktrace of the entire go toolchain build/test. I tried taking one of "ktrace go test -v" for my personal project written in Go where I initially encountered the problem since it's smaller but still fails with the same time out/stack trace as above in the panic, but it weighs in at 5.9Mb and there seems to be a 5Mb attachment limit.. I can attach it if there's some way to increase the limit by 1Mb, for now I've just attached stderr for the run I have a ktrace.out for as another slightly more detailed data point.

(This test suite took ~20 seconds on my previous install of DragonFly, now it's timing out after 10 minutes..)

#3 Updated by sepherosa about 2 months ago

Can you try attaching xz'ed ktrace output?

On Thu, Nov 30, 2017 at 11:48 AM,
<> wrote:
> Issue #3106 has been updated by driusan.
>
> File stderr added
>
> I can't attach a ktrace of the entire go toolchain build/test. I tried taking one of "ktrace go test -v" for my personal project written in Go where I initially encountered the problem since it's smaller but still fails with the same time out/stack trace as above in the panic, but it weighs in at 5.9Mb and there seems to be a 5Mb attachment limit.. I can attach it if there's some way to increase the limit by 1Mb, for now I've just attached stderr for the run I have a ktrace.out for as another slightly more detailed data point.
>
> (This test suite took ~20 seconds on my previous install of DragonFly, now it's timing out after 10 minutes..)
>
> ----------------------------------------
> Bug #3106: go from pkg install hanging when running tests
> http://bugs.dragonflybsd.org/issues/3106#change-13332
>
> * Author: driusan
> * Status: New
> * Priority: Normal
> * Assignee:
> * Category: Vendor software
> * Target version: Latest stable
> ----------------------------------------
> I reinstalled Dragonfly 5.0.1 and after re-installing Go with "pkg install go" (which installed go version 1.9) and running "go test" now seems to randomly freeze blocking for IO. It worked reliably on this hardware previously to this reinstall (it was running HAMMER2 when it was working, now it's running HAMMER. I don't think that's relevant because as far as I know Go uses /tmp which is tmpfs for its tests anyways. Unfortunately, I can't recall what version of Go was on it..)
>
> I tried compiling 1.9.2 from source to see if it's something that's been fixed but just hasn't made it to dports yet, but there's enough tests in the go compiler that whenever I try at least one of them freezes and after a few minutes fails dumping a stack trace like the following:
>
> goroutine 53 [IO wait, 2 minutes]:
> internal/poll.runtime_pollWait(0x800788f20, 0x72, 0x1)
> /home/driusan/go1.9.2/src/runtime/netpoll.go:173 +0x57
> internal/poll.(*pollDesc).wait(0xc42040bb98, 0x72, 0xffffffffffffff01, 0x725b40, 0x7241e0)
> /home/driusan/go1.9.2/src/internal/poll/fd_poll_runtime.go:85 +0xae
> internal/poll.(*pollDesc).waitRead(0xc42040bb98, 0xc4204b4201, 0x200, 0x200)
> /home/driusan/go1.9.2/src/internal/poll/fd_poll_runtime.go:90 +0x3d
> internal/poll.(*FD).Read(0xc42040bb80, 0xc4204b4200, 0x200, 0x200, 0x0, 0x0, 0x0)
> /home/driusan/go1.9.2/src/internal/poll/fd_unix.go:126 +0x18a
> os.(*File).read(0xc4203c4df0, 0xc4204b4200, 0x200, 0x200, 0x0, 0xc42047cde8, 0x4ce4cb)
> /home/driusan/go1.9.2/src/os/file_unix.go:216 +0x4e
> os.(*File).Read(0xc4203c4df0, 0xc4204b4200, 0x200, 0x200, 0x0, 0x0, 0x0)
> /home/driusan/go1.9.2/src/os/file.go:103 +0x6d
> bytes.(*Buffer).ReadFrom(0xc4200c4380, 0x725640, 0xc4203c4df0, 0x8007890c8, 0xc4200c4380, 0xc42047cf01)
> /home/driusan/go1.9.2/src/bytes/buffer.go:209 +0x177
> io.copyBuffer(0x725400, 0xc4200c4380, 0x725640, 0xc4203c4df0, 0x0, 0x0, 0x0, 0xc4200d00f0, 0x0, 0x0)
> /home/driusan/go1.9.2/src/io/io.go:386 +0x2bb
> io.Copy(0x725400, 0xc4200c4380, 0x725640, 0xc4203c4df0, 0x403663, 0xc4200aa000, 0xc42047cfb0)
> /home/driusan/go1.9.2/src/io/io.go:362 +0x68
> os/exec.(*Cmd).writerDescriptor.func1(0xc4200aa000, 0xc42047cfb0)
> /home/driusan/go1.9.2/src/os/exec/exec.go:264 +0x4d
> os/exec.(*Cmd).Start.func1(0xc4204f0580, 0xc42019d820)
> /home/driusan/go1.9.2/src/os/exec/exec.go:380 +0x27
> created by os/exec.(*Cmd).Start
> /home/driusan/go1.9.2/src/os/exec/exec.go:379 +0x646
> FAIL go/internal/gcimporter 180.018s
>
> Which test freezes and eventually dies isn't deterministic as far as I can tell, but a stack trace like that with an IO timing out is pretty consistent. (Other threads also panic with timeouts but I've only included one here..)
>
> I'm not sure if this bug report belongs with DragonFly or Go (or what details would be useful to help reproduce it), but since it's a (relatively) clean install of DragonFly with a go version installed from pkg, I thought I'd start here..
>
> ---Files--------------------------------
> stderr (7.43 KB)
>
>
> --
> You have received this notification because you have either subscribed to it, or are involved in it.
> To change your notification preferences, please click here: http://bugs.dragonflybsd.org/my/account

--
Tomorrow Will Never Die

#4 Updated by t_dfbsd about 2 months ago

When you reinstalled 5.0.1, did you make any changes to the install aside from switching from HAMMER2 to HAMMER?

#5 Updated by mneumann about 2 months ago

Have you tried master? Dillon fixed a pthread/libc bug, which lead to hangs in cargo (rust package manager):

https://gitweb.dragonflybsd.org/dragonfly.git/commit/e2caf0e77d5adc5ddec07633ec1c9ceaae464acf

#6 Updated by driusan about 2 months ago

It went from being a system that was upgraded from 4.5 to a fresh install of 5.0.1, so any changes that would have accumulated over the time it was installed, but nothing's configured differently other than going back to hammer1.

I've attached the ktrace. I don't know why I didn't think of compressing it before..

I'll try master tonight, "to avoid corruption when heavily threaded programs call fork()." sounds promising..

#7 Updated by driusan about 2 months ago

  • Status changed from New to Closed

Works after updating to master. Thanks!

Also available in: Atom PDF