Get thread and task backtraces before terminating a worker on timeout

[kpamnany]

Implements the alternative described in #105 as an option.

Alternatively, we could dump julia task backtraces and/or CPU stacktraces.

I think a similar incantation can be assembled for lldb, to support Mac. We could support FreeBSD/OpenBSD since they have gdb, but I don't have a machine that would let me test. Note that Sys.isbsd() returns true for Mac.

Closing comment.

[kpamnany]

CI doesn't seem to have gdb so the tests aren't running. They pass locally, FWIW.

[nickrobinson251]

CI doesn't seem to have gdb so the tests aren't running

I think this is worth investigating a little before we merge this

also left a few minor suggestions

[nickrobinson251]

there is a lot going on here... i think we need a comment (maybe a long one)

perhaps we write this out over multiple lines so we can add comments about blocks of -ex commands, like

Suggested change

	gdb_cmd = `gdb -ex "handle SIGSEGV noprint nostop pass" -ex "handle SIGUSR2 noprint nostop pass" -ex "set pagination 0" -ex "set logging overwrite on" -ex "set logging file gdb.btall" -ex "set logging redirect on" -ex "set logging enabled on" -ex "thread apply all bt" -ex "set logging enabled off" -ex "call jl_print_task_backtraces(1)" --batch -p $(w.pid)`
	gdb_cmd = Cmd([
	"gdb",
	# comment here about SIGSEGV and SIGUSR2
	"-ex", "handle SIGSEGV noprint nostop pass",
	"-ex", "handle SIGUSR2 noprint nostop pass",
	"-ex", "set pagination 0",
	"-ex", "set logging overwrite on",
	"-ex", "set logging file gdb.btall",
	"-ex", "set logging redirect on",
	# comment here about getting traces for all threads
	"-ex", "set logging enabled on",
	"-ex", "thread apply all bt",
	"-ex", "set logging enabled off",
	# comment here...
	"-ex", "call jl_print_task_backtraces(1)",
	"--batch", "-p", "$(w.pid)"
	])

e.g. i'm thinking a block like this needs explaining (probably not to a gdb connoisseur but to some of us less familiar with this wizardry):

-ex "set logging enabled on" -ex "thread apply all bt" -ex "set logging enabled off"

[nickrobinson251]

is there a way to run CI on machines with gdb available? why don't the linux CI machines have it?

[kpamnany]

Yes, we need a step in the CI workflow that has:

      - name: Install GDB
        run: sudo apt-get install -y gdb

But CI.yml is written in a platform-independent way and I haven't figured out how to add that step for Ubuntu only. If you know, that would be great.

[nickrobinson251]

yeah, i think that just needs an if: line like

- name: Install GDB
  if: matrix.os == 'ubuntu-latest'
  run: sudo apt-get install -y gdb

not certain on the syntax, might need to be

- name: Install GDB
  if: ${{ matrix.os }} == 'ubuntu-latest'
  run: sudo apt-get install -y gdb

or

- name: Install GDB
  if: ${{ matrix.os == 'ubuntu-latest' }}
  run: sudo apt-get install -y gdb

[kpamnany]

I added it, but the CI machine does not allow gdb to attach to the process:

Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Inappropriate ioctl for device.

So I've backed it out. 🤷‍♂️

[kpamnany]

After much testing, it turns out that gdb is actually pretty terrible when the number of threads in the process is large. And maybe also when you run multiple gdbs at once?

In my latest (still running) test:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
10695 kpamnany  20   0 7165224   6.0g 208860 R 100.3  20.4  87:57.87 gdb
10712 kpamnany  20   0 7259628   6.1g 208852 R 100.3  20.7  86:42.86 gdb
    1 root      20   0    2276   1416   1412 S   0.0   0.0   0:00.10 init(Ubuntu)

Whatever the cause, using gdb is not practical and we'll need another way to get thread/task backtraces from Julia. I'm closing this, won't work.