Skip to content

Commit

Permalink
Remove deadlock from qa_contact fixture.
Browse files Browse the repository at this point in the history
The qa_contact.py is spawning `git blame` to get some info about the code.
Right after spawning it, `wait()` is called to wait for the `git blame`
to exit. But no read is being done between these two actions. This
creates a danger for git blame to fill the stdout buffer and get stopped.
The parent process is waiting as well on this point so no read from the
git blame's stdout pipe will ever be done and no process is allowed to
advance -> DEADLOCK.

In fact, even the Python doc does warn about this situation:
https://docs.python.org/3.7/library/subprocess.html#subprocess.Popen.wait

pytest being blocked:
Traceback (most recent call first):
  <built-in method waitpid of module object at remote 0x7efcd68d7350>
  File "/usr/lib64/python3.7/subprocess.py", line 1582, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
  File "/usr/lib64/python3.7/subprocess.py", line 1624, in _wait
    (pid, sts) = self._try_wait(0)
  File "/usr/lib64/python3.7/subprocess.py", line 990, in wait
    return self._wait(timeout=timeout)
  File "/cfme/cfme_tests/cfme/fixtures/qa_contact.py", line 20, in dig_code
    proc.wait()
  File "/cfme/cfme_tests/cfme/fixtures/qa_contact.py", line 43, in pytest_runtest_teardown
    results = dig_code(item)
  File "/cfme/cfme_tests/.cfme_venv/lib64/python3.7/site-packages/pluggy/callers.py", line 180, in _multicall

 git blame being blocked:
 (gdb) bt
 #0  0x00007f70c93340f8 in __GI___libc_write (fd=1, buf=0x56289f2a1620, nbytes=1799)
     at ../sysdeps/unix/sysv/linux/write.c:26
 #1  0x00007f70c92c4d4d in _IO_new_file_write (f=0x7f70c9409780 <_IO_2_1_stdout_>, data=0x56289f2a1620,
    n=1799) at fileops.c:1183
 #2  0x00007f70c92c40a6 in new_do_write (fp=0x7f70c9409780 <_IO_2_1_stdout_>,
     data=0x56289f2a1620 " +0200 553)     with LogValidator(evm_log,\ne4df8147c6 (<[email protected]>        2019-08-05 16:03:32 +0200 554)", ' ' <repeats 23 times>, "matched_patterns=['Starting to execute failover'],\ne4df8147c6 (<"..., to_do=to_do@entry=1799) at libioP.h:904
 ManageIQ#3  0x00007f70c92c5e99 in _IO_new_do_write (to_do=1799, data=<optimized out>, fp=<optimized out>)
     at fileops.c:430
 ManageIQ#4  _IO_new_do_write (fp=<optimized out>, data=<optimized out>, to_do=1799) at fileops.c:430
 ManageIQ#5  0x00007f70c92c7a5a in _IO_flush_all_lockp (do_lock=do_lock@entry=0) at libioP.h:904
 ManageIQ#6  0x00007f70c92c7c69 in _IO_cleanup () at genops.c:858
 ManageIQ#7  0x00007f70c9282722 in __run_exit_handlers (status=status@entry=0, listp=<optimized out>,
     run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:130
 ManageIQ#8  0x00007f70c92827b0 in __GI_exit (status=status@entry=0) at exit.c:139
 ManageIQ#9  0x000056289dc303db in handle_builtin (argc=<optimized out>, argv=<optimized out>) at git.c:440
 ManageIQ#10 0x000056289dc312a4 in run_argv (argv=0x7ffc54799c40, argcp=0x7ffc54799c4c) at git.c:702
 ManageIQ#11 cmd_main (argc=<optimized out>, argv=<optimized out>) at git.c:799
 ManageIQ#12 0x000056289dc2ff34 in main (argc=5, argv=0x7ffc54799eb8) at common-main.c:45
 (gdb)

 The reproducer:

    from unittest.mock import MagicMock
    from cfme.tests.cli.test_appliance_console_db_restore import get_ha_appliances_with_providers
    from cfme.fixtures.qa_contact import dig_code

    m=MagicMock()
    m.function = get_ha_appliances_with_providers
    print("Deadlock may happen now!")
    print(dig_code(m))

will output (depending on git blame output) something like:
Deadlock may happen now!
('[email protected]', 91.04477611940298), ('[email protected]', ...

...and ends when not reproduced.

When reproduced, it will not output anything after:
    Deadlock may happen now!
  • Loading branch information
Jaroslav Henner committed Aug 22, 2019
1 parent c6913cc commit f10903b
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions cfme/fixtures/qa_contact.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,8 @@ def dig_code(node):
line_param = '-L {},+{}'.format(lineno, offset)
cmd_params = ['git', 'blame', line_param, filename, '--show-email']

proc = subprocess.Popen(cmd_params, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.wait()
lc_info = proc.stdout.readlines()
proc = subprocess.run(cmd_params, stdin=subprocess.DEVNULL, capture_output=True)
lc_info = proc.stdout.decode().strip().split('\n')
contact_stats = defaultdict(int)
for line in lc_info:
contact = re.findall(r'.{8} \(\<(.*?)\> ', line)
Expand Down

0 comments on commit f10903b

Please sign in to comment.