Skip to content

Commit

Permalink
Close probe finding pool even if no work is submitted
Browse files Browse the repository at this point in the history
Previously, probe.close_probe_finding_pool() only closed
the multiprocessing.Pool if work was submitted to the pool
(see previous comment for why -- this was to avoid a bug
in Python 2.7).

This seems to be causing issues. Travis CI is sporadically
failing on the test `test_probe.test_too_short_sequence_large_k`
with `OSError: [Errno 12] Cannot allocate memory`. This test
uses very little memory. Instead, this is probably occurring
because creating a multiprocessing.Pool (when opening the
probe finding pool) is unable to call `os.fork()` due to there
being too many open file descriptors. Many file descriptors are
not closed if no work was submitted to an open pool, because
pool.close_probe_finding_pool() deliberately did not close
its pool in this case. This unit test, as well as others,
open probe finding pools but do not submit any work to them.
As a result, they leave open file descriptors.

We no longer have to worry about the bug in Python 2.7 because
we test only in Python 3. This change attempts to fix the
sporadic failures in Travis CI.
  • Loading branch information
haydenm committed Jan 11, 2019
1 parent d33fda1 commit 80641b2
Showing 1 changed file with 2 additions and 4 deletions.
6 changes: 2 additions & 4 deletions catch/probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -932,6 +932,8 @@ def close_probe_finding_pool():
del _pfp_kmer_probe_map_native
del _pfp_kmer_probe_map_use_native

_pfp_pool.close()

# In Python versions earlier than 2.7.3 there is a bug (see
# http://bugs.python.org/issue12157) that occurs if a pool p is
# created and p.join() is called, but p.map() is never called (i.e.,
Expand All @@ -941,11 +943,7 @@ def close_probe_finding_pool():
# but find_probe_covers_in_sequence() is never called; the variable
# _pfp_work_was_submitted ensures that join() is only called on
# the pool if work was indeed submitted.
# Similarly, when no work is submitted, a call to p.close() may yield
# a RuntimeError that is printed but ignored; so only call close()
# when work was indeed submitted.
if _pfp_work_was_submitted:
_pfp_pool.close()
# Due to issues that likely stem from bugs in the multiprocessing
# module, calls to _pfp_pool.terminate() and _pfp_pool.join()
# sometimes hang indefinitely (even when work was indeed submitted
Expand Down

0 comments on commit 80641b2

Please sign in to comment.