The memory usage computation on MacOS seems to be wrong

[bvdmitri]

On MacOS I consistently get 99.9% mem usage, like in the following report (part of it)

test/distributions/normal_family/mv_normal_weighted_mean_precision_tests.jl:72
  Worker 2006:  10:52:43 | maxrss  2.7% | mem 99.7% | DONE  ( 32/187) test item "MvNormalWeightedMeanPrecision: vague" 0.3 secs (8.9% compile), 76.22 K allocs (4.366 MB)
  Worker 2008:  10:52:43 | maxrss  4.7% | mem 99.9% | DONE  ( 19/187) test item "InverseWishart: mean(::typeof(logdet))" 6.6 secs (12.3% compile, 9.5% GC), 16.84 M allocs (1.626 GB)
  Worker 2006:  10:52:43 | maxrss  2.7% | mem 99.9% | START ( 33/187) test item "MvNormalWeightedMeanPrecision: prod" at test/distributions/normal_family/mv_normal_weighted_mean_precision_tests.jl:92

which shows 99.9% memory use for all tests, even small and even those which do not allocate at all. GitHub Actions CI shows mem at about 30-40%.

In the system settings I can see that I have plenty of RAM available, more than 8GB (out of 16) are free during the tests. My guess is that mem computation is wrong on MacOS. This issue basically forces to use memory_threshold = 1.0 on MacOS, otherwise all workers are killed and restarted after each test.

julia> versioninfo()
Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (x86_64-apple-darwin22.4.0)
  CPU: 12 × Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 12 virtual cores

[nickrobinson251]

🤔 Could you try changing these lines to use Sys.free_physical_memory and Sys.total_physical_memory and see what that reports?

ReTestItems.jl/src/log_capture.jl

Lines 148 to 149 in fb8ddef

	maxrss_percent() = 100 * Float64(Sys.maxrss()/Sys.total_memory())
	memory_percent() = 100 * Float64(1 - (Sys.free_memory()/Sys.total_memory()))

[bvdmitri]

I tried the change but it didn't help. In a fresh Julia REPL without any dependencies I get this

➜  ReTestItems.jl git:(main) julia --startup-file=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.3 (2023-08-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> Float64(1 - (Sys.free_memory()/Sys.total_memory()))
0.8697254657745361

julia> Float64(1 - Sys.free_physical_memory()/Sys.total_physical_memory())
0.8693380355834961

AFAIK memory management system in MacOS is different from Linux and it can easily report 99% case, but it does not mean that the memory is not available, it simply means that most of its reserved for idle applications but it still can be used. Not sure if it can be fixed easily. Maybe it should be documented more clearly.

[nickrobinson251]

🤔

i'd love help on figuring this out!

personally i don't have much time to look into this right now, so i've at least opened a PR to change the default on Mac til this is fixed: #115