Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lagrangian subgrid bubble model #749

Merged
merged 42 commits into from
Dec 24, 2024
Merged

Conversation

dgvacarevelo
Copy link
Contributor

@dgvacarevelo dgvacarevelo commented Nov 28, 2024

Description

The Euler-Lagrange model for sub-grid bubble dynamics is based on the volume-averaged equations
of motion to describe the dynamics of a mixture of dispersed bubbles and a compressible liquid.

Type of change

  • New feature (non-breaking change which adds functionality)

Scope

  • This PR comprises a set of related changes with a common goal

If you cannot check the above box, please split your PR into multiple PRs that each have a common goal.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.
Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

  • Test A: Bubble screen problem
    A cuboidal bubble cloud interacts with a sinusoidal planar wave. This case is included as an example case (examples/3D_lag_bubbles_bubblescreen). The pressure at the center of the cloud is measured and compared, as follows.
image
  • Test B: Singel bubble collapse
    A single bubble of 8 micrometers encounters a sinusoidal planar wave. This case is also included as an example case (examples/3D_lag_bubbles_shbubcollapse). The radius evolution of the bubble is compared with the experiment reported by (Ohl et. al, 1999) (https://doi.org/10.1098/rsta.1999.0327).
image

Test Configuration:

  • What computers and compilers did you use to test this:
  1. Delta: I tested both problems with CPUs (gcc/11.4.0, openmpi/4.1.6) and GPUs (nvhpc/24.1, openmpi/4.1.5+cuda, cuda/12.3.0, cmake/3.27.9). I used the NVIDIA A100 to run my GPU cases.
  2. Turing (WPI HPC): Here, I used the following CPU compilers: gcc/12.1.0, openmpi/4.1.3, python/3.11.6

Checklist

  • I have added comments for the new code
  • I added Doxygen docstrings to the new code
  • I have made corresponding changes to the documentation (docs/)
  • I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
  • I have added example cases in examples/ that demonstrate my new feature performing as expected.
    They run to completion and demonstrate "interesting physics"
  • I ran ./mfc.sh format before committing my code
  • New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled. Here I only had access to NVIDIA hardware with NVHPC compilers.
  • This PR does not introduce any repeated code (it follows the DRY principle)
  • I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

  • Checked that the code compiles using NVHPC compilers
  • Checked that the code compiles using CRAY compilers
  • Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
  • Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
  • Enclosed the new feature via nvtx ranges so that they can be identified in profiles
  • Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
  • Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
  • Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

Copy link

codecov bot commented Nov 29, 2024

Codecov Report

Attention: Patch coverage is 34.23332% with 1291 lines in your changes missing coverage. Please review.

Project coverage is 43.80%. Comparing base (1d58b45) to head (6f739e1).
Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
src/simulation/m_bubbles_EL.fpp 31.18% 558 Missing and 60 partials ⚠️
src/post_process/m_data_input.f90 19.71% 136 Missing and 31 partials ⚠️
src/simulation/m_bubbles_EL_kernels.fpp 26.70% 107 Missing and 11 partials ⚠️
src/simulation/m_bubbles_EE.fpp 49.06% 90 Missing and 19 partials ⚠️
src/post_process/m_mpi_proxy.fpp 0.00% 71 Missing and 8 partials ⚠️
src/simulation/m_time_steppers.fpp 43.75% 41 Missing and 13 partials ⚠️
src/simulation/m_data_output.fpp 43.18% 19 Missing and 6 partials ⚠️
src/post_process/m_data_output.fpp 67.85% 11 Missing and 7 partials ⚠️
src/simulation/m_rhs.fpp 29.41% 2 Missing and 10 partials ⚠️
src/simulation/m_start_up.fpp 42.85% 4 Missing and 8 partials ⚠️
... and 19 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #749      +/-   ##
==========================================
- Coverage   44.72%   43.80%   -0.92%     
==========================================
  Files          62       65       +3     
  Lines       17555    19011    +1456     
  Branches     2044     2313     +269     
==========================================
+ Hits         7851     8328     +477     
- Misses       8439     9276     +837     
- Partials     1265     1407     +142     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sbryngelson
Copy link
Member

Thanks! This is failing some tests, like Frontier w/ CCE18 and AMD GPUs (MI250X). I'm not sure if you have access to this machine to debug yourself @dgvacarevelo

@dgvacarevelo
Copy link
Contributor Author

@sbryngelson No, I don't have access to Frontier. I implemented the GPU acceleration using DELTA through ACCESS resources.

@sbryngelson
Copy link
Member

sbryngelson commented Nov 29, 2024

Requesting code review from @anshgupta1234 @wilfonba @haochey @ChrisZYJ and Frontier help from @wilfonba and @anandrdbz

@sbryngelson sbryngelson added the enhancement New feature or request label Nov 29, 2024
Copy link
Contributor

@wilfonba wilfonba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice and well put together PR @dgvacarevelo! I made a few minor comments, but overall every thing looks good. I'll look into the Frontier issues when I get a chance. Hopefully this weekend.

src/simulation/m_acoustic_src.fpp Outdated Show resolved Hide resolved
src/simulation/m_global_parameters.fpp Outdated Show resolved Hide resolved
src/simulation/m_kernel_functions.fpp Outdated Show resolved Hide resolved
src/simulation/m_start_up.fpp Outdated Show resolved Hide resolved
@sbryngelson sbryngelson requested review from anandrdbz and removed request for haochey, ChrisZYJ and anandrdbz December 23, 2024 23:55
@sbryngelson sbryngelson merged commit 06f4418 into MFlowCode:master Dec 24, 2024
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

Successfully merging this pull request may close these issues.

4 participants