diff --git a/README.md b/README.md index eaad666..3ca0dda 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,37 @@ -# NextGen In A Box (NGIAB) +# **NextGen In A Box (NGIAB)** + +**Run the NextGen National Water Resources Modeling Framework locally with ease.** + +NGIAB provides a containerized and user-friendly solution for running the NextGen framework, allowing you to control inputs, configurations, and execution on your local machine. -"NextGen In A Box" (NGIAB) is a containerized version of the NextGen National Water Resources Modeling Framework.

- +

-The NextGen Water Resources Modeling Framework (NextGen) is a data-centric framework developed by the NOAA OWP team to enhance the forecasting of flooding and drought, improve water resource management, and protect lives, property, and the environment. -The Cooperative Institute for Research to Operations in Hydrology (CIROH) along with Lynker has developed “NextGen In A Box” - ready-to-run, containerized and cloud-friendly version of NextGen framework, packaged with scripts to help prepare data and get you modeling more quickly. Leveraging open-source technical tools like Git, GitHub, CI/CD, Docker, NextGen In A Box fosters open research practices, enabling transparent and reproducible research outcomes within the NextGen framework. +| | | +| --- | --- | +| ![alt text](https://ciroh.ua.edu/wp-content/uploads/2022/08/CIROHLogo_200x200.png) | Funding for this project was provided by the National Oceanic & Atmospheric Administration (NOAA), awarded to the Cooperative Institute for Research to Operations in Hydrology (CIROH) through the NOAA Cooperative Agreement with The University of Alabama (NA22NWS4320003). | + + -We are doing a case study : NWM run for Sipsey Fork, Black Warrior river -- We don’t want to run all of CONUS -- We want to run NextGen locally -- We want to have control over inputs / config. -- How can we do it? Answer: NextGen In A Box +**Why NextGen In A Box?** -## Table of Contents +- **Run NextGen Locally:** Experiment with the framework and customize configurations on your local machine. +- **Control Over Inputs:** Choose specific regions or basins for analysis and modify input data as needed. +- **Simplified Setup:** Utilize Docker containers for effortless deployment, avoiding complex software installations. +- **Open Research Practices:** Promote transparency and reproducibility through open-source tools like Git and GitHub. + + + ## Repository Information - This branch specifically for the users of Singularity container image to run simulation on NextGen Framework @@ -31,54 +39,124 @@ We are doing a case study : NWM run for Sipsey Fork, Black Warrior river ```bash . ├── guide.sh - └── README.md + ├── README.md + └── singularity + ├── build.log + ├── singularity_ngen.def + └── templates + ├── extern + │   └── install_extern_libraries.sh + ├── guide + │   └── HelloNGEN.sh + ├── netcdf + │   └── install_netcdf_cxx.sh + ├── ngen + │   └── install_ngen.sh + └── t-route + └── install_t_route.sh ``` 1. [`guilde.sh`](guide.sh) : The guide script to run the simulations on the singularity image 2. [`README.md`](README.md) : Documentation of how to run the model and contribute in development on NGIAB + 3. [`singularity_ngen.def`](singularity/singularity_ngen.def) : The singularity definition file to build image + 4. [`build.log`](singularity/build.log) : This is the last build log that shows all detailed information about the last image build process. + 5. [`install_extern_libraries.sh`](singularity/templates/extern/install_extern_libraries.sh) : This is helper script to install external NGen module during image building process. + 6. [`HelloNGEN.sh`](singularity/templates/guide/HelloNGEN.sh) : This is NGen execution script, which runs when the image is being executed by users. + 7. [`install_netcdf_cxx.sh`](singularity/templates/netcdf/install_netcdf_cxx.sh) : This is helper script to install latest NetCDF C++ version during image building process. + 8. [`install_ngen.sh`](singularity/templates/ngen/install_ngen.sh) : This is helper script to install latest NextGen version during image building process. + 9. [`install_t_route.sh`](singularity/templates/t-route/install_t_route.sh) : This is helper script to install latest T-route version during image building process. ## Prerequisites ### Install SigularityCE and validate SigularityCE is up -- On *Windows*: - - To install SingularityCE on Windows, first you will need to install [WSL](#install-wsl-on-windows) - - [Install SingularityCE Desktop on Windows](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#windows) - - Once SingularityCE is installed, the singularity command will now be available in your WSL2 environment. - - Type `singularity exec library://ubuntu echo "Hello World!"` to make sure singularity is working. - -- On *Mac*: - - To install SingularityCE on Mac, the Sylabs recommend to use `Lima`, a Linux virtual machine with automatic file sharing and port forwarding (similar to WSL2). - - [Install SingularityCE on Mac](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#mac) - - Once Lima is installed, start Lima virtual machine by downloading `AlmaLinux 9` based [template](https://raw.githubusercontent.com/sylabs/singularity/main/examples/lima/singularity-ce.yml). - ```bash +
+ + On Windows + +* To install SingularityCE on Windows, first you will need to install [WSL](#install-wsl-on-windows) +* [Install SingularityCE Desktop on Windows](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#windows) +* Once SingularityCE is installed, the singularity command will now be available in your WSL2 environment. +* Type `singularity exec library://ubuntu echo "Hello World!"` to make sure singularity is working. + +
+ +
+ + On Mac + +- To install SingularityCE on Mac, the Sylabs recommend to use `Lima`, a Linux virtual machine with automatic file sharing and port forwarding (similar to WSL2). +- [Install SingularityCE on Mac](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#mac) +- Once Lima is installed, start Lima virtual machine by downloading `AlmaLinux 9` based [template](https://raw.githubusercontent.com/sylabs/singularity/main/examples/lima/singularity-ce.yml). + ```bash limactl start ./singularity-ce.yml - ``` - - Type `limactl shell singularity-ce` to start Lima VM with SingularityCE. Here you should be able to access Shell within Lima VM. - - Type `singularity --version` to make sure singularity is working. - -- On *Linux*: - > [!NOTE] - > Please make sure you **install all the prerequisites** of installing SingularityCE on Linux. - - [Install docker on Linux](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#installation-on-linux) + ``` +- Type `limactl shell singularity-ce` to start Lima VM with SingularityCE. Here you should be able to access Shell within Lima VM. +- Type `singularity --version` to make sure singularity is working. +
+ +
+ + On Linux + +> [!NOTE] +> Please make sure you **install all the prerequisites** of installing SingularityCE on Linux. +- [Install docker on Linux](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#installation-on-linux) +
### Install WSL on Windows 1. Follow Microsofts latest [instructions](https://learn.microsoft.com/en-us/windows/wsl/install) to install WSL 2. Once this is complete, follow the instructions for linux inside your wsl terminal. - -### Download the input data in "ngen-data" folder from S3 bucket -#### Linux & Mac & WSL +### Input Data +- **Download Sample Data:** Use the provided commands to download sample data for the Sipsey Fork case study. +- **To generate your own data:** Refer to the [NGIAB-datapreprocessor](https://github.com/AlabamaWaterInstitute/NGIAB_data_preprocess) for instructions on generating custom input data. +- **To generate your own data and run using NGIAB:** Refer to the [ngen-datastream repository](https://github.com/CIROH-UA/ngen-datastream/tree/main) for instructions on generating custom input data. + + +This section guides you through downloading and preparing the sample input data for the NextGen In A Box project. + +**Step 1: Create Project Directory** + +- **Linux/Mac:** Open your terminal and go to your desired folder where you want to checkout repo and ngen-data folder and run the following commands: +```bash +mkdir -p NextGen/ngen-data +``` -```bash - mkdir -p NextGen/ngen-data - cd NextGen/ngen-data - wget --no-parent https://ciroh-ua-ngen-data.s3.us-east-2.amazonaws.com/AWI-004/AWI_09_004.tar.gz - tar -xf AWI_09_004.tar.gz - # to rename your folder - mv AWI_09_004 my_data +```bash +cd NextGen/ngen-data +``` +- **WSL (Right click and run as Admin):** Open WSL with administrator privileges and execute: +```bash +cd /mnt/c/Users/ +``` + +```bash +mkdir -p NextGen/ngen-data +``` + +```bash +cd NextGen/ngen-data ``` +**Step 2: Download Sample Data** + +- **Linux/Mac/Windows WSL:** Use wget to download the compressed data file: +```bash +wget --no-parent https://ciroh-ua-ngen-data.s3.us-east-2.amazonaws.com/AWI-006/AWI_16_2853886_006.tar.gz +``` + +**Step 3: Extract and Rename** + +- **All Platforms:** Extract the downloaded file and optionally rename the folder: +```bash +tar -xf AWI_16_2853886_006.tar.gz +``` +### Below is Optional: Rename the folder +```bash +mv AWI_16_2853886_006 my_data +``` +Now you have successfully downloaded and prepared the sample input data in the NextGen/ngen-data directory. Remember to replace "my_data" with your preferred folder name if you choose to rename it. ## Run NextGen In A Box To run NextGen framework, hydrologist only have to execute the [guide script](https://github.com/CIROH-UA/Ngen-Singularity/blob/main/guide.sh) to run simulations on self-contained NextGen framework container image. @@ -92,7 +170,7 @@ To run NextGen framework, hydrologist only have to execute the [guide script](ht 1. Run image in **Interactive shell** mode -### Clone Ngen-Singularity repository + diff --git a/guide.sh b/guide.sh index 2cbcbdb..b3ce28f 100755 --- a/guide.sh +++ b/guide.sh @@ -157,7 +157,7 @@ if uname -a | grep arm64 || uname -a | grep aarch64 ; then IMAGE_NAME=ciroh-ngen-singularity_latest.sif else ARCH=amd64 - IMAGE_URL=library://trupeshkumarpatel/awiciroh/ciroh-ngen-singularity:latest_x86 + IMAGE_URL=library://ciroh-it-support/ngiab/ciroh-ngen-singularity:latest_x86 IMAGE_NAME=ciroh-ngen-singularity_latest.sif fi @@ -173,7 +173,7 @@ select option in "${options[@]}"; do ;; "Run Nextgen using remote singularity image") echo "pulling container and running the model" - singularity pull --arch $ARCH $IMAGE_NAME $IMAGE_URL + singularity pull -F --arch $ARCH $IMAGE_NAME $IMAGE_URL break ;; Exit) @@ -190,7 +190,7 @@ echo -e "Mounting local host directory $HOST_DATA_PATH to /ngen/ngen/data within singularity run --bind $HOST_DATA_PATH:/ngen/ngen/data $IMAGE_NAME /ngen/ngen/data # Final output count -Final_Outputs_Count=$(ls "$HOST_DATA_PATH/outputs" | wc -l) +Final_Outputs_Count=$(find "$HOST_DATA_PATH/outputs/" -type f | wc -l) echo -e "$Final_Outputs_Count new outputs created." echo -e "Any copied files can be found here: $HOST_DATA_PATH/outputs" echo -e "Thank you for running NextGen In A Box: National Water Model! Have a nice day!" diff --git a/singularity/singularity_ngen.def b/singularity/singularity_ngen.def index a41bab3..47ad5fa 100644 --- a/singularity/singularity_ngen.def +++ b/singularity/singularity_ngen.def @@ -23,6 +23,8 @@ From: rockylinux:9.2 dnf install libstdc++ libstdc++-devel glibc glibc-devel libgfortran openmpi openmpi-devel -y dnf --enablerepo=devel install texinfo -y dnf install netcdf netcdf-devel netcdf-fortran netcdf-fortran-devel netcdf-openmpi netcdf-openmpi-devel netcdf-fortran-openmpi netcdf-fortran-openmpi-devel -y + + # Boost installation curl -L -o boost_1_79_0.tar.bz2 https://sourceforge.net/projects/boost/files/boost/1.79.0/boost_1_79_0.tar.bz2/download tar -xjf boost_1_79_0.tar.bz2 -C /tmp/ cp -r /tmp/boost_1_79_0/boost /usr/include/ @@ -34,6 +36,8 @@ From: rockylinux:9.2 /tmp/t-route/install_t_route.sh /tmp/ngen/install_ngen.sh + cp -R /usr/share/modulefiles/mpi /usr/share/lmod/lmod/modulefiles/Core/ + rm -rf /tmp/ngen /tmp/t-route /tmp/netcdf /tmp/extern /tmp/guide %runscript diff --git a/singularity/templates/extern/install_extern_libraries.sh b/singularity/templates/extern/install_extern_libraries.sh index c4ff296..2377a36 100644 --- a/singularity/templates/extern/install_extern_libraries.sh +++ b/singularity/templates/extern/install_extern_libraries.sh @@ -27,7 +27,7 @@ echo "-- Now Installing External Library noah-owp-modular ..." echo "-----------------------------------------------------------" cd /ngen/extern/noah-owp-modular # cmake -B cmake_build -S . -cmake -DnetCDF_MOD_PATH=/usr/include/openmpi-x86_64/ -B cmake_build -S . +cmake -DNETCDF_MODULE_DIR=/usr/lib64/gfortran/modules/openmpi/ -B cmake_build -S . cmake --build cmake_build --target surfacebmi -- -j 2 echo "===========================================================" @@ -87,7 +87,9 @@ cp /tmp/guide/HelloNGEN.sh /ngen/HelloNGEN.sh chown -R root /dmod /ngen /root chmod a+x /dmod/bin/* /ngen/HelloNGEN.sh /root find /ngen/extern/ -name "*.so*" -exec ln -s "{}" /dmod/shared_libs/ \; - +find /ngen/extern/noah-owp-modular -type f -iname "*.TBL" -exec cp '{}' /dmod/datasets/static \; ls -ahl /dmod/shared_libs echo "===========================================================" +ls -ahl /dmod/datasets/static +echo "===========================================================" echo "===========================================================" \ No newline at end of file diff --git a/singularity/templates/guide/HelloNGEN.sh b/singularity/templates/guide/HelloNGEN.sh index 0b06615..1a3dae7 100644 --- a/singularity/templates/guide/HelloNGEN.sh +++ b/singularity/templates/guide/HelloNGEN.sh @@ -9,7 +9,7 @@ CYAN='\e[36m' RESET='\e[0m' # Increasing `ulimit` to Open files -ulimit -n 1000000 +ulimit -n 500000 # Loading Lmod source /etc/profile.d/modules.sh @@ -24,6 +24,8 @@ echo -e "${CYAN}Working directory is:${RESET}" pwd echo -e "\n" + + # Function to automatically select file if only one is found auto_select_file() { local files=($1) @@ -35,8 +37,8 @@ auto_select_file() { } # Finding files -HYDRO_FABRIC_CATCHMENTS=$(find . -name "*datastream*.gpkg") -HYDRO_FABRIC_NEXUS=$(find . -name "*datastream*.gpkg") +HYDRO_FABRIC_CATCHMENTS=$(find . -name "*.gpkg") +HYDRO_FABRIC_NEXUS=$(find . -name "*.gpkg") NGEN_REALIZATIONS=$(find . -name "*realization*.json") # Auto-selecting files if only one is found @@ -78,8 +80,7 @@ if [ "$2" == "auto" ] echo "Run completed successfully, exiting, have a nice day!" exit 0 else - echo "Entering Interactive Mode" - continue + echo "Entering Interactive Mode" fi echo -e "${YELLOW}Select an option (type a number): ${RESET}" @@ -91,12 +92,15 @@ select option in "${options[@]}"; do n1=${selected_catchment:-$(read -p "Enter the hydrofabric catchment file path: " n1; echo "$n1")} n2=${selected_nexus:-$(read -p "Enter the hydrofabric nexus file path: " n2; echo "$n2")} n3=${selected_realization:-$(read -p "Enter the Realization file path: " n3; echo "$n3")} - + echo -e "${GREEN}Selected files:\nCatchment: $n1\nNexus: $n2\nRealization: $n3${RESET}\n" if [ "$option" == "Run NextGen model framework in parallel mode" ]; then procs=$(nproc) - procs=2 # Temporary fixed value + num_catchments=$(find forcings -name *.csv | wc -l) + if [ $num_catchments -lt $procs ]; then + procs=$num_catchments + fi generate_partition "$n1" "$n2" "$procs" run_command="mpirun -n $procs /dmod/bin/ngen-parallel $n1 all $n2 all $n3 $(pwd)/partitions_$procs.json" else @@ -104,7 +108,6 @@ select option in "${options[@]}"; do fi echo -e "${YELLOW}Your NGEN run command is $run_command${RESET}" - sleep 3 break ;; "Run Bash shell") @@ -114,7 +117,7 @@ select option in "${options[@]}"; do "Exit") exit 0 ;; - *) + *) echo -e "${RED}Invalid option $REPLY${RESET}" ;; esac @@ -159,9 +162,9 @@ select option in "${options[@]}"; do echo -e "${GREEN}Have a nice day.${RESET}" break ;; - *) + *) echo -e "${RED}Invalid option $REPLY${RESET}" ;; esac done -exit \ No newline at end of file +exit diff --git a/singularity/templates/ngen/install_ngen.sh b/singularity/templates/ngen/install_ngen.sh index bbbce8e..e58cab4 100755 --- a/singularity/templates/ngen/install_ngen.sh +++ b/singularity/templates/ngen/install_ngen.sh @@ -46,7 +46,9 @@ echo "-- Now Building NGen at /ngen/serialbuild ..." echo "-----------------------------------------------------------" cd /ngen # cmake -B /ngen/mpibuild -S . -DNGEN_WITH_MPI=ON -DNGEN_WITH_PYTHON=ON -DNGEN_WITH_ROUTING=ON -DNGEN_WITH_NETCDF=ON -DNGEN_QUIET=ON -cmake -B $current_path/ngen/serialbuild -S . -DNGEN_WITH_MPI:BOOL=OFF -DNGEN_WITH_NETCDF:BOOL=ON -DNGEN_WITH_SQLITE:BOOL=ON -DNGEN_WITH_UDUNITS:BOOL=ON -DNGEN_WITH_BMI_FORTRAN:BOOL=ON -DNGEN_WITH_BMI_C:BOOL=ON -DNGEN_WITH_PYTHON:BOOL=ON -DNGEN_WITH_ROUTING:BOOL=ON -DNGEN_WITH_TESTS:BOOL=ON -DNGEN_QUIET:BOOL=ON -DNETCDF_CXX_INCLUDE_DIR=/usr/local/include -DNETCDF_CXX_LIBRARY=/usr/local/lib64/libnetcdf-cxx4.so # -DNETCDF_INCLUDE_DIR=/usr/include -DNETCDF_LIBRARY=/usr/lib64/libnetcdf.so +cmake -B $current_path/ngen/serialbuild -S . -DCMAKE_INSTALL_PREFIX=/ngen/ngen -DNGEN_WITH_EXTERN_ALL=ON -DNGEN_WITH_MPI:BOOL=OFF -DNGEN_WITH_NETCDF:BOOL=ON -DNGEN_WITH_SQLITE:BOOL=ON -DNGEN_WITH_UDUNITS:BOOL=ON -DNGEN_QUIET:BOOL=ON -DNGEN_WITH_BMI_FORTRAN:BOOL=ON -DNGEN_WITH_BMI_C:BOOL=ON -DNGEN_WITH_PYTHON:BOOL=ON -DNGEN_WITH_ROUTING:BOOL=ON -DNGEN_WITH_TESTS:BOOL=ON -DNETCDF_CXX_INCLUDE_DIR=/usr/local/include -DNETCDF_CXX_LIBRARY=/usr/local/lib64/libnetcdf-cxx4.so +# -DNETCDF_INCLUDE_DIR=/usr/include -DNETCDF_LIBRARY=/usr/lib64/libnetcdf.so + echo "===========================================================" echo "===========================================================" @@ -65,7 +67,7 @@ echo "-- Now Building NGen at /ngen/parallelbuild ..." echo "-----------------------------------------------------------" cd /ngen # cmake -B /ngen/mpibuild -S . -DNGEN_WITH_MPI=ON -DNGEN_WITH_PYTHON=ON -DNGEN_WITH_ROUTING=ON -DNGEN_WITH_NETCDF=ON -DNGEN_QUIET=ON -cmake -B $current_path/ngen/parallelbuild -S . -DNGEN_WITH_MPI:BOOL=ON -DNGEN_WITH_NETCDF:BOOL=ON -DNGEN_WITH_SQLITE:BOOL=ON -DNGEN_WITH_UDUNITS:BOOL=ON -DNGEN_WITH_BMI_FORTRAN:BOOL=ON -DNGEN_WITH_BMI_C:BOOL=ON -DNGEN_WITH_PYTHON:BOOL=ON -DNGEN_WITH_ROUTING:BOOL=ON -DNGEN_WITH_TESTS:BOOL=ON -DNGEN_QUIET:BOOL=ON -DNETCDF_CXX_INCLUDE_DIR=/usr/local/include -DNETCDF_CXX_LIBRARY=/usr/local/lib64/libnetcdf-cxx4.so # -DNETCDF_INCLUDE_DIR=/usr/include -DNETCDF_LIBRARY=/usr/lib64/libnetcdf.so +cmake -B $current_path/ngen/parallelbuild -S . -DCMAKE_INSTALL_PREFIX=/ngen/ngen -DNGEN_WITH_EXTERN_ALL=ON -DNGEN_WITH_MPI:BOOL=ON -DNGEN_WITH_NETCDF:BOOL=ON -DNGEN_WITH_SQLITE:BOOL=ON -DNGEN_WITH_UDUNITS:BOOL=ON -DNGEN_QUIET:BOOL=ON -DNGEN_WITH_BMI_FORTRAN:BOOL=ON -DNGEN_WITH_BMI_C:BOOL=ON -DNGEN_WITH_PYTHON:BOOL=ON -DNGEN_WITH_ROUTING:BOOL=ON -DNGEN_WITH_TESTS:BOOL=ON -DNETCDF_CXX_INCLUDE_DIR=/usr/local/include -DNETCDF_CXX_LIBRARY=/usr/local/lib64/libnetcdf-cxx4.so # -DNETCDF_INCLUDE_DIR=/usr/include -DNETCDF_LIBRARY=/usr/lib64/libnetcdf.so echo "===========================================================" echo "===========================================================" diff --git a/singularity/templates/t-route/install_t_route.sh b/singularity/templates/t-route/install_t_route.sh index be070ac..5904f79 100755 --- a/singularity/templates/t-route/install_t_route.sh +++ b/singularity/templates/t-route/install_t_route.sh @@ -19,6 +19,8 @@ echo "-----------------------------------------------------------" export FC=gfortran NETCDF=/usr/include git clone --progress --single-branch --branch master https://github.com/NOAA-OWP/t-route cd t-route +git submodule update --init + ./compiler.sh no-e cd /t-route/src/troute-network @@ -27,10 +29,15 @@ python setup.py --use-cython bdist_wheel cd /t-route/src/troute-routing python setup.py --use-cython bdist_wheel +# troute-config doesn't use setup.py, use build to make the wheel +cd /t-route/src/troute-config +python -m build . \ + cd /t-route/src/troute-nwm python setup.py bdist_wheel cd /t-route cp /t-route/src/troute-network/dist/*.whl /t-route/wheels cp /t-route/src/troute-routing/dist/*.whl /t-route/wheels +cp /t-route/src/troute-config/dist/*.whl /t-route/wheels cp /t-route/src/troute-nwm/dist/*.whl /t-route/wheels