Fixed ulimit, and MPI. Added recent features from CloudInfra

README.md

@@ -1,84 +1,162 @@
-# NextGen In A Box (NGIAB)
+# **NextGen In A Box (NGIAB)**
+
+**Run the NextGen National Water Resources Modeling Framework locally with ease.**
+
+NGIAB provides a containerized and user-friendly solution for running the NextGen framework, allowing you to control inputs, configurations, and execution on your local machine.
 
-"NextGen In A Box" (NGIAB) is a containerized version of the NextGen National Water Resources Modeling Framework.
 <p align="center">
-    <img src="https://github.com/CIROH-UA/NGIAB-CloudInfra/assets/54657/1a647024-67f8-489a-9f5e-86437449b6ff" width="300">
+<img src="https://github.com/CIROH-UA/NGIAB-CloudInfra/blob/main/image/README/ngiab.png" width="300">
 </p>
-The NextGen Water Resources Modeling Framework (NextGen) is a data-centric framework developed by the NOAA OWP team to enhance the forecasting of flooding and drought, improve water resource management, and protect lives, property, and the environment. 
 
-The Cooperative Institute for Research to Operations in Hydrology (CIROH) along with Lynker has developed “NextGen In A Box” - ready-to-run, containerized and cloud-friendly version of NextGen framework, packaged with scripts to help prepare data and get you modeling more quickly. Leveraging open-source technical tools like Git, GitHub, CI/CD, Docker, NextGen In A Box fosters open research practices, enabling transparent and reproducible research outcomes within the NextGen framework.
+| | |
+| --- | --- |
+| ![alt text](https://ciroh.ua.edu/wp-content/uploads/2022/08/CIROHLogo_200x200.png) | Funding for this project was provided by the National Oceanic & Atmospheric Administration (NOAA), awarded to the Cooperative Institute for Research to Operations in Hydrology (CIROH) through the NOAA Cooperative Agreement with The University of Alabama (NA22NWS4320003). |
+
+<!-- [![ARM Build and push final image](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_branch.yml/badge.svg)](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_branch.yml)
+[![X86 Build and push final image](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_x86.yml/badge.svg)](https://github.com/CIROH-UA/NGIAB-CloudInfra/actions/workflows/docker_image_main_x86.yml) -->
 
-We are doing a case study : NWM run for Sipsey Fork, Black Warrior river
-- We don’t want to run all of CONUS
-- We want to run NextGen locally
-- We want to have control over inputs / config.
-- How can we do it? Answer: NextGen In A Box
+**Why NextGen In A Box?**
 
-## Table of Contents
+- **Run NextGen Locally:** Experiment with the framework and customize configurations on your local machine.
+- **Control Over Inputs:** Choose specific regions or basins for analysis and modify input data as needed.
+- **Simplified Setup:** Utilize Docker containers for effortless deployment, avoiding complex software installations.
+- **Open Research Practices:** Promote transparency and reproducibility through open-source tools like Git and GitHub.
+
+
+<!-- ## Table of Contents
 * [Repository Information](#repository-information)
 * [Prerequisites](#prerequisites)
     + [Install Singularity](#install-sigularityce-and-validate-sigularityce-is-up)
     + [Install WSL on Windows](#install-wsl-on-windows)
-    + [Download the input data in "ngen-data" folder from S3 bucket ](#download-the-input-data-in-ngen-data-folder-from-s3-bucket)
 * [Run NextGen-In-A-Box](#run-nextgen-in-a-box)
     + [Clone Ngen-Singularity repository](#clone-ngen-singularity-repository)
     + [How to run the model script?](#how-to-run-the-model-script)
-    + [Output of the model script](#output-of-the-model-guide-script)
+    + [Output of the model script](#output-of-the-model-guide-script) -->
 
 ## Repository Information
 - This branch specifically for the users of Singularity container image to run simulation on NextGen Framework
 - The file structure and brife information of each file:
     ```bash 
     .
     ├── guide.sh
-    └── README.md
+    ├── README.md
+    └── singularity
+        ├── build.log
+        ├── singularity_ngen.def
+        └── templates
+            ├── extern
+            │   └── install_extern_libraries.sh
+            ├── guide
+            │   └── HelloNGEN.sh
+            ├── netcdf
+            │   └── install_netcdf_cxx.sh
+            ├── ngen
+            │   └── install_ngen.sh
+            └── t-route
+                └── install_t_route.sh
     ```
     1. [`guilde.sh`](guide.sh) : The guide script to run the simulations on the singularity image
     2. [`README.md`](README.md) : Documentation of how to run the model and contribute in development on NGIAB
+    3. [`singularity_ngen.def`](singularity/singularity_ngen.def) : The singularity definition file to build image
+    4. [`build.log`](singularity/build.log) : This is the last build log that shows all detailed information about the last image build process.
+    5. [`install_extern_libraries.sh`](singularity/templates/extern/install_extern_libraries.sh) : This is helper script to install external NGen module during image building process.
+    6. [`HelloNGEN.sh`](singularity/templates/guide/HelloNGEN.sh) : This is NGen execution script, which runs when the image is being executed by users.
+    7. [`install_netcdf_cxx.sh`](singularity/templates/netcdf/install_netcdf_cxx.sh) : This is helper script to install latest NetCDF C++ version during image building process.
+    8. [`install_ngen.sh`](singularity/templates/ngen/install_ngen.sh) : This is helper script to install latest NextGen version during image building process.
+    9. [`install_t_route.sh`](singularity/templates/t-route/install_t_route.sh) : This is helper script to install latest T-route version during image building process. 
 
 ## Prerequisites
 
 ### Install SigularityCE and validate SigularityCE is up
-- On *Windows*:
-    - To install SingularityCE on Windows, first you will need to install [WSL](#install-wsl-on-windows) 
-    - [Install SingularityCE Desktop on Windows](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#windows)
-    - Once SingularityCE is installed, the singularity command will now be available in your WSL2 environment.
-    - Type `singularity exec library://ubuntu echo "Hello World!"` to make sure singularity is working.
-
-- On *Mac*:
-    - To install SingularityCE on Mac, the Sylabs recommend to use `Lima`, a Linux virtual machine with automatic file sharing and port forwarding (similar to WSL2).
-    - [Install SingularityCE on Mac](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#mac) 
-    - Once Lima is installed, start Lima virtual machine by downloading `AlmaLinux 9` based [template](https://raw.githubusercontent.com/sylabs/singularity/main/examples/lima/singularity-ce.yml).
-        ```bash
+<details>
+
+<summary> On Windows </summary>
+
+* To install SingularityCE on Windows, first you will need to install [WSL](#install-wsl-on-windows) 
+* [Install SingularityCE Desktop on Windows](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#windows)
+* Once SingularityCE is installed, the singularity command will now be available in your WSL2 environment.
+* Type `singularity exec library://ubuntu echo "Hello World!"` to make sure singularity is working.
+
+</details>
+
+<details>
+
+<summary> On Mac </summary>
+
+- To install SingularityCE on Mac, the Sylabs recommend to use `Lima`, a Linux virtual machine with automatic file sharing and port forwarding (similar to WSL2).
+- [Install SingularityCE on Mac](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#mac) 
+- Once Lima is installed, start Lima virtual machine by downloading `AlmaLinux 9` based [template](https://raw.githubusercontent.com/sylabs/singularity/main/examples/lima/singularity-ce.yml).
+    ```bash
         limactl start ./singularity-ce.yml
-        ```
-    - Type `limactl shell singularity-ce` to start Lima VM with SingularityCE. Here you should be able to access Shell within Lima VM.
-    - Type `singularity --version` to make sure singularity is working.
-
-- On *Linux*:
-    > [!NOTE]
-    > Please make sure you **install all the prerequisites** of installing SingularityCE on Linux.
-    - [Install docker on Linux](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#installation-on-linux)
+    ```
+- Type `limactl shell singularity-ce` to start Lima VM with SingularityCE. Here you should be able to access Shell within Lima VM.
+- Type `singularity --version` to make sure singularity is working.
+</details>
+
+<details>
+
+<summary> On Linux </summary>
+
+> [!NOTE]
+> Please make sure you **install all the prerequisites** of installing SingularityCE on Linux.
+- [Install docker on Linux](https://docs.sylabs.io/guides/4.0/admin-guide/installation.html#installation-on-linux)
 
+</details>
 
 ### Install WSL on Windows
 
 1. Follow Microsofts latest [instructions](https://learn.microsoft.com/en-us/windows/wsl/install) to install WSL
 2. Once this is complete, follow the instructions for linux inside your wsl terminal.
 
-
-### Download the input data in "ngen-data" folder from S3 bucket
 
-#### Linux & Mac & WSL
+### Input Data
+- **Download Sample Data:** Use the provided commands to download sample data for the Sipsey Fork case study.
+- **To generate your own data:** Refer to the [NGIAB-datapreprocessor](https://github.com/AlabamaWaterInstitute/NGIAB_data_preprocess) for instructions on generating custom input data.
+- **To generate your own data and run using NGIAB:** Refer to the [ngen-datastream repository](https://github.com/CIROH-UA/ngen-datastream/tree/main) for instructions on generating custom input data.
+
+
+This section guides you through downloading and preparing the sample input data for the NextGen In A Box project.
+
+**Step 1: Create Project Directory**
+
+- **Linux/Mac:** Open your terminal and go to your desired folder where you want to checkout repo and ngen-data folder and run the following commands:
+```bash
+mkdir -p NextGen/ngen-data
+```
 
-```bash   
-    mkdir -p NextGen/ngen-data
-    cd NextGen/ngen-data
-    wget --no-parent https://ciroh-ua-ngen-data.s3.us-east-2.amazonaws.com/AWI-004/AWI_09_004.tar.gz
-    tar -xf AWI_09_004.tar.gz
-    # to rename your folder
-    mv AWI_09_004 my_data
+```bash
+cd NextGen/ngen-data
+```
+- **WSL (Right click and run as Admin):** Open WSL with administrator privileges and execute:
+```bash
+cd /mnt/c/Users/<Folder>
+```
+
+```bash
+mkdir -p NextGen/ngen-data
+```
+
+```bash
+cd NextGen/ngen-data
 ```
+**Step 2: Download Sample Data**
+
+- **Linux/Mac/Windows WSL:** Use wget to download the compressed data file:
+```bash
+wget --no-parent https://ciroh-ua-ngen-data.s3.us-east-2.amazonaws.com/AWI-006/AWI_16_2853886_006.tar.gz
+```
+
+**Step 3: Extract and Rename**
+
+- **All Platforms:** Extract the downloaded file and optionally rename the folder:
+```bash
+tar -xf AWI_16_2853886_006.tar.gz
+```
+### Below is Optional: Rename the folder
+```bash
+mv AWI_16_2853886_006 my_data
+```
+Now you have successfully downloaded and prepared the sample input data in the NextGen/ngen-data directory. Remember to replace "my_data" with your preferred folder name if you choose to rename it.
 
 ## Run NextGen In A Box
 To run NextGen framework, hydrologist only have to execute the [guide script](https://github.com/CIROH-UA/Ngen-Singularity/blob/main/guide.sh) to run simulations on self-contained NextGen framework container image.
@@ -92,7 +170,7 @@ To run NextGen framework, hydrologist only have to execute the [guide script](ht
         1. Run image in **Interactive shell** mode
 
 
-### Clone Ngen-Singularity repository
+<!-- ### Clone Ngen-Singularity repository
 
 Navigate to NextGen directory and clone the repository using below commands:
 
@@ -158,4 +236,4 @@ Example NGEN run command for serial mode:
 ### Output of the model guide script
 
 The output files are copied to the `outputs` folder in the 'NextGen/ngen-data/AWI_03W_113060_002/' directory you created in the first step
-
+ -->

guide.sh

@@ -157,7 +157,7 @@ if uname -a | grep arm64 || uname -a | grep aarch64 ; then
     IMAGE_NAME=ciroh-ngen-singularity_latest.sif
 else
     ARCH=amd64
-    IMAGE_URL=library://trupeshkumarpatel/awiciroh/ciroh-ngen-singularity:latest_x86
+    IMAGE_URL=library://ciroh-it-support/ngiab/ciroh-ngen-singularity:latest_x86
     IMAGE_NAME=ciroh-ngen-singularity_latest.sif
 fi
 
@@ -173,7 +173,7 @@ select option in "${options[@]}"; do
             ;;
         "Run Nextgen using remote singularity image")
             echo "pulling container and running the model"
-            singularity pull --arch $ARCH $IMAGE_NAME $IMAGE_URL
+            singularity pull -F --arch $ARCH $IMAGE_NAME $IMAGE_URL
             break
             ;;
         Exit)
@@ -190,7 +190,7 @@ echo -e "Mounting local host directory $HOST_DATA_PATH to /ngen/ngen/data within
 singularity run --bind $HOST_DATA_PATH:/ngen/ngen/data $IMAGE_NAME /ngen/ngen/data
 
 # Final output count
-Final_Outputs_Count=$(ls "$HOST_DATA_PATH/outputs" | wc -l)
+Final_Outputs_Count=$(find "$HOST_DATA_PATH/outputs/" -type f | wc -l)
 echo -e "$Final_Outputs_Count new outputs created."
 echo -e "Any copied files can be found here: $HOST_DATA_PATH/outputs"
 echo -e "Thank you for running NextGen In A Box: National Water Model! Have a nice day!"

singularity/singularity_ngen.def

@@ -23,6 +23,8 @@ From: rockylinux:9.2
      dnf install libstdc++ libstdc++-devel glibc glibc-devel libgfortran openmpi openmpi-devel -y
      dnf --enablerepo=devel install texinfo -y
      dnf install netcdf netcdf-devel netcdf-fortran netcdf-fortran-devel netcdf-openmpi netcdf-openmpi-devel netcdf-fortran-openmpi netcdf-fortran-openmpi-devel -y 
+
+     # Boost installation
      curl -L -o boost_1_79_0.tar.bz2 https://sourceforge.net/projects/boost/files/boost/1.79.0/boost_1_79_0.tar.bz2/download
      tar -xjf boost_1_79_0.tar.bz2 -C /tmp/
      cp -r /tmp/boost_1_79_0/boost /usr/include/
@@ -34,6 +36,8 @@ From: rockylinux:9.2
      /tmp/t-route/install_t_route.sh
      /tmp/ngen/install_ngen.sh
 
+     cp -R /usr/share/modulefiles/mpi /usr/share/lmod/lmod/modulefiles/Core/
+
      rm -rf /tmp/ngen /tmp/t-route /tmp/netcdf /tmp/extern /tmp/guide
 
 %runscript

singularity/templates/extern/install_extern_libraries.sh

@@ -27,7 +27,7 @@ echo "-- Now Installing External Library noah-owp-modular ..."
 echo "-----------------------------------------------------------"
 cd /ngen/extern/noah-owp-modular
 # cmake -B cmake_build -S .
-cmake -DnetCDF_MOD_PATH=/usr/include/openmpi-x86_64/ -B cmake_build -S .
+cmake -DNETCDF_MODULE_DIR=/usr/lib64/gfortran/modules/openmpi/ -B cmake_build -S .
 cmake --build cmake_build --target surfacebmi -- -j 2
 
 echo "==========================================================="
@@ -87,7 +87,9 @@ cp /tmp/guide/HelloNGEN.sh /ngen/HelloNGEN.sh
 chown -R root /dmod /ngen /root
 chmod a+x /dmod/bin/* /ngen/HelloNGEN.sh /root
 find /ngen/extern/ -name "*.so*" -exec ln -s "{}" /dmod/shared_libs/ \;
-
+find /ngen/extern/noah-owp-modular -type f -iname "*.TBL" -exec cp '{}' /dmod/datasets/static  \; 
 ls -ahl /dmod/shared_libs
 echo "==========================================================="
+ls -ahl /dmod/datasets/static
+echo "==========================================================="
 echo "==========================================================="

singularity/templates/guide/HelloNGEN.sh

@@ -9,7 +9,7 @@ CYAN='\e[36m'
 RESET='\e[0m'
 
 # Increasing `ulimit` to Open files
-ulimit -n 1000000
+ulimit -n 500000
 
 # Loading Lmod
 source /etc/profile.d/modules.sh
@@ -24,6 +24,8 @@ echo -e "${CYAN}Working directory is:${RESET}"
 pwd
 echo -e "\n"
 
+
+
 # Function to automatically select file if only one is found
 auto_select_file() {
   local files=($1)
@@ -35,8 +37,8 @@ auto_select_file() {
 }
 
 # Finding files
-HYDRO_FABRIC_CATCHMENTS=$(find . -name "*datastream*.gpkg")
-HYDRO_FABRIC_NEXUS=$(find . -name "*datastream*.gpkg")
+HYDRO_FABRIC_CATCHMENTS=$(find . -name "*.gpkg")
+HYDRO_FABRIC_NEXUS=$(find . -name "*.gpkg")
 NGEN_REALIZATIONS=$(find . -name "*realization*.json")
 
 # Auto-selecting files if only one is found
@@ -78,8 +80,7 @@ if [ "$2" == "auto" ]
     echo "Run completed successfully, exiting, have a nice day!"
     exit 0
   else
-    echo "Entering Interactive Mode"
-    continue
+    echo "Entering Interactive Mode"    
 fi
 
 echo -e "${YELLOW}Select an option (type a number): ${RESET}"
@@ -91,20 +92,22 @@ select option in "${options[@]}"; do
       n1=${selected_catchment:-$(read -p "Enter the hydrofabric catchment file path: " n1; echo "$n1")}
       n2=${selected_nexus:-$(read -p "Enter the hydrofabric nexus file path: " n2; echo "$n2")}
       n3=${selected_realization:-$(read -p "Enter the Realization file path: " n3; echo "$n3")}
-      
+
       echo -e "${GREEN}Selected files:\nCatchment: $n1\nNexus: $n2\nRealization: $n3${RESET}\n"
 
       if [ "$option" == "Run NextGen model framework in parallel mode" ]; then
         procs=$(nproc)
-        procs=2 # Temporary fixed value
+        num_catchments=$(find forcings -name *.csv | wc -l)
+        if [ $num_catchments -lt $procs ]; then
+                procs=$num_catchments
+        fi
         generate_partition "$n1" "$n2" "$procs"
         run_command="mpirun -n $procs /dmod/bin/ngen-parallel $n1 all $n2 all $n3 $(pwd)/partitions_$procs.json"
       else
         run_command="/dmod/bin/ngen-serial $n1 all $n2 all $n3"
       fi
 
       echo -e "${YELLOW}Your NGEN run command is $run_command${RESET}"
-      sleep 3
       break
       ;;
     "Run Bash shell")
@@ -114,7 +117,7 @@ select option in "${options[@]}"; do
     "Exit")
       exit 0
       ;;
-    *) 
+    *)
       echo -e "${RED}Invalid option $REPLY${RESET}"
       ;;
   esac
@@ -159,9 +162,9 @@ select option in "${options[@]}"; do
       echo -e "${GREEN}Have a nice day.${RESET}"
       break
       ;;
-    *) 
+    *)
       echo -e "${RED}Invalid option $REPLY${RESET}"
       ;;
   esac
 done
-exit
+exit