Skip to content

Commit

Permalink
add architecture
Browse files Browse the repository at this point in the history
  • Loading branch information
michaelmattig committed Dec 6, 2024
1 parent 76f1a18 commit 0a08bb6
Show file tree
Hide file tree
Showing 3 changed files with 114 additions and 1 deletion.
63 changes: 62 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,62 @@
# fair-ds-oc1-docs
# FAIR-DS Demonstrator: Copernicus Data SpaceMachine Learning - ECOMETRICS

The ECOMETRICS app is a demonstrator for the FAIR-DS project.
This document presents the software architecture.
You can find more information about the app itself in the [FAIR-DS wiki](https://fair-ds4nfdi.github.io/wiki/).

## Architecture

There are four components to the ECOMETRICS app demonstrator: the Data Spaces, the Geo Engine instance, the machine learning in Jupyter and the ECOMETRICS dashboard.
There are two kinds of users: a Data Scientist that trains and refines a model and a user that select area and time of interest and performs an analysis.
The following diagram shows the architecture of the ECOMETRICS app. ![ECOMETRICS architecture](./assets/architecture.svg)

```mermaid
flowchart LR
subgraph DataSpaces
NFDI[NFDI 4 Biodiversity] --> Aruna
Copernicus[Copernicus Data Space Ecosystem] --> Aruna
end
subgraph GeoEngine
Aruna[Data Connectors - Aruna & STAC] --> ProcessingEngine[Processing Engine]
ProcessingEngine --> API[API]
end
API --> Dashboards[ECOMETRICS Dashboards]
API --> PythonLibrary[Python Library]
User[User] -->|Select area/time and analyze| Dashboards
DataScientist[Data Scientist] -->|Train and refine model| PythonLibrary
PythonLibrary --> Jupyter[Jupyter Notebook]
PythonLibrary --> ScikitLearn[Scikit-Learn]
PythonLibrary --> ONNX[ONNX]
Dashboards -->|Show insights| User
```

### Data Spaces

The demonstrator connects to the Copernicus Data Space and the NFDI4Biodiversity Data Space.
The NFDI4Biodiversity provides training labels for the machine learning model.
The Copernicus Data Space provides the satellite data.

### Geo Engine

Geo Engine has a data connector for each of the Data Spaces.
The connectors are able to browser metadata and access the raw data for analysis.
They map the files to the Geo Engine data model that supports temporal and spatial queries.
The processing engine harmonizes and enriches the data and makes it ready for machine learning.
Standardized and custom API methods make the data available.

### Machine Learning in Jupyter

The machine learning notebook access the Geo Engine API to retrieve the data.
It uses Sci-kit Learn to train and refine a model.
The model is stored in the ONNX format and uploaded to the Geo Engine where it is registered and can be used as an operator.

### ECOMETRICS Dashboard

The ECOMETRICS Dashboard is an easy-to-use web app that build upon the Geo Engine UI toolkit.
The dashboard has an interactive map and lets the user select an area and time of interest.
It has an analysis functionality that triggers the Geo Engine to run the machine learning model.
51 changes: 51 additions & 0 deletions assets/architecture.mmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
flowchart TB
%% Data Spaces
subgraph DataSpaces["Data Spaces"]
nfdi["NFDI 4 BIODIVERSITY"]
copernicus["Copernicus Data Space Ecosystem"]
end

%% Geo Engine Components
subgraph GeoEngine["Geo Engine"]
subgraph DataConnectors["Data Connectors"]
aruna["ARUNA"]
stac["STAC"]
end
processingEngine["Processing Engine"]
api["API"]

subgraph PythonLibrary["Python Library"]
pythonLib["Python Library"]
scikit["Scikit-learn"]
onnx["ONNX"]
jupyter["Jupyter"]
end
end

%% ECOMETRICS Dashboards
subgraph Dashboards["ECOMETRICS Dashboards"]
dashboard["Dashboard"]
end

%% Actors
user["User"]
dataScientist["Data Scientist"]

%% Connections within Geo Engine
nfdi --> aruna
copernicus --> stac
aruna --> processingEngine
stac --> processingEngine
processingEngine --> api
api --> dashboard
api --> pythonLib
pythonLib --> scikit
pythonLib --> onnx
pythonLib --> jupyter

%% User Interactions
user --> dashboard:::interaction["Select area/time and analyze"]
dataScientist --> pythonLib:::interaction["Train and refine model"]

%% Styling for better readability (optional)
classDef interaction fill:#f9f,stroke:#333,stroke-width:2px;
1 change: 1 addition & 0 deletions assets/architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 0a08bb6

Please sign in to comment.