Generate GX conformant credentials for VM Images

.github/workflows/python-app.yml

@@ -0,0 +1,41 @@
+# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: Python application
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.10"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        # pip install flake8 pytest
+        pip install pytest
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    #- name: Lint with flake8
+    #  run: |
+    #    # stop the build if there are Python syntax errors or undefined names
+    #    flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+    #    # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+    #    flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        ls
+        python -m pytest tests/

README.md

@@ -1,131 +1,177 @@
 # gx-credential-generator
-Tools for creating [Gaia-X Crecential](https://gitlab.com/gaia-x/technical-committee/architecture-document/-/blob/master/architecture_document/gx_conceptual_model.md#gaia-x-credentials), formally knwon as Self-Descriptions for SCS compliant cloud infrastructures (OpenStack, k8s, ...)
+Tool for creating [Gaia-X Crecentials](https://gitlab.com/gaia-x/technical-committee/architecture-document/-/blob/master/architecture_document/gx_conceptual_model.md#gaia-x-credentials), previously known as Gaia-X Self-Descriptions, for SCS compliant cloud infrastructures (OpenStack, k8s, ...)
 
-## OpenStack
+## Introduction
+
+### OpenStack
 We want to collect discoverable information from an OpenStack cloud,
 assuming that we have access to it (as normal tenant user).
 
-We start with the region list and then read the OpenStack catalog to collect
-- OS_AUTH_URL (Keystone Endpoint)
-- List of services (along with supported versions, min thr. max)
-- Per service: extensions (cinderv3, nova)
-- Flavors for compute incl. flavor details (SCS spec)
-- AZs (for nova, cinderv3, neutron)
-- UI (URL, type: horizon or custom)
-
-This then should be output as JSON-LD (or YAML-LD) for the Gaia-X catalogue.
-
-References:
-- <https://gaia-x.gitlab.io/gaia-x-community/gaia-x-self-descriptions/service/service.html>
-- <https://gitlab.com/gaia-x/gaia-x-community/gx-hackathon/gx-hackathon-3/-/blob/main/gxfs-track/self-descriptions/service_taxonomy.md>
-- <https://gitlab.com/gaia-x/gaia-x-community/gx-hackathon/gx-hackathon-3/-/blob/main/gxfs-track/self-descriptions/sd_attributes.md>
-- <https://github.com/SovereignCloudStack/Docs/blob/main/Design-Docs/flavor-naming.md>
-- <https://github.com/garloff/openstack-api-discovery>
-
-Notes from reviewing the SD attributes:
-* Virtualized CPU types: It might be of limited use to reference exact model names, rather characterize properties
-  (generation, speed, insn set extensions, ...)
-* NICs: Virtual NICs are almost unlimited, but there may be a limited amount of hardware-accelerated
-  NICs (using SR-IOV and multiqueue features) available -- these may need to be added to SCS flavor
-  naming.
-* Other extension hardware like FPGAs need to be specified
-* Tenant isolation needs a list of criteria. Virt OK? V(x)LANs OK? Shared storage cluster OK? ...
-* Availability Zones: Provider needs to create transparency over what it means. Fire protection zones?
-  Power supply zones? Internet connectivity zones? Minimal and maximal physical distance? Network
-  latency distance?
-
-
-## k8s
-Same thing for k8s
+We read the OpenStack catalog to collect
+- public VM Images
+
+This then should be output as GX compliant credential in JSON-LD for the Gaia-X catalogue.
+
+### k8s
+We want to collect discoverable information from a k8s cloud,
+assuming that we have access to it.
 
 Collect information on a k8s cluster:
 - Metadata
 - API Version
 - Nodes information
 - Pods information
 
-## K8s as-a-Service (KaaS) offering considerations
+#### K8s as-a-Service (KaaS) offering considerations
 
 For typical k8s aaS offerings, every cluster is different,
 and we probably don't want to have a description for every single
 customer specific cluster. (Some providers may offer self-service,
-so we would not want to push of a new SD into the G-X catalogue on
+so we would not want to push of a new GX Credential into the GX catalogue on
 creation, changing or deletion of clusters.) Still it might be
-helpful to have a SD on demand for an existing cluster to characterize
-it, so users can use the SD to match it to app requirements.
+helpful to have a GX Credential on demand for an existing cluster to characterize
+it, so users can use the GX Credentials to match it to app requirements.
 
-So the SD for a k8s aaS solution would list possible options and
+So the GX Credential for a k8s aaS solution would list possible options and
 ranges: What k8s versions are supported, what max number of workers,
 flavors, etc.? What services are optionally delivered (and supported)
 by the provider?
 
 For KaaS, the option space really needs to be described.
 As of now, this can not be discovered, short of using external sources,
-like the IaaS SD, the list of node images (osism minio), ...
+like the IaaS GX Credential, the list of node images (osism minio), ...
 
 
 ## Quick Start Guide
 
 1. Clone the repository into a location of your choice
 ```bash
-git clone git@github.com:SovereignCloudStack/gx-self-description-generator.git
-cd gx-self-description-generator
+git clone git@github.com:SovereignCloudStack/gx-credential-generator.git
+cd gx-credential-generator
 ```
 
 2. Install scripts dependencies (installing them into a Python [virtualenv](https://virtualenv.pypa.io/en/stable/) is recommended)
 ```bash
 pip install -r requirements.txt
 ```
 
-3. Create `clouds.yaml` configuration file
-  - Gaia-x generator has to be configured with user credentials, auth-url, ... to access your Openstack cloud. This is done using [clouds.yaml](https://docs.openstack.org/python-openstackclient/ussuri/configuration/index.html) 
-  - Make sure the following keys exist in our `clouds.yaml`
+
+### OpenStack 
+
+a. Create `clouds.yaml` configuration file
+  - GX credential Generator requires access to OpenStack cluster as normal tenant user and has to be configured with these user credentials to access your Openstack cloud. This is done using [clouds.yaml](https://docs.openstack.org/python-openstackclient/ussuri/configuration/index.html). clouds.yaml is a yaml file containing several cloud configurations. Each configuration is referred by name.
+  - Make sure the following keys exist in our `clouds.yaml`.  
      - `auth.user_domain_name`
      - `auth.project_domain_name`
      - `region_name`
 
-4. Generate Gaia-X Credentials
+
+b. Generate Gaia-X Credentials
 
-   - OpenStack to json file (timestamp and extension is added to file name and script assumes OpenStack access (as normal tenant user)
-   ```bash
-   ./gx-sd-generator.py --gaia-x --os-cloud=<os-cloud> --file=<file-name>
+   - To print OpenStack properties in JSON-LD
+```bash
+   python3 cli.py openstack <os-cloud>
    ```
-   - To use generated Gaia-X Credential in [Gaia-X Wizard](https://wizard.lab.gaia-x.eu/) add `--wizard` option
-     - '@' has to be removed from @id and @type in generated SD, to be able to sign and verify it in Gaia-X Wizard
-   - K8s (script assumes K8s access)
+[//]: <> (- To store OpenStack properties as several credentials in wallet
+   ```bash
+   python3 cli.py openstack <os-cloud> --wallet
+   ```)
+
+[//]: <> (- To omit to print OpenStack properties on screen and to store OpenStack properties as credential in wallet
+   ```bash
+   python3 cli.py openstack <os-cloud> --wallet --no-print
+   ```)
+
+### K88
+Generate Gaia-X Credentials
+   - To print K8s properties ... 
    ```bash
    ./gx-sd-generator.py k8s
    ```
-
-4. Start the gaiax-pipeline
+### Airflow Pipeline
+Start the gaiax-pipeline (deprecated)
 - To modify the airflow pipeline you have to touch the gaiax-pipeline.py file inside the dags folder
-```
-cd devops
-docker-compose up -d
-```
+  ```
+  cd devops
+  docker-compose up -d
+  ```
 
-## Simple SelfDescription validator
+### Validation
 
-Generated SelfDescriptions could be validated against their schemas (shapes) by the 
+Generated GX Credentials could be validated against their schemas (shapes) by the 
 simple SD validator script. Visit the `sd` directory and try to validate your 
 generated SD. Find the examples in `sd` directory and do the validation as follows:
-```bash
-./sd/validate.py sd/example.jsonld sd/example.ttl
-```
 
-### GX SelfDescription - Service Offering minimal example
 
-SD definition `sd/gx_service_offering_example.jsonld` should represent
-a minimal GX Service Offering example that is valid against the latest GX shacl shapes `sd/gx_shapes_latest.ttl`.
-The latest GX shacl shapes (at the time of Hackathon#6 23/05/3-4) are
-used by the [GX wizard](https://wizard.lab.gaia-x.eu/), and they have been downloaded from the [GX registry](https://registry.lab.gaia-x.eu/v1/api/trusted-shape-registry/v1/shapes/trustframework).
 
 Try to validate a minimal example against the latest GX shapes (feel free to remove some
 required attribute and check validation result):
 ```bash
 ./sd/validate.py sd/gx_service_offering_example.jsonld sd/gx_shapes_latest.ttl
 ```
 
+
+## Compliance
+
+GX Credential Generator creates credentials compliant with the latest (3024/01/19) Credential Schema, which can be downloaded from the [GX registry](https://registry.lab.gaia-x.eu/v1/api/trusted-shape-registry/v1/shapes/trustframework).
+GX Credential Generator **does not** create W3C complaint [Verifiable Credentials](https://www.w3.org/TR/vc-data-model/).
+
+## Configuration
+
+GX Credential generator is configured by `config.yaml`. The configuration includes:
+
+- Mandatory: Default values for mandatory attributes
+- Optional: Values for optional attributes
+- Optional: Wallets
+
+### Mandatory Attributes
+Gaia-X Credential schema dictates mandatory attributes for some class. If values for mandatory attributes can not be access from OpenStack or K8S cluster, default values are taken from configuration file in section `default`.
+Providers are able to change default values. In doing so, attribute values for **ALL** instances of impacted cloud resource are modified. 
+
+#### CopyrightOwner, License and ResourcePolicy of VM images
+
+`copyrigthowner`, `license` and `resourcePolicy` are mandatory attributes for VM Images and their operating systems. As these values are not accessible from OpenStack cloud, default values are used. The values for operating system are defined in section `operating system` with one section for each operating system. Operating systems are refered by name, e.g. for Alpine Linux. 
+
+```yaml
+default:
+  operating system:
+    Alpine Linux:
+      copyright owner: "Alpine Linux"
+      resource policy: "default: allow intent"
+      license:
+        - https://gitlab.alpinelinux.org/alpine/aports/-/issues/9074
+```
+
+By default, generator uses operating system values for VM Image as well. I.e. by default, VM Image and operating system have the same values for `copyrigthOwner`, `license` and `resourcePolicy`. Providers are able to change values for each VM image, individually. Therefore, the section `own images` in `cloud resources` exists. To set individual values for a specific VM image, add a new section, started by image's name (as defined in OpenStack cloud) to configuration file. The following example defines ìndividual values for `copyrigthOwner`, `license` and `resourcePolicy` for VM image called `AlmaLinux 8`.
+
+```yaml
+cloud resources:
+  own images:
+    AlmaLinux 8:
+      copyright owner:
+        - "AlmaLinux OS Foundation"
+        - "ABC"
+      resource policy: "abc"
+      license:
+          - https://www.abc.org
+```
+
+### Optional Attributes
+
+Similar to mandatory attributes, GX Credential Schemas supports optional attributes, whose values can not be retrieved from OpenStack cloud. These values can be set in configuration file in section `cloud resources` as well.
+
+#### AggregationOf of VM Images
+
+GX Credential schema allows to describe cloud resources, such VM images, in a fine granular manner, by using attribute `aggregationOf`. This attribute refers a list of GX credentials either as [DID](https://www.w3.org/TR/did-core/) or as file path relative to generators working directory. The latter is copied to cloud resource's credentials. For more details on GX credential schema and its attributes, please refer [Gaia-X Service Characteristics GitLab](https://gitlab.com/gaia-x/technical-committee/service-characteristics)   
+
+[//]: <> (### Wallets)
+[//]: <> (Wallets are digital secure storages to store Verifiable Credentials, like GX credentials. GX generator stores generated GX credentials in all configured wallets. Wallets are configured in section `wallets`. The following wallets are supported:)
+
+[//]: <> (- File System:
+  - Stores credentials on dedicated folder in disk. Use this wallet for testing purpose only.
+) 
+
+
 ## Docker
 
 The docker environment creates a general and portable environment for the gx-sd-generator module. Before running the container, don't forget to mount your credentials for the correct path. OpenStack-related secret located under `~/.config/openstack`
@@ -149,6 +195,14 @@ mkdir -p os_secret && cp secret1 /os_secret
 docker run -v "${PWD}/os_secret:/root/.config/openstack" $(docker build -q .)  ./gx-sd-generator.py --os-cloud gx-h61.1
 ```
 
+Generated SelfDescriptions could be validated against their schemas (shapes) by the 
+
+simple SD validator script. Visit the `sd` directory and try to validate your 
+generated SD. Find the examples in `sd` directory and do the validation as follows:
+```bash
+./sd/validate.py sd/example.jsonld sd/example.ttl
+```
+
 ## Status (2023-05-04)
 The current PoC code can discover OpenStack capabilities and produces
 an entry for the services in the service catalogue, with name,
@@ -167,7 +221,6 @@ https://signer.demo.delta-dao.com/#signer
 
 From an OpenStack perspective, this still incomplete.
 - We lack flavor details (though we need SCS specs to discover more)
-- We lack a list of public images (along with image details)
 - Neutron probably has a few things to detect.
 
 During Hackathon#6, the JSON-LD was updated match the current