✨ Add zuul e2e for openstack (#53)

* Add zuul e2e for openstack Signed-off-by: Matej Feder <matej.feder@dnation.cloud> * Install and build csctl-openstack from separate repository Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Use openstack-csp-helper chart Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add cluster-stack template Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add cluster template Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add capo_version into variables Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Cleanup workload cluster Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Update csctl plugin branch Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add block in ansible to create workload cluster and execute checks Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Import sonobuoy tests Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Import scs-compliance pre-tasks and tasks Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Debug sonobouy retrieve error in zuul pipeline Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Enable scs-compliance tests in zuul Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Increase pause to be able to retrieve results from sonobuoy Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Ensure that pip3 is installed Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add zuul pipeline timeout Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Wait for sonobuoy results and insert them to the warning message Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Change how to get kubeconfig of the workload cluster Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Increase wait_for_cluster timeout Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Override cluster stack if defined in Zuul config Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add server groups to nodes Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Use helm to create secrets and ClusterResourceSet Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Import task for creating a server groups Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Use helm via shell to create secrets and ClusterResourceSet Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Change zuul config and also organization to SCS Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Update .ansible-lint Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update .zuul.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update .zuul.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update .zuul.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/tasks/create_server_groups.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/openstack/templates/cluster-stack-template.yaml.j2 Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/openstack/e2e.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/openstack/e2e.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Update playbooks/openstack/e2e.yaml Co-authored-by: Roman Hros <roman.hros@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> * Fix zuul config Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Remove unnecessary --os-compute-api-version flag Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Remove hardcoded timeouts and use variables instead Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Parse k8s version to format major.minor Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Parse cluster_stack_version_name from zuul config Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Wait for clusteraddons resource to become ready Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Workaround until the csctl binary can be built from the main branch again Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add some retries to the task called Get kubeadmcontrolplane name Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add cleanup run into the pipeline Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add wait for control-plane machines to be ready Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add cloud_name variable to cleanup.yaml playbook Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Install csctl and csctl-openstack via releases Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Fix retries keyword issue for ansible versions older then 9 Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> * Add cleanup into always block as well Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> --------- Signed-off-by: Matej Feder <matej.feder@dnation.cloud> Signed-off-by: michal.gubricky <michal.gubricky@dnation.cloud> Signed-off-by: Michal Gubricky <michal.gubricky@dnation.cloud> Co-authored-by: michal.gubricky <michal.gubricky@dnation.cloud> Co-authored-by: Roman Hros <roman.hros@dnation.cloud>
SovereignCloudStack · Jun 17, 2024 · 3699457 · 3699457
1 parent 114ea2b
commit 3699457
Show file tree

Hide file tree

Showing 19 changed files with 908 additions and 0 deletions.
diff --git a/.ansible-lint b/.ansible-lint
@@ -0,0 +1,10 @@
+---
+use_default_rules: true
+skip_list:
+  - yaml # disabled because we use yamllint
+# Roles and modules imported from https://opendev.org/zuul/zuul-jobs
+mock_roles:
+  - ensure-docker
+  - ensure-go
+mock_modules:
+  - zuul_return
diff --git a/.github/workflows/ansible-lint.yml b/.github/workflows/ansible-lint.yml
@@ -0,0 +1,21 @@
+---
+name: Ansible lint
+
+"on":
+  push:
+    branches:
+      - main
+    paths:
+      - 'playbooks/**'
+  pull_request:
+    paths:
+      - 'playbooks/**'
+
+jobs:
+  build:
+    name: Ansible Lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run ansible-lint
+        uses: ansible/ansible-lint@v24
diff --git a/.github/workflows/check-yaml-syntax.yml b/.github/workflows/check-yaml-syntax.yml
@@ -0,0 +1,25 @@
+---
+name: Check yaml syntax
+
+"on":
+  push:
+    branches:
+      - main
+    paths:
+      - '**.yaml'
+      - '**.yml'
+  pull_request:
+    paths:
+      - '**.yaml'
+      - '**.yml'
+
+jobs:
+  check-yaml-syntax:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+      - run: pip3 install yamllint
+      - run: yamllint .
diff --git a/.yamllint.yml b/.yamllint.yml
@@ -0,0 +1,13 @@
+---
+extends: default
+
+rules:
+  comments: enable
+  line-length: disable
+  # accept both     key:
+  #                   - item
+  # and             key:
+  #                 - item
+  # (the latter is very common in k8s land)
+  indentation:
+    indent-sequences: whatever
diff --git a/.zuul.yaml b/.zuul.yaml
@@ -0,0 +1,72 @@
+---
+- job:
+    name: openstack-e2e-abstract
+    abstract: true
+    parent: openstack-access-base
+    description: |
+      An abstract job for e2e testing of cluster stacks project.
+      This job is not intended to be run directly, but instead
+      must be inherited from it.
+    pre-run: playbooks/dependencies.yaml
+    run: playbooks/openstack/e2e.yaml
+    cleanup-run: playbooks/openstack/cleanup.yaml  # executed also when the job is canceled
+    vars:
+      wait_for_cluster_stack_resource: 120  # 2min
+      wait_for_clusteraddons: 120  # 2min
+      wait_for_cluster_stack: 1440  # 24min
+      wait_for_cluster: 600  # 10min
+      sonobouy:
+        enabled: false
+      scs_compliance:
+        enabled: false
+
+- job:
+    name: e2e-openstack-conformance
+    parent: openstack-e2e-abstract
+    description: |
+      Run e2e tests of cluster-stacks project using
+      [sonobuoy](https://sonobuoy.io/) with mode conformance and
+      SCS compliance checks meaning it will test if the Kubernetes
+      cluster is conformant to the CNCF and to the SCS.
+    timeout: 10800  # 3h
+    vars:
+      wait_for_cluster: 1200  # 20min
+      sonobouy:
+        enabled: true
+        mode: conformance
+      scs_compliance:
+        enabled: true
+
+- job:
+    name: e2e-openstack-quick
+    parent: openstack-e2e-abstract
+    description: |
+      Run e2e tests of cluster-stacks project using
+      [sonobuoy](https://sonobuoy.io/) with mode quick and
+      SCS compliance checks.
+    timeout: 7200  # 2h
+    vars:
+      wait_for_cluster: 1200  # 20min
+      sonobouy:
+        enabled: true
+        mode: quick
+      scs_compliance:
+        enabled: true
+
+
+- project:
+    name: SovereignCloudStack/cluster-stacks
+    default-branch: main
+    merge-mode: "squash-merge"
+    e2e-test:
+      jobs:
+        - e2e-openstack-conformance
+    unlabel-on-update-e2e-test:
+      jobs:
+        - noop
+    e2e-quick-test:
+      jobs:
+        - e2e-openstack-quick
+    unlabel-on-update-e2e-quick-test:
+      jobs:
+        - noop
diff --git a/playbooks/dependencies.yaml b/playbooks/dependencies.yaml
@@ -0,0 +1,61 @@
+---
+- name: Ensure cluster stacks dependencies
+  hosts: all
+  vars:
+    kind_version: "0.22.0"
+    kubectl_version: "1.29.3"
+    clusterctl_version: "1.7.2"
+    helm_version: "3.14.4"
+    yq_version: "4.44.1"
+    envsubst_version: "1.4.2"
+    install_dir: "{{ ansible_user_dir }}/.local/bin"
+  roles:  # https://opendev.org/zuul/zuul-jobs
+    - role: ensure-docker
+    - role: ensure-go
+      vars:
+        go_version: 1.21.6
+  environment:
+    PATH: "{{ install_dir }}:{{ ansible_env.PATH }}"
+  tasks:
+    - name: Make sure installation directory exists
+      ansible.builtin.file:
+        path: "{{ install_dir }}"
+        state: directory
+        mode: 0755
+    - name: Install clusterctl
+      ansible.builtin.get_url:
+        url: "https://github.com/kubernetes-sigs/cluster-api/releases/download/v{{ clusterctl_version }}/clusterctl-linux-amd64"
+        dest: "{{ install_dir }}/clusterctl"
+        mode: "+x"
+    - name: Install envsubst
+      ansible.builtin.get_url:
+        url: "https://github.com/a8m/envsubst/releases/download/v{{ envsubst_version }}/envsubst-Linux-x86_64"
+        dest: "{{ install_dir }}/envsubst"
+        mode: "+x"
+    - name: Install yq
+      ansible.builtin.get_url:
+        url: "https://github.com/mikefarah/yq/releases/download/v{{ yq_version }}/yq_linux_amd64"
+        dest: "{{ install_dir }}/yq"
+        mode: "+x"
+    - name: Install KinD
+      ansible.builtin.get_url:
+        url: "https://kind.sigs.k8s.io/dl/v{{ kind_version }}/kind-linux-amd64"
+        dest: "{{ install_dir }}/kind"
+        mode: "+x"
+    - name: Install kubectl
+      ansible.builtin.get_url:
+        url: "https://dl.k8s.io/release/v{{ kubectl_version }}/bin/linux/amd64/kubectl"
+        dest: "{{ install_dir }}/kubectl"
+        mode: "+x"
+    # TODO: Install csctl and csctl-openstack from the release when it will be available
+    - name: Install csctl and csctl-openstack
+      ansible.builtin.import_tasks: tasks/csctl.yaml
+    - name: Install helm
+      ansible.builtin.unarchive:
+        src: "https://get.helm.sh/helm-v{{ helm_version }}-linux-amd64.tar.gz"
+        dest: "{{ install_dir }}"
+        extra_opts: "--strip-components=1"
+        mode: "+x"
+        remote_src: true
+      args:
+        creates: "{{ install_dir }}/helm"
diff --git a/playbooks/openstack/cleanup.yaml b/playbooks/openstack/cleanup.yaml
@@ -0,0 +1,48 @@
+---
+- name: Cleanup
+  hosts: all
+  vars:
+    cloud_name: "{{ cloud }}"  # inherited from the parent job
+  environment:
+    PATH: "{{ ansible_user_dir }}/.local/bin:{{ ansible_env.PATH }}"
+  tasks:
+    - name: Delete server groups
+      when: scs_compliance.enabled
+      block:
+      - name: List existing server groups
+        ansible.builtin.command: "openstack server group list -f value -c Name -c ID"
+        register: server_groups
+        environment:
+          OS_CLOUD: "{{ cloud_name }}"
+        changed_when: true
+      - name: Parse test-cluster-controller srvgrp and assign ID to srvgrp_controller
+        ansible.builtin.set_fact:
+          srvgrp_controller: "{{ item.split(' ')[0] }}"
+        loop: "{{ server_groups.stdout_lines }}"
+        when: "server_groups is defined and server_groups.stdout_lines | length > 0 and 'test-cluster-controller' in item.split(' ')"
+      - name: Parse test-cluster-worker srvgrp and assign ID to srvgrp_worker
+        ansible.builtin.set_fact:
+          srvgrp_worker: "{{ item.split(' ')[0] }}"
+        loop: "{{ server_groups.stdout_lines }}"
+        when: "server_groups is defined and server_groups.stdout_lines | length > 0 and 'test-cluster-worker' in item.split(' ')"
+      - name: Delete Server Group for worker nodes
+        ansible.builtin.command: "openstack server group delete {{ srvgrp_worker }}"
+        environment:
+          OS_CLOUD: "{{ cloud_name }}"
+        when: srvgrp_worker is defined
+        changed_when: true
+      - name: Delete Server Group for control-plane nodes
+        ansible.builtin.command: "openstack server group delete {{ srvgrp_controller }}"
+        environment:
+          OS_CLOUD: "{{ cloud_name }}"
+        when: srvgrp_controller is defined
+        changed_when: true
+    - name: Check if test-cluster exists
+      ansible.builtin.command: "kubectl get cluster test-cluster"
+      register: cluster_check
+      ignore_errors: true
+      changed_when: true
+    - name: Cleanup workload cluster
+      ansible.builtin.command: "kubectl delete -f {{ ansible_user_dir }}/cluster.yaml"
+      when: cluster_check.rc == 0
+      changed_when: true