Skip to content

Commit

Permalink
Update structure, grammar and tabs (#345)
Browse files Browse the repository at this point in the history
Most of the documents needed small updates either for text flow, grammar, tabs or structure.

Signed-off-by: Hannes Baum <hannes.baum@cloudandheat.com>
  • Loading branch information
cah-hbaum committed Jan 25, 2024
1 parent a3132da commit 1801efc
Show file tree
Hide file tree
Showing 8 changed files with 132 additions and 149 deletions.
99 changes: 46 additions & 53 deletions Standards/scs-0200-v1-sonobuoy-kaas-conformance-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ In order to use the existing conformance tests as a Sonobuoy plugin, a wrapper
around the individual test scripts would be required. This wrapper would need to
have the following effects:

* Gathers all test results and provides them in the results file
* Run tests in sequence and signal the worker when it is finished by generating a "done" file
* gathers all test results and provides them in the results file
* run tests in sequence and signal the worker when it's finished by generating a "Done" file

Apart from providing the test results, a plugin container must also forward the
status of each test by setting a status flag in the results file.
Expand All @@ -49,9 +49,10 @@ There are different approaches to create a Sonobuoy plugin, which are discussed
below in order to find a best practice for the SCS project. The documented approaches
show one example each in order to give a better representation to the reader.

Sonobuoy provides plugin examples in the repository <https://github.com/vmware-tanzu/sonobuoy-plugins>, which are referenced throughout this section.
Sonobuoy provides plugin examples in the repository <https://github.com/vmware-tanzu/sonobuoy-plugins>,
which are referenced throughout this section.

### _Option 1_ Go approach 1: Pick framework from the Sonobuoy plugin examples
### _Option 1_ Go approach 1: Pick a framework from the Sonobuoy plugin examples

The seemingly most interesting plugin is the [e2e-skeleton][e2e-skel], which uses
the [kubernetes-sigs/e2e-framework][e2e-frame]. The [kubernetes-sigs/e2e-framework][e2e-frame]
Expand All @@ -60,26 +61,26 @@ The framework provides proper documentation as well as helper functions that abs
client functionalities, similar to those found in "kubernetes/kubernetes/test/e2e/framework" repository.

As mentioned in the [motivation][e2e-frame-motivation] of the [e2e-framework][e2e-frame],
the project was created to circumvent the disadvantages of [kubernetes' own e2e-tests][k8s-e2e-tests],
the project was created to circumvent the disadvantages of [Kubernetes' own e2e-tests][k8s-e2e-tests],
which are described in more detail in the [goals][e2e-frame-goals].

PROS:

* arguments in favor of this framework can also be found under the [goals description][e2e-frame-goals] of the documentation
* [e2e-framework][e2e-frame] is a well defined framework, that allows the handling of resource creation and deletion
* [e2e-framework][e2e-frame] is a well-defined framework, that allows the handling of resource creation and deletion
* official framework provided by "Kubernetes-sigs"

CONS:

* compared to Python, Go is a less common programming language
* arguments not in favor of this framework can also be derived from the [Non-Goals][e2e-frame-nongoals] description of the documentation:
* "no responsibility for bootstrapping or the execution of the tests themselves" can be ignored, as this is partly taken over by Sonobuoy
* "no mock or fake cluster components" can be ignored, since the e2e tests of SCS should be used to test real clusters and their functionality
* for this test procedure, the Sonobuoy e2e plugin should be run in addition to the SCS kaas conformance tests
* "no responsibility for bootstrapping or the execution of the tests themselves" can be ignored, as this is partly taken over by Sonobuoy
* "no mock or fake cluster components" can be ignored, since the e2e tests of SCS should be used to test real clusters and their functionality
* for this test procedure, the Sonobuoy e2e plugin should be run in addition to the SCS kaas conformance tests

> proof of concept: `../Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/`
### _Option 2_ Go approach 2: Reuse the kubernetes own e2e test infrastructure and framework
### _Option 2_ Go approach 2: Reuse the Kubernetes own e2e test infrastructure and framework

The existing Sonobuoy e2e plugin already provides a vast number of tests that could
be adapted or reused for the SCS project.
Expand All @@ -106,7 +107,7 @@ CONS:

* not easy to implement, as we would have to copy part of the Kubernetes repository and track the changes from the upstream
* according to [README.md](https://github.com/kubernetes/kubernetes/tree/master/cluster#readme), part of it seems to be outdated and might change with a future version
* compared to _option 1_, the [goals][e2e-frame-goals] of the [e2e-framework][e2e-frame] can be seen as the disadvantages of using [Kubernetes' own e2e-tests][k8s-e2e-tests].
* compared to _option 1_, the [goals][e2e-frame-goals] of the [e2e-framework][e2e-frame] can be seen as the disadvantages of using [Kubernetes' own e2e-tests][k8s-e2e-tests].

> TODO: provide proof of concept: _kaas-sonobuoy-go-example-k8s-e2e_
Expand Down Expand Up @@ -134,7 +135,7 @@ used for Python, which should be decided in a secondary Decision Record.
PROS:

* continue using the already available Python tests
* only a small number of tests is implemented thus far
* only a small number of tests is implemented thus far

CONS:

Expand All @@ -143,85 +144,77 @@ CONS:

## Pros and Cons of Different Approaches

### Providing Sonobuoy plug in image
### Providing Sonobuoy plugin image

There are two potential approaches for building the Sonobuoy images.
They are completely independent of which of the above-mentioned frameworks is selected.
Here we discuss the pros and cons of these two approaches.
There are two potential approaches for building the Sonobuoy images, which can be selected
independent of the chosen framework. The pros and cons of the approaches can be seen
in the following section.

#### _Option 1_ GitHub container registry

Make the image available via the container registry on GitHub.
Hence we would need to apply a CI/CD job to build the images.
The image can be made available via the GitHub container registry, which would require
a regular job (probably CI/CD) to build and publish the image.

PROS

* The tests do not have to be created each time before usage.

CONS

* _?_

#### _Option 2_ local image upload
#### _Option 2_ Local image upload

Create the image locally on the "clusterctl admin control node" and then upload
it manually to the Kubernetes cluster under test.

PROS

* _?_

CONS
Both approaches are useful in different ways. While the usage of a container registry
allows easy distribution of tests and guarantees new images through the usage of a
CI/CD job, it also makes it harder to test changes quickly, since perhaps a wait time
is necessary to let the CI/CD job run through and access the image. This can be solved
by using the "Local image upload" or a combination of both approaches.

* To be able to use the tests, you always have to build them first
In general, the usage of the "GitHub container registry" option is highly recommended,
since it enables easy building and distribution of test images for different processes.

## Decision

The KaaS conformance test MUST be provided as a test suite holding the
test cases for the Kubernetes clusters to be checked.
Furthermore, the test cases themselves MUST be wrapped by a test framework to:

* Handle the creation and deletion of resources
* Collect and present results
* Consolidate redundant code across test cases
* Support the creation of test cases through predefined structures
* handle the creation and deletion of resources
* collect and present results
* consolidate redundant code across test cases
* support the creation of test cases through predefined structures

As with the [k8s-cluster-api-provider][k8s-api] the SCS provides a tooling to generate
its KaaS infrastructure. Part of the [k8s-cluster-api-provider][k8s-api] is the usage
of Sonobuoy as a test suite to execute the [Kubernetes own e2e tests][k8s-e2e-tests].
Investigating those e2e test lead to the conclusion that they are not always reusable.
The main purpose of Kubernetes own e2e tests is to test the functionality of
the Kubernetes code itself and not the resources and setup of a specific KaaS infrastrcture
the Kubernetes code itself and not the resources and setup of a specific KaaS infrastructure
as it is the aim of the SCS KaaS conformance test.

However considering that the SCS has an ongoing process of defining standards and
implementing test cases to check their compliance. In some cases future standards
might already be covored by the e2e test inside the Kubernetes repository.
Hence before writing tests, a developer SHOULD check the kubernetes e2e tests for
existing test cases that might cover conformance of a standard he is currently
working on.
However, considering that the SCS has an ongoing process of defining standards and
implementing test cases to check their compliance, that means in some cases future standards
might already be covered by the e2e test inside the Kubernetes repository.
Hence, before writing tests, a developer SHOULD check the Kubernetes e2e tests for
existing test cases that might cover conformance of a standard currently being worked on.

As described above, Sonobuoy offers the possibility to generate custom plugins
that provide self-created test cases. Therefore, future conformance tests MUST
be able to be executed by Sonobuoy by wrapping them in a Sonobuoy plugin.
be executable by Sonobuoy by wrapping them in a Sonobuoy plugin.
Moreover, three options for the implementation of the SCS KaaS compliance test
cases are described above so that it can be decided in this decision record which
cases are described above so that it can be decided within this decision record which
options should be used.

As a first decision, "_Option 2_ Go Approach 2: Reuse Kubernetes' own e2e test infrastructure and framework"
is the least viable, as it would mean copying almost all of the files used for
is the least viable, as it would mean copying almost all the files used from
the Kubernetes e2e tests. This framework is closely linked to the development of
the Kubernetes code. Therefore, changes to its structure mainly concerns its use
within the Kubernetes repository itself and not its use for other parties.
the Kubernetes code. Therefore, changes to its structure mainly are in line with its usage
in the Kubernetes repository itself and is probably not relevant for other parties.
Changes in the framework cloud have a greater impact on our side as they are
predictable. The development effort gained by reusing the many examples could be
outweighed by the effort that might be invested in the future to adapt our tests
to corresponding framework changes.
predictable. The development effort gained by reusing these examples could be
outweighed by the investment necessary to adapt tests to the corresponding framework changes.

This leaves "_Option 1_ Go approach 1: Pick framework from the Sonobuoy plugin examples"
and "_Option 3_ Write Python scripts for tests" as methods for implementing test cases.
Both have their advantages, so it could also be considered to use both and thus
generate two Sonobuoy plugins.
It is possible to use both approaches in parallel, which would enable usage of both Python
and Go code. This could possibly generate two Sonobuoy plugins, but this isn't a necessity.

## Documents

Expand Down
2 changes: 1 addition & 1 deletion Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ KUBERNETES_SERVICE_PORT=34743
ifeq ($(IMAGE_VERSION_TAG),)
export TAG=dev
else
export TAG=${IMAGE_VERSION_TAG}
export TAG=${IMAGE_VERSION_TAG}
endif

SONOBUOY_IMAGE = "${IMAGE_REGISTRY}/${IMAGE_NAME}:${TAG}"
Expand Down
51 changes: 24 additions & 27 deletions Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/README.md
Original file line number Diff line number Diff line change
@@ -1,88 +1,85 @@
# SCS kaas sonobuoy testsuie and k8s e2e-test-framework framework
# SCS kaas Sonobuoy testsuite and K8s e2e-test-framework

This directory holds all conformance tests that aim to validate the scs kaas infrastructure

## From Sonobuoy: Plug description
## Sonobuoy plugin description

> NOTE: There is a Sonobuoy blog post walking through this plugin, its benefits, and how to use it. See: <https://sonobuoy.io/plugin-starter/>
### Custom End-To-End (E2E) Tests
### Custom End-to-End (E2E) Tests

This plugin is meant as a skeleton for you to grab and run with to implement your
own custom tests in Kubernetes.
This plugin is intended as a skeleton to implement custom tests in Kubernetes.

The benefits of using this plugin instead of starting from scratch:
The benefits of using this plugin instead of starting from scratch are:

* Automatically comes with the [e2e-test-framework](https://github.com/kubernetes-sigs/e2e-framework) imported/configured
* Includes basic examples so you don't have to look up basic boilerplate
* Automatically comes with a Dockerfile and plugin.yaml so there is less overhead to getting started
* Will get support as the e2e-test-framework and Sonobuoy evolve to get the best features supported by default
* automatically comes with the [e2e-test-framework](https://github.com/kubernetes-sigs/e2e-framework) imported/configured
* includes basic examples, so you don't have to look up boilerplate code
* comes with a Dockerfile and plugin.yaml, meaning there is less overhead to getting started
* long-time support, since the e2e-test-framework and Sonobuoy are still actively maintained

### How to use this plugin

* Write tests (using main_test.go as a jumping off point)
* Run ./build.sh to build the image and push it to your registry
* `sonobuoy run -p plugin.yaml` to run your own plugin
1) Write tests (using the main_test.go as a starting point)
2) Run ./build.sh to build the image and push it to your registry
3) `sonobuoy run -p plugin.yaml` to run your own plugin

## sonobuoy usage for development of tests
## Sonobuoy usage for development of tests

The development is based on the useage of [kind](https://kind.sigs.k8s.io/) as a test cluster.
For test development with Sonobuoy, [KinD](https://kind.sigs.k8s.io/) is used as a test cluster.

* (Optional): check if all pre requests for development are met and create a kind test cluster
* (Optional): check if all prerequisites for the development are met and create a `KinD` test cluster

```bash
make dev-prerequests
make dev-setup
```

1. Set enviornment variables
1. Set environment variables

```bash
export IMAGE_VERSION_TAG="dev"
export K8S_HOST=<kind-cluster-ip>
export K8S_PORT=<kind-cluster-port>
```

2. Build the image and upload it to the kind cluster

This rule
2. Build the image and upload it to the KinD cluster

```bash
make dev-build
```

3. Execute the sonobuoy plugin
3. Execute the Sonobuoy plugin

```bash
make dev-run
```

This lunches the sonobuoy plugin on the kind cluster in the background
If you want to see the current status of the plugin you can do so by:
This launches the Sonobuoy plugin on the KinD cluster in the background.
If you want to see the current status of the plugin you can do so by:

```bash
sonobuoy status
```

4. Retrieve the Results

Once sonobuoy is done running the plug in you can retrieve the results as following:
Once Sonobuoy is done running the plugin you can retrieve the results as following:

```bash
make dev-result
```

5. Clean the Sonobuoy testcase form the kind cluster
5. Clean the Sonobuoy testcase from the KinD cluster

Cleaning up all Kubernetes resources which were placed on the kind cluster by sonobuoy
Cleaning up all Kubernetes resources which were placed on the KinD cluster by sonobuoy

```bash
make dev-clean
```

6. Purge everything

Deleting the kind cluster
Deleting the KinD cluster

```bash
make dev-purge
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"sigs.k8s.io/e2e-framework/pkg/envconf"
)


const (
ProgressReporterCtxKey = "SONOBUOY_PROGRESS_REPORTER"
NamespacePrefixKey = "NS_PREFIX"
Expand Down Expand Up @@ -39,46 +40,44 @@ func TestMain(m *testing.M) {
})

testenv.BeforeEachTest(func(ctx context.Context, cfg *envconf.Config, t *testing.T) (context.Context, error) {
fmt.Println("BeforeEachTest")
fmt.Println("BeforeEachTest")
updateReporter.StartTest(t.Name())
return createNSForTest(ctx, cfg, t, runID)
})

testenv.AfterEachTest(func(ctx context.Context, cfg *envconf.Config, t *testing.T) (context.Context, error) {
fmt.Println("AfterEachTest")
fmt.Println("AfterEachTest")
updateReporter.StopTest(t.Name(),t.Failed(),t.Skipped(),nil)
return deleteNSForTest(ctx, cfg, t, runID)
})

/*
/*
testenv.BeforeEachFeature(func(ctx context.Context, config *envconf.Config, info features.Feature) (context.Context, error) {
// Note that you can also add logic here for before a feature is tested. There may be
// more than one feature in a test.
fmt.Println("BeforeEachFeature")
fmt.Println("BeforeEachFeature")
return ctx, nil
})
testenv.AfterEachFeature(func(ctx context.Context, config *envconf.Config, info features.Feature) (context.Context, error) {
// Note that you can also add logic here for after a feature is tested. There may be
// more than one feature in a test.
fmt.Println("AfterEachFeature")
fmt.Println("AfterEachFeature")
return ctx, nil
})
*/
*/

testenv.Finish(
// Teardown func: delete kind cluster
func(ctx context.Context, cfg *envconf.Config) (context.Context, error) {

fmt.Println("Finished go test suite")
fmt.Println("Finished go test suite")
//~ if err := ???; err != nil{
//~ return ctx, err
//~ }
return ctx, nil
},
)

os.Exit(testenv.Run(m))
}

Expand Down
Loading

0 comments on commit 1801efc

Please sign in to comment.