diff --git a/Standards/scs-0200-v1-sonobuoy-kaas-conformance-tests.md b/Standards/scs-0200-v1-sonobuoy-kaas-conformance-tests.md index 1515e9758..202851f2a 100644 --- a/Standards/scs-0200-v1-sonobuoy-kaas-conformance-tests.md +++ b/Standards/scs-0200-v1-sonobuoy-kaas-conformance-tests.md @@ -23,13 +23,13 @@ A Sonobuoy worker supervises this pod and forwards all test results to the aggre It does this by waiting for a specific "Done" file to be created. Once this file is recognized, the worker forwards the results to the aggregator, using a predefined location for the results file within a folder, as seen in following image: ![image search api](https://sonobuoy.io/img/plugin-contract.png) -Therefore, in order to apply the conformance tests as a plugin, we need a tooling/wrapper/framework around the individual test scripts that: +In order to use the existing conformance tests as a Sonobuoy plugin, a wrapper around the individual test scripts would be required. This wrapper would need to have the following effects: -* Gathers all test results and provides them in the results file. -* Run tests in sequence and signal the worker when it is finished by writing the "done file". +* Gathers all test results and provides them in the results file +* Run tests in sequence and signal the worker when it is finished by generating a "done" file -Apart from providing the test results, a plugin container must also pass on the status of each test by setting the status flag in the results file. -In addition, to run the tests as a Sonobuoy plugin, we need to create an image that can be run inside the k8s cluster under test. +Apart from providing the test results, a plugin container must also forward the status of each test by setting a status flag in the results file. +Additionally, the tests would need to be able to run inside a pod in the K8s cluster under test. ## Design Considerations @@ -38,27 +38,26 @@ These are described in more detail below, where we discuss the pros and cons of One of these three approaches must be chosen for implementation. For the purposes of the investigation and the preparation of this decision record, there are examples(proof of work) for each different approach. -#### _Option 1_ go approach [1]: Pick framework from the Sonobuoy plugin examples +#### _Option 1_ GO [1]: Pick framework from the Sonobuoy plugin examples -Sonobuoy provides plugin examples in the following repository: [https://github.com/vmware-tanzu/sonobuoy-plugins][sonobuoy-plugins-repo]. -The most suitable plugin for us is the [e2e-skeleton][e2e-skel], which uses the [kubernetes-sigs/e2e-framework][e2e-frame]. -The [kubernetes-sigs/e2e-framework][e2e-frame] is a stand-alone project that is separate from the official [kubernetes e2e tests][k8s-e2e-tests]. -It is suitable for us to make us of it, as it provides a properly documented framework. -In addition, it contains helper functions that abstract client go functions similar to those in the kubernetes/kubernetes/test/e2e/framework repository -As mentioned in the [motivation][e2e-frame-motivation] of the [e2e-framework][e2e-frame], it was created to circumvent the disadvantages of +The seemingly most interesting plugin is the [e2e-skeleton][e2e-skel], which uses the [kubernetes-sigs/e2e-framework][e2e-frame]. +The [kubernetes-sigs/e2e-framework][e2e-frame] is a stand-alone project that is separate from the official [Kubernetes e2e tests][k8s-e2e-tests]. +The framework provides proper documentation as well as helper functions that abstract client functionalities, similar to those found in "kubernetes/kubernetes/test/e2e/framework" repository. + +As mentioned in the [motivation][e2e-frame-motivation] of the [e2e-framework][e2e-frame], the project was created to circumvent the disadvantages of [kubernetes' own e2e-tests][k8s-e2e-tests], which are described in more detail in the [goals][e2e-frame-goals]. PROS: -- Arguments in favor of this framework can also be found under the [goals description][e2e-frame-goals] of the documentation -- [e2e-framework][e2e-frame] is a well defined framework who allows the handling of resources creation and deletion -- official framework provided by Kubernetes-sigs +- arguments in favor of this framework can also be found under the [goals description][e2e-frame-goals] of the documentation +- [e2e-framework][e2e-frame] is a well defined framework, that allows the handling of resource creation and deletion +- official framework provided by "Kubernetes-sigs" CONS: - compared to Python, Go is a less common programming language - arguments not in favor of this framework can also be derived from the [Non-Goals][e2e-frame-nongoals] description of the documentation: - - However, the point "No responsibility for bootstrapping or the execution of the tests themselves" can be ignored, as this is partly taken over by Sonobuoy. - - Also the point "no mock or fake cluster components" can be ignored as we want to use the e2e tests to test the Kubernetes cluster environment itself and not the functionality of the Kubernetes source code. - - For this test procedure we should run the Sonobuoy e2e plugin in addition to the SCS kaas conformance tests. + - "no responsibility for bootstrapping or the execution of the tests themselves" can be ignored, as this is partly taken over by Sonobuoy + - "no mock or fake cluster components" can be ignored, since the e2e tests of SCS should be used to test real clusters and their functionality + - for this test procedure, the Sonobuoy e2e plugin should be run in addition to the SCS kaas conformance tests > proof of work: [kaas-sonobuoy-go-example-e2e-framework](../Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/) @@ -81,14 +80,14 @@ More precisely from: PROS: - - The [Kubernetes' own e2e tests][k8s-e2e-tests] already provide a vast amount of examples from which we can develop our own tests - - Compared to _option 1_, the [non-goals][e2e-frame-nongoals] of the [e2e-framework][e2e-frame] can be seen as the advantages of using [Kubernetes' own e2e-tests][k8s-e2e-tests]. + - [Kubernetes' own e2e tests][k8s-e2e-tests] already provide a vast amount of examples, which could be reused to develop specific SCS tests + - compared to _option 1_, the [non-goals][e2e-frame-nongoals] of the [e2e-framework][e2e-frame] can be seen as the advantages of using [Kubernetes' own e2e-tests][k8s-e2e-tests]. CONS: - - not easy to implement, as we would have to copy part of the Kubernetes repository and track the changes there as well. - - According to [README.md](https://github.com/kubernetes/kubernetes/tree/master/cluster#readme), part of it seems to be outdated and might change in the future. - - Compared to _option 1_, the [goals][e2e-frame-goals] of the [e2e-framework][e2e-frame] can be seen as the disadvantages of using [Kubernetes' own e2e-tests][k8s-e2e-tests]. + - not easy to implement, as we would have to copy part of the Kubernetes repository and track the changes from the upstream + - according to [README.md](https://github.com/kubernetes/kubernetes/tree/master/cluster#readme), part of it seems to be outdated and might change with a future version + - compared to _option 1_, the [goals][e2e-frame-goals] of the [e2e-framework][e2e-frame] can be seen as the disadvantages of using [Kubernetes' own e2e-tests][k8s-e2e-tests]. > TODO: provide proof of work: _kaas-sonobuoy-go-example-k8s-e2e_ @@ -113,11 +112,12 @@ framework of our own. PROS: - - We can continue to use the current Python scripts - - However, only 2 scripts have been implemented so far. Rewriting them in Go should be a feasible task. + - continue using the already available Python tests + - only a small number of tests is implemented thus far CONS: - - We would need to write our own test framework + - no "native" support in Sonobuoy, a wrapper is needed + - decision for a framework is still not done ## Pros and Cons of Different Approaches