Refine regulations regarding continuous testing and result tracking (…

…see #346)

Signed-off-by: Matthias Büchse <matthias.buechse@cloudandheat.com>

Standards/scs-xxxx-v1-achieving-certification.md

@@ -1,37 +1,40 @@
 ---
-title: Regulations for achieving SCS certification
+title: Regulations for achieving SCS-compatible certification
 type: Procedural
 status: Draft
 track: Global
 ---
 
 ## Introduction
 
-The Sovereign Cloud Stack (SCS) issues certificates of various types, among them _SCS-compatible IaaS_ (infrastructure as a service) and _SCS-compatible KaaS_ (Kubernetes as a service).
+The Sovereign Cloud Stack (SCS) issues certificates with various scopes, among them _SCS-compatible IaaS_ (infrastructure as a service) and _SCS-compatible KaaS_ (Kubernetes as a service).
 
-This document details how a cloud service provider (henceforth, CSP) can attain such a certificate for one of their clouds.
+This document details how a cloud service provider (henceforth also called operator) can attain such a certificate for one of their clouds.
 
 ## Motivation
 
-As Operator, I want to achieve the Certificate SCS compatible.
+As operator, I want to obtain a certificate with the scope SCS-compatible IaaS or SCS-compatible KaaS.
 
 ## Regulations
 
-1. Each certificate issued pertains to a certain cloud, a certain type (here, SCS-compatible IaaS or SCS-compatible KaaS), and a certain version of that type with a fixed expiry date. The certificate is only valid for that cloud and for the time frame that ends on that expiry date.
+1. Each certificate issued pertains to a given cloud, a given scope, and a given version of that scope with a fixed expiry date. The certificate is only valid for that cloud and for the time frame that ends on that expiry date.
 
-2. For achieving the certificate "SCS compatible" an operator MUST include the official SCS compliance test suite in his continuous test infrastructure – for public clouds the recommended way to achieve this is to offer the SCS project access to the infrastructure so the test suite runs can be triggered continuously by the SCS team and ensure continuous compliance. For non-public clouds, the results (log files) can be submitted to SCS by a mechanism of SCS' choice and need to be reproduced again on request by SCS.
-
-   **FIXME:** Mention the possibility to create a Zuul project and add nodepools to Zuul? How would that even work?
+2. The operator MUST include the official SCS compliance test suite (which does not require admin privileges) in their continuous test infrastructure (e.g., Zuul). The tests MUST be run at least nightly. For public clouds, it is recommended to offer the SCS project access to the infrastructure (ideally, including a Zuul nodepool) so the test suite runs can be triggered continuously by the SCS team. Alternatively, and for non-public clouds, the results (log files) MUST be submitted to SCS by a mechanism of SCS' choice and need to be reproduced again on request by SCS.
 
    <!-- Initially this will probably be eMail -->
-   <!-- What happens when the tests fail? Who will be notified, will the certificate be revoked after a given amount of time? -->
 
-3. The SCS certification assessment body (or entities empowered to do so) WILL review the certification application and either grant the certification, reject it or ask for further measures or information.
+3. If the desired certificate requires manual checks, then the operator MUST offer the SCS project suitable access. Manual checks SHOULD be repeated once every 3 months.
+
+4. Details on the standards achieved, as well as the current state and the history of all test and check results will be displayed on a public webpage (henceforth, _certificate status page_) owned by SCS.
+
+5. The SCS certification assessment body (or entities empowered to do so) WILL review the certification application and either grant the certification, reject it or ask for further measures or information.
    <!-- body might as well just be a machine, at least on the scs compatible level -->
 
-4. After having received a confirmation of a successful achievement of a certificate granted by the SCS certification assessment body, the operator MAY use the corresponding logo and publicly state the certified "SCS compatibility" on the respective layer for the time of the validity of the certification. The logos will contain a link to SCS-owned pages that contain details on the achieved standards, either by embedding a hyperlink or a QR code for printed assets.
+6. Once the certificate is granted by the SCS certification assessment body, the operator MAY use the corresponding logo and publicly state the certified "SCS compatibility" on the respective layer for the time of the validity of the certification. The logo MUST be accompanied by a hyperlink (a QR code for printed assets) to the respective certificate status page.
+
+7. If the certificate is to be revoked for any reason, it will be included in a publicly available Certificate Revokation List (CRL). This fact will also be reflected in the certificate status page.
 
-   <!-- Make clear that we – the SCS project – are allowed to test a cloud enviroment on request or request docs that prove the fulfillment of certification requirements -->
+8. If any of the automated tests or manual checks fail after the certificate has been issued, the certificate is not immediately revoked. Rather, the automated tests MUST pass 99.x % of the runs, and the operator SHALL be notified at the second failed attempt in a row at the latest. In case a manual check fails, it has to be repeated at a date to be negotiated with SCS. It MAY NOT fail more than two times in a row.
 
 ## Design Considerations