From 50ab0f5d560f2a6ec6a750f1c065925228472bc8 Mon Sep 17 00:00:00 2001 From: Hannes Baum Date: Tue, 30 Jan 2024 11:12:00 +0100 Subject: [PATCH] Examples for documents (#352) Rewrite some of the documents slightly to fit the structure of the standard. Signed-off-by: Hannes Baum --- Standards/scs-0100-v3-flavor-naming.md | 237 +++++++++--------- .../scs-0111-v1-volume-type-decisions.md | 48 ++-- Standards/scs-0210-v2-k8s-version-policy.md | 18 +- .../scs-0211-v1-kaas-default-storage-class.md | 19 +- Standards/scs-0302-v1-domain-manager-role.md | 32 +-- 5 files changed, 183 insertions(+), 171 deletions(-) diff --git a/Standards/scs-0100-v3-flavor-naming.md b/Standards/scs-0100-v3-flavor-naming.md index 1712f51e7..665bc926e 100644 --- a/Standards/scs-0100-v3-flavor-naming.md +++ b/Standards/scs-0100-v3-flavor-naming.md @@ -20,6 +20,14 @@ but try to avoid changing in incompatible ways. (See at the end for the v1 to v2 transition where we have not met that goal, but at least managed to have a 1:1 relationship between v1 and v2 names.) +## Terminology + +Error correction code memory (abbr. ECC memory) + Memory able to detect and correct n-bit data corruption + +L1 Terminal Fault (abbr. L1TF) + Speculative execution vulnerability on Intel processors + ## Motivation In OpenStack environments there is a need to define different flavors for instances. @@ -41,8 +49,8 @@ Note that not all relevant properties of flavors can be discovered; creating a s to address this is a separate but related effort to the name standardization. Commonly used infrastructure-as-code tools do not provide a way to use discoverability features to express something like "I want a flavor with 2 vCPUs, 8GiB of RAM, a local -20GB SSD disk and Infiniband support but I don't care whether it's AMD or intel" in a -reasonable manner. Using flavor names to express this will thus continue to be useful +20GB SSD disk and Infiniband support, but I don't care whether it's AMD or intel" in a +reasonable manner. Using flavor names to express this will thus continue to be useful, and we don't expect the need for standardization of flavor names to go away until the commonly used IaC tools work on a higher abstraction layer than they currently do. @@ -54,7 +62,7 @@ From discussions of our operators with their customers we learned that the following characteristics are important in a flavor description: | Type | Description | -| :---------------- | :----------------------------------------------------- | +|:------------------|:-------------------------------------------------------| | Generation | CPU Generation | | Number of CPU | Number of vCPUs - suffixed by L,V,T,C (see below) | | Amount of RAM | Amount of memory available for the VM | @@ -69,14 +77,14 @@ to them, we wanted to come up with a scheme that allows to systematically derive names from properties and vice versa. The scheme allows for short names (by not encoding all details) as well as very detailed longer names. -## Complete Proposal for systematic flavor naming +### Complete Proposal for systematic flavor naming -| Prefix | CPUs & Suffix | RAM[GiB] | optional: Disk[GB]&type | opt: extensions | -| ------ | ----------------- | ------------------ | ----------------------------- | ----------------| +| Prefix | CPUs & Suffix | RAM[GiB] | opt: Disk[GB]&type | opt: extensions | +|--------|-------------------|--------------------|-------------------------------|-----------------| | `SCS-` | N`L/V/T/C`\[`i`\] | `-`N\[`u`\]\[`o`\] | \[`-`\[M`x`\]N\[`n/h/s/p`\]\] | \[`_`EXT\] | Note that N and M are placeholders for numbers here. -The optional fields are denoted in brackets (and have opt: in the header. +The optional fields are denoted in brackets (and have opt: in the header). See below for extensions. Note that all letters are case-sensitive. @@ -84,20 +92,20 @@ Note that all letters are case-sensitive. Typical flavor names look like `SCS-4V-16-50` for a flavor with 4vCPUs (with limited oversubscription), 16GiB RAM and a 50GB disk (of unspecified type). -## Proposal Details +### Proposal Details -### [REQUIRED] CPU Suffixes +#### [REQUIRED] CPU Suffixes Next to the number of vCPUs, these vCPUs need to be characterized to describe their nature. | Suffix | Meaning | -| ------ | ----------------------------- | +|--------|-------------------------------| | C | dedicated Core | | T | dedicated Thread (SMT) | | V | vCPU (oversubscribed) | | L | vCPU (heavily oversubscribed) | -#### Baseline +##### Baseline Note that vCPU oversubscription for a `V` vCPU should be implemented such, that we can guarantee _at least 20% of a core in >99% of the time_; this can be achieved by @@ -123,18 +131,18 @@ if the provider wants to enable hyperthreading on compute hosts despite having C (and no SCS-accepted core-scheduling mechanism is used for mitigation), the flavors must be declared insecure with the `i` suffix (see below). -#### Higher oversubscription +##### Higher oversubscription Must be indicated with a `L` vCPU type (low performance for > 5x/core or > 3x/thread oversubscription and the lack of workload management that would prevent worst case performance <20% in more than 7.2h per month). -#### Insufficient microcode +##### Insufficient microcode Not using these mitigations must be indicated by an additional `i` suffix for insecure -(weak protection against CPU vulns through insufficient microcode, lack of disabled hyperthreading +(weak protection against CPU vulnerabilities through insufficient microcode, lack of disabled hyperthreading on L1TF susceptible CPUs w/o effective core scheduling or disabled protections on the host/hypervisor). -#### Examples +##### Examples - SCS-**2C**-4-10n - SCS-**2T**-4-10n @@ -144,23 +152,23 @@ on L1TF susceptible CPUs w/o effective core scheduling or disabled protections o - ~~SCS-**2**-\*\*4-10n~~ <- CPU suffix missing - ~~SCS-**2iT**-4-10n~~ <- This order is forbidden -### [REQUIRED] Memory +#### [REQUIRED] Memory -#### Baseline +##### Baseline Cloud providers should use ECC memory. Memory oversubscription should not be used. -It is allowed to specify half GiBs (e.g. 3.5), though this is should not be done for larger memory sizes (>= 10GiB). +It is allowed to specify half GiBs (e.g. 3.5), though this should not be done for larger memory sizes (>= 10GiB). -#### No ECC +##### No ECC If no ECC is used, the `u` suffix must indicate this. -#### Enabled Oversubscription +##### Enabled Oversubscription If memory is oversubscribed, you must expose this with the `o` suffix. -#### Examples +##### Examples - SCS-2C-**4**-10n - SCS-2C-**3.5**-10n @@ -169,18 +177,18 @@ If memory is oversubscribed, you must expose this with the `o` suffix. - SCS-2C-**4uo**-10n - ~~SCS-2C-**4ou**-10n~~ <- This order is forbidden -### [OPTIONAL] Disk sizes and types +#### [OPTIONAL] Disk sizes and types Disk sizes (in GB) should use sizes 5, 10, 20, 50, 100, 200, 500, 1000. | Disk type | Meaning | -| --------- | ------------------------------------ | +|-----------|--------------------------------------| | n | Network shared storage (ceph/cinder) | | h | Local disk (HDD: SATA/SAS class) | | s | Local SSD disk | | p | Local high-perf NVMe | -#### Baseline +##### Baseline Note that disk type might be omitted — the user then can not take any assumptions on what storage is provided for the root disk (that the image gets provisioned to). @@ -202,13 +210,13 @@ tell the instance to boot from it or use the [`block_device_mapping_v2`](https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server) mechanism explicitly to create the boot volume from an image. -#### Multi-provisioned Disk +##### Multi-provisioned Disk The disk size can be prefixed with `Mx prefix`, where M is an integer specifying that the disk is provisioned M times. Multiple disks provided this way should be independent storage media, so users can expect some level of parallelism and independence. -#### Examples +##### Examples - SCS-2C-4-**10n** - SCS-2C-4-**10s** @@ -228,59 +236,7 @@ so users can expect some level of parallelism and independence. - SCS-2C-4-3x10 <- Cloud decides type and creates three 10GB volumes - ~~SCS-2C-4-**1.5n**~~ <- You must not specify disk sizes which are not in full GiBs -## Naming policy compliance - -Every flavor you offer MUST satisfy the following assertion: - -- If its name starts with `SCS-`, the name has to conform to the syntax outlined above, and - the flavor must _at least_ provide the capabilities indicated by the name. - -That is to say: - -- You may offer flavors not following the above scheme, as long as the name does not - start with `SCS-`. - -- You are allowed to understate your performance; for instance, a flavor that satisfies - `SCS-1C-1.5-8s` (1 dedicated core, 1.5 GiB RAM, 8 GiB SSD) may also be named - `SCS-1T-1-5n` (1 dedicated hyperthread, 1 GiB RAM, 5 GiB network volume) or even - `SCS-1V-1-5`. Similarly, you may offer the (v3 mandatory) `SCS-2V-4-20s` with a `SCS-2V-4-20p` - implementation (using a local NVMe instead of an SSD). - -You must be very careful to expose low vCPU guarantees (`L` instead of `V`), insecure -hyperthreading/microcode `i`, non-ECC-RAM `u`, memory oversubscription `o`. Note that omitting these qualifiers -is _overstating_ your security, reliability or performance properties and may be reason for -clients to feel betrayed or claim damages. This would prevent SCS compliance and certification; -in extreme cases, the SCS project might be forced to work with public statements. - -We expect all cloud providers to offer the short, less specific flavor names (such as `SCS-8V-32-100`). -Larger providers that offer more details (using the extension below) are expected to still also -offer the short variants for usability and easier portability, even beyond the mandated flavors. - -You must not extend the SCS naming scheme with your own extensions; you are encouraged however -to suggest extensions that we can discuss and add to the official scheme. - -## Conformance Tests - -There is a script in [`flavor-name-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/flavor-naming/flavor-name-check.py) -which can be used to decode, validate and construct flavor names. -[`flavor-name-describe.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/flavor-naming/flavor-name-describe.py) outputs a human-readable decoding of the SCS flavor names. -These scripts must stay in sync with the specification text. - -Ensure you have your OpenStack tooling (`python3-openstackclient`, `OS_CLOUD`) setup and call -`tools/flavor-name-check.py -c $(openstack flavor list -f value -c Name)` to get a report -on the flavor list compliance of the cloud environment. - -The script `flavor-names-openstack.py` talks to the OpenStack API of the -cloud specified by the `OS_CLOUD` environment and queries properties and checks -the names for standards compliance. -It goes beyond the above example in checking that the discoverable -features of flavors (vCPUs, RAM, Disk) match what the flavor names claim. -This is used for SCS-compatible compliance testing. - -The functionality of the `flavor-name-check.py` script is also -(partially) exposed via the web page . - -## Extensions +### Extensions Extensions provide a possibility for providers that offer a very differentiated set of flavors to indicate hypervisors, support for hardware/nested virtualization, @@ -319,7 +275,7 @@ Remember that letters are case-sensitive. In case you wonder: Feature indicators are capitalized, modifiers are lower case. (An exception is the uppercase `_G` for a pass-through GPU vs. lowercase `_g` for vGPU.) -### [OPTIONAL] Hypervisor +#### [OPTIONAL] Hypervisor Format: `_`hyp @@ -327,20 +283,20 @@ The _default Hypervisor_ is assumed to be `KVM`. Clouds that offer different hyp or Bare Metal Systems should indicate the Hypervisor according to the following table: | hyp | Meaning | -| --- | ----------------- | +|-----|-------------------| | kvm | KVM | | xen | Xen | | vmw | VMware | | hyv | Hyper-V | | bms | Bare Metal System | -#### Examples +##### Examples - SCS-2C-4-10n - SCS-2C-4-10n\_**bms** - SCS-2C-4-10n\_**bms**\_z3h -### [OPTIONAL] Hardware virtualization / Nested virtualization +#### [OPTIONAL] Hardware virtualization / Nested virtualization Format: `_hwv` @@ -354,7 +310,7 @@ Flavors without the `_hwv` flag may or may not support hardware virtualization ( recommend enabling nesting, but don't require flavor names to reflect all capabilities. Flavors may over-deliver ...) -#### Examples +##### Examples - SCS-2C-4-10 <- may or may not support HW virtualization in VMs - SCS-2C-4-10_kvm_**hwv** <- kvm with enabled nested virtualization @@ -362,7 +318,7 @@ capabilities. Flavors may over-deliver ...) - SCS-2C-4-10\_bms\_**hwv** <- better: bare metal with HW virt support (VMX on intel, SVM on AMD, ...) - ~~SCS-2C-4-10\_**hwv**\_xen~~ <- illegal, wrong ordering -### [OPTIONAL] CPU Architecture Details +#### [OPTIONAL] CPU Architecture Details Format: `_`arch\[N\]\[`h`\] @@ -372,23 +328,23 @@ This extension provides more details on the specific CPU: - generation (N) - frequency (h) -#### Generation and Vendor +##### Generation and Vendor The options for arch are as follows: -| Letter | vendor/architecture | Corresponding image architecture | -| ------- | -------------------- | --------------------------------- | -| (none) | Generic x86-64 | `x86_64` | -| `i` | Intel x86-64 | `x86_64` | -| `z` | AMD (Zen) x86-64 | `x86_64` | -| `a` | ARM v8+ | `aarch64` | -| `r` | RISC-V | (not yet listed in Glance) | +| Letter | vendor/architecture | Corresponding image architecture | +|--------|---------------------|----------------------------------| +| (none) | Generic x86-64 | `x86_64` | +| `i` | Intel x86-64 | `x86_64` | +| `z` | AMD (Zen) x86-64 | `x86_64` | +| `a` | ARM v8+ | `aarch64` | +| `r` | RISC-V | (not yet listed in Glance) | The generation is vendor specific and can be left out, but it can only be specified in conjunction with a vendor. At present, these values are possible: | Generation | i (Intel x86-64) | z (AMD x86-64) |  a (AArch64) | r (RISC-V) | -| ---------- | ---------------- | -------------- | ------------------ | ---------- | +|------------|------------------|----------------|--------------------|------------| | 0 | pre Skylake | pre Zen | pre Cortex A76 | TBD | | 1 | Skylake | Zen-1 (Naples) | A76/NeoN1 class | TBD | | 2 | Cascade Lake | Zen-2 (Rome) | A78/x1/NeoV1 class | TBD | @@ -400,25 +356,22 @@ help the parser tool, which assumes 0 for an unspecified value and does leave it out when generating the name for comparison. In other words: 0 has a meaning of "rather old or unspecified". -:::note +> _Note_: +> We don't differentiate between Zen-4 (Genoa) and Zen-4c (Bergamo); L3 cache per +> Siena core is smaller on Bergamo and the frequency lower but the cores are otherwise +> identical. As we already have a qualifier `h` that allows to specify higher frequencies +> (which Genoa thus may use more and Bergamo less or not), we have enough distinction +> capabilities. -We don't differentiate between Zen-4 (Genoa) and Zen-4c (Bergamo); L3 cache per -Siena core is smaller on Bergamo and the frequency lower but the cores are otherwise -identical. As we already have a qualifier `h` that allows to specify higher frequencies -(which Genoa thus may use more and Bergamo less or not), we have enough distinction -capabilities. - -::: - -#### Frequency Suffixes +##### Frequency Suffixes | Suffix | Meaning | -| ------ | ----------------- | +|--------|-------------------| | h | >2.75GHz all-core | | hh | >3.25GHz all-core | | hhh | >3.75GHz all-core | -#### Examples +##### Examples - SCS-2C-4-10n - SCS-2C-4-10n\_**z** @@ -431,7 +384,7 @@ capabilities. - SCS-2C-4-10n\_bms\_**z3h** - SCS-2C-4-10n\_bms\_**z3hh** <- Bare Metal, AMD Milan with > 3.25GHz all core freq -### [OPTIONAL] GPU support +#### [OPTIONAL] GPU support Format: `_`\[`G/g`\]X\[N\]\[`-`M\]\[`h`\] @@ -446,7 +399,7 @@ This extension provides more details on the specific GPU: Note that the vendor letter X is mandatory, generation and processing units are optional. | letter X | vendor | processing units | -| -------- | ------ | ------------------------------- | +|----------|--------|---------------------------------| | `N` | nVidia | streaming multiprocessors (SMs) | | `A` | AMD | compute units (CUs) | | `I` | Intel | execution units (EUs) | @@ -454,17 +407,16 @@ Note that the vendor letter X is mandatory, generation and processing units are For nVidia, the generation N can be f=Fermi, k=Kepler, m=Maxwell, p=Pascal, v=Volta, t=turing, a=Ampere, l=Ada Lovelace, ..., for AMD GCN-x=0.x, RDNA1=1, RDNA2=2, RDNA3=3, for Intel Gen9=0.9, Xe(12.1)=1, ... -(Note: This may need further work to properly reflect what's out there.) +> _Note_: This may need further work to properly reflect what's out there.) The optional `h` suffix to the compute unit count indicates high-performance (e.g. high freq or special -high bandwidth gfx memory such as HBM); -`h` can be duplicated for even higher performance. +high bandwidth gfx memory such as HBM); `h` can be duplicated for even higher performance. Example: `SCS-16V-64-500s_GNa-14h` This flavor has a pass-through GPU nVidia Ampere with 14 SMs and either high-bandwidth memory or specially high frequencies. Looking through GPU specs you could guess it's 1/4 of an A30. -### [OPTIONAL] Infiniband +#### [OPTIONAL] Infiniband Format: `_ib` @@ -474,7 +426,7 @@ More extensions may be forthcoming and appended in a later revision of this spec Extensions need to be specified in the above-mentioned order. -### Naming options advice +#### Naming options advice Note that we expect most clouds to prefer short flavor names, not indicating CPU details or hypervisor types. See above list @@ -504,10 +456,10 @@ may not be needed much. However, there must be a way to request flavor properties without encoding the need into an image — the indirection via an image is considered broken by the SCS team. -## Proposal Examples +### Proposal Examples | Example | Decoding | -| ------------------------- | ---------------------------------------------------------------------------------------------- | +|---------------------------|------------------------------------------------------------------------------------------------| | SCS-2C-4-10n | 2 dedicated cores (x86-64), 4GiB RAM, 10GB network disk | | SCS-8Ti-32-50p_i1 | 8 dedicated hyperthreads (insecure), Skylake, 32GiB RAM, 50GB local NVMe | | SCS-1L-1u-5 | 1 vCPU (heavily oversubscribed), 1GiB Ram (no ECC), 5GB disk (unspecific) | @@ -515,6 +467,58 @@ an image is considered broken by the SCS team. | SCS-4C-16-2x200p_a1 | 4 dedicated Arm64 cores (A76 class), 16GiB RAM, 2x200GB local NVMe drives | | SCS-1V-0.5 | 1 vCPU, 0.5GiB RAM, no disk (boot from cinder volume) | +## Standard + +Every flavor you offer MUST satisfy the following assertion: + +- If its name starts with `SCS-`, the name has to conform to the syntax outlined above, and + the flavor must _at least_ provide the capabilities indicated by the name. + +That is to say: + +- You MAY offer flavors not following the above scheme, as long as the name does not + start with `SCS-`. + +- You MAY understate your performance; for instance, a flavor that satisfies + `SCS-1C-1.5-8s` (1 dedicated core, 1.5 GiB RAM, 8 GiB SSD) may also be named + `SCS-1T-1-5n` (1 dedicated hyperthread, 1 GiB RAM, 5 GiB network volume) or even + `SCS-1V-1-5`. Similarly, you may offer the (v3 mandatory) `SCS-2V-4-20s` with a `SCS-2V-4-20p` + implementation (using a local NVMe instead of an SSD). + +You must be very careful to expose low vCPU guarantees (`L` instead of `V`), insecure +hyperthreading/microcode `i`, non-ECC-RAM `u`, memory oversubscription `o`. Note that omitting these qualifiers +is _overstating_ your security, reliability or performance properties and may be reason for +clients to feel betrayed or claim damages. This would prevent SCS compliance and certification; +in extreme cases, the SCS project might be forced to work with public statements. + +We expect all cloud providers to offer the short, less specific flavor names (such as `SCS-8V-32-100`). +Larger providers that offer more details (using the extension below) are expected to still also +offer the short variants for usability and easier portability, even beyond the mandated flavors. + +You must not extend the SCS naming scheme with your own extensions; you are encouraged however +to suggest extensions that we can discuss and add to the official scheme. + +## Conformance Tests + +There is a script in [`flavor-name-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/flavor-naming/flavor-name-check.py) +which can be used to decode, validate and construct flavor names. +[`flavor-name-describe.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/flavor-naming/flavor-name-describe.py) outputs a human-readable decoding of the SCS flavor names. +These scripts must stay in sync with the specification text. + +Ensure you have your OpenStack tooling (`python3-openstackclient`, `OS_CLOUD`) setup and call +`tools/flavor-name-check.py -c $(openstack flavor list -f value -c Name)` to get a report +on the flavor list compliance of the cloud environment. + +The script `flavor-names-openstack.py` talks to the OpenStack API of the +cloud specified by the `OS_CLOUD` environment and queries properties and checks +the names for standards compliance. +It goes beyond the above example in checking that the discoverable +features of flavors (vCPUs, RAM, Disk) match what the flavor names claim. +This is used for SCS-compatible compliance testing. + +The functionality of the `flavor-name-check.py` script is also +(partially) exposed via the web page . + ## Previous standard versions Previous versions up to version 3.0 contained the list of @@ -562,8 +566,7 @@ However, we have been reaching out to the OpenStack Public Cloud SIG and the ALA members to seek further alignment. Getting upstream OpenStack support for flavor aliases would provide more flexibility -and ease migrations between providers, also providers that don't offer the SCS- -flavors. +and ease migrations between providers, also providers that don't offer the SCS-flavors. We also would like to see upstream `extra_specs` standardizing the discoverability of some properties exposed via the SCS names and work on IaC tooling (terraform ...) diff --git a/Standards/scs-0111-v1-volume-type-decisions.md b/Standards/scs-0111-v1-volume-type-decisions.md index 9b4ebf08c..dc153cb80 100644 --- a/Standards/scs-0111-v1-volume-type-decisions.md +++ b/Standards/scs-0111-v1-volume-type-decisions.md @@ -9,27 +9,27 @@ track: IaaS Volumes in OpenStack are virtual drives. They are managed by the storage service Cinder, which abstracts creation and usage of many different storage backends. While it is possible to use a backend like lvm which can reside on the same host as the hypervisor, the SCS wants to make a more clear differentiation between volumes and the ephemeral storage of a virtual machine. For all SCS deployments we want to assume that volumes are always residing in a storage backend that is NOT on the same host as a hypervisor - in short terms: Volumes are network storage. Ephemeral storage on the other hand is the only storage residing on a compute host. It is created by creating a VM directly from an Image and is automatically los as soon as the VM cease to exist. Volumes on the other hand have to be created from Images and only after that can be used for VMs. They are persistent and will remain in the last state a VM has written on them before they cease to exit. Being persistent and not relying on the host where the VM resides, Volumes can easily be attached to another VM in case of a node outage and VMs be migrated way more easily, because only metadata and data in RAM has to be shifted to another host, accelerating any migration or evacuation of a VM. -Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes very be backend-specific and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. Nevertheless the most of configuration is done in the backends themself, so volume types only work as a rough classification. +Volume Types are used to classify volumes and provide a basic decision for what kind of volume should be created. These volume types can sometimes very be backend-specific, and it might be hard for a user to choose the most suitable volume type, if there is more than one default type. Nevertheless, the most configuration is done in the backends themselves, so volume types only work as a rough classification. ## Motivation -We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of volume. Encryption of volumes for example is solely decided by the volume type. And whether the volume will be replicated is a mix between definiton in the volume type and backend specific configuration, but it's visiblity can only be reached in the volume type. +We want to standardize a few varieties of volume types. While a user can choose simple things like size when creating a volume, Volume Types define a few broader aspects of volume. Encryption of volumes for example is solely decided by the volume type. And whether the volume will be replicated is a mix between definition in the volume type and backend specific configuration, but it's visibility can only be reached in the volume type. -In General: what the different volume types are capable of is highly dependend on both the used backend and the configurations of OpenStack. A few options are worth being at least recommended. +In General: what the different volume types are capable of is highly dependent on both the used backend and the configurations of OpenStack. A few options are worth being at least recommended. -## Design Considerations +## Context We want to have a discoverable Standard. So there should be no naming conventions as per request by operators. -This first decision will have impacts on upstream OpenStack development, as those things, that would be nice to discover, may not be currently dicoverable by users or not at all. +This first decision will have impacts on upstream OpenStack development, as those things, that would be nice to discover, may not be currently discoverable by users or not at all. -There are severel aspects of volume types, which will be discussed in the following: +There are several aspects of volume types, which will be discussed in the following: ### Options considered #### Encryption -Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. As an admin it is also currently possible to see these configurations in a volume type with list and show commands. A user should not see these parameters in detail, but a boolean value that descibes whether encryption is used or not. Currently this is not possible in upstream OpenStack. +Encryption for volumes is an option which has to be configured within the volume type. As an admin it is possible to set encryption-provider, key size, cipher and control location. As an admin it is also currently possible to see these configurations in a volume type with list and show commands. A user should not see these parameters in detail, but a boolean value that describes whether encryption is used or not. Currently, this is not possible in upstream OpenStack. **Conclusion**: This is a solid aspect to be standardized. But it will need work on OpenStack, to have a boolean value presented to the users. @@ -41,7 +41,7 @@ OpenStack Cinder works with a lot of different backends. They all have some kind #### Availability Zones -Availability Zones are used in Nova and Cinder seperatly to provide an often also physical separation of compute hosts or storage nodes. This leads to two options to consider: +Availability Zones are used in Nova and Cinder separately to provide an often also physical separation of compute hosts or storage nodes. This leads to two options to consider: 1. Multiple Volume AZs: This might be used if there are different backends present in one IaaS structure. The different volume types are usually used for the different volume AZs. This makes migration between those AZs only be possible for administrators. @@ -49,24 +49,24 @@ Availability Zones are used in Nova and Cinder seperatly to provide an often als Another question is how many providers use one of these options or both. -**Conclusion**: The first part doesn't make much sense to standardize, as migration between the volume types can only be done by admins. However the second part might be noteable, but due to the variety of configuration options very hard to standardize. +**Conclusion**: The first part doesn't make much sense to standardize, as migration between the volume types can only be done by admins. However, the second part might be noteable, but due to the variety of configuration options very hard to standardize. -#### Multiattach +#### Multi-attach -It is possible in a few backends to attach a volume to multiple VMs. This has to be configured in the Volume Type and this information is also accessable for users. Nevertheless this option also needs a lot of work from users, as those types of volumes have to have a file system, that is capable of multiattach. Many providers don't provide multiattach. +It is possible in a few backends to attach a volume to multiple VMs. This has to be configured in the Volume Type and this information is also accessible for users. Nevertheless, this option also needs a lot of work from users, as those types of volumes have to have a file system, that is capable of multi-attach. Many providers don't provide multi-attach. **Conclusion**: It might be noteable, that this already is a discoverable option. #### Replication -Replication states, whether or not there are multiple replicas of a volume. Thus answers the question, whether the data could survive a node outage. Again there are different ways to achive replicated volumes. It can either be defined in the volume type and is discoverable also by normal users or it is configured in the backend. The last option is usually used with ceph for example. This makes it hard to discover, whether a volume is replicated or not. Another point is the number of replicas, that exist. +Replication states, whether there are multiple replicas of a volume. Thus answers the question, whether the data could survive a node outage. Again there are different ways to achieve replicated volumes. It can either be defined in the volume type and is discoverable also by normal users, or it is configured in the backend. The last option is usually used with ceph for example. This makes it hard to discover, whether a volume is replicated or not. Another point is the number of replicas, that exist. -**Conclusion**: Replication is a good option to be standardized. Whether this should be done as a boolean option or if the number of replicas is also something users need to know should still be discussed. Nevertheless due to the different options to configure replication this will be quite complex. +**Conclusion**: Replication is a good option to be standardized. Whether this should be done as a boolean option or if the number of replicas is also something users need to know should still be discussed. Nevertheless, due to the different options to configure replication this will be quite complex. #### QoS -Quality of Service parameters can be stated in a volume qos object. These objects can then be associated to a volume type (or directly to a volume as an admin only option). But this is optional and thus even good or very good volume QoS parameters that are aquired through hardware configuration and storage parameters, might go by unmentioned. -Furthermore the indirection makes it harder to discover the qos for a volume type. Only admins will see the associated qos ID and will have to take a closer look at the qos after discovering the volume type. PLUS: there can only be one qos association for one volume type. But a qos can be used for multiple volumes. +Quality of Service parameters can be stated in a volume qos object. These objects can then be associated to a volume type (or directly to a volume as an admin only option). But this is optional and thus even good or very good volume QoS parameters that are acquired through hardware configuration and storage parameters, might go by unmentioned. +Furthermore, the indirection makes it harder to discover the qos for a volume type. Only admins will see the associated qos ID and will have to take a closer look at the qos after discovering the volume type. PLUS: there can only be one qos association for one volume type. But a qos can be used for multiple volumes. **Conclusion**: The benefit of displaying qos parameters is clear, thus this option should be noted. But are volume qos objects widely used? If not, standardization process would be too much work. @@ -81,15 +81,15 @@ While every option above described things, that can at least be partly or for ad ## Decision -| Aspect | Standardize? | Discoverability | other Things | -| ---- | ---- | ---- | ------ | -| encryption | **Recommended** | work needed | extra_spec: encrypted=True/False | -| Backend name | - | - | - | -| AZs | - | - | describe as optional and backend-dependend | -| multiattach | - | yes | describe as optional | -| Replication | **Recommended** | lot of work | either get from backend to OS or as extra_spec defined by deployer | -| Number of Replicas, etc | ? | lot of work | optional, work on it after Replication is standardized | -| Volume QoS | ? | admin only | needs further discussion, should be at least described as optional | +| Aspect | Standardize? | Discoverability | other Things | +|-------------------------|-----------------|-----------------|--------------------------------------------------------------------| +| encryption | **Recommended** | work needed | extra_spec: encrypted=True/False | +| Backend name | - | - | - | +| AZs | - | - | describe as optional and backend-dependent | +| Multi-attach | - | yes | describe as optional | +| Replication | **Recommended** | lot of work | either get from backend to OS or as extra_spec defined by deployer | +| Number of Replicas, etc | ? | lot of work | optional, work on it after Replication is standardized | +| Volume QoS | ? | admin only | needs further discussion, should be at least described as optional | ## Related Documents diff --git a/Standards/scs-0210-v2-k8s-version-policy.md b/Standards/scs-0210-v2-k8s-version-policy.md index 710c1c0e3..da113b9ed 100644 --- a/Standards/scs-0210-v2-k8s-version-policy.md +++ b/Standards/scs-0210-v2-k8s-version-policy.md @@ -23,13 +23,21 @@ More information can be found under [Kubernetes Support Period]. The Kubernetes release cycle is set around 4 months, which usually results in about **3 minor** releases per year [Kubernetes Release Cycle](https://kubernetes.io/releases/release/#the-release-cycle). -Patches to these releases are provided monthly, with the exception of the first patch, +Patches to these releases are provided monthly, except the first patch, which is usually provided 1-2 weeks after the initial release [Patch Release Cadence](https://kubernetes.io/releases/patch-releases/#cadence). +## Terminology + +Common Vulnerabilities and Exposures (abbr. CVE) + System to provide a references for publicly known information-security vulnerabilities and exposures + +Cloud Native Computing Foundation (abbr. CNCF) + Linux Foundation project that was founded to help advance container technology + ## Motivation Kubernetes is a living, fast-paced project, which follows a pre-defined release cycle. -This enables forward planning with regards to releases and patches, but also implies a +This enables forward planning in regard to releases and patches, but also implies a necessity to upgrade to newer versions quickly, since these often include new features, important security updates or especially if a previous version falls out of the support period window. @@ -38,7 +46,7 @@ We want to achieve an up-to-date policy, meaning that providers should be mostly sync with the upstream and don't fall behind the official Kubernetes releases. This is achievable, since new versions are released periodical on a well communicated schedule, enabling providers and users to set up processes around it. -Being up to date ensures that security issues and bugs are addressed and new features +Being up-to-date ensures that security issues and bugs are addressed and new features are made available when using SCS compliant clusters. It is nevertheless important to at least support all Kubernetes versions that are still @@ -48,7 +56,7 @@ longer to upgrade their workloads to a newer version. The standard therefore should provide a version recency policy as well as a support window period. -## Decision +## Standard In order to keep up-to date with the latest Kubernetes features, bug fixes and security improvements, the provided Kubernetes versions should be kept up to date with the upstream. @@ -56,7 +64,7 @@ the provided Kubernetes versions should be kept up to date with the upstream. - The latest minor version MUST be provided no later than 4 months after release. - The latest patch version MUST be provided no later than 1 week after release. - This time period MUST be even shorter for patches that target critical CVEs (CVSS >= 8). - It is RECOMMENDED to provide a new patch version in a 2 day time period after their release. + It is RECOMMENDED to provide a new patch version in a 2-day time period after their release. - New versions MUST be tested before being rolled out on productive infrastructure; at least the CNCF E2E tests should be passed beforehand. diff --git a/Standards/scs-0211-v1-kaas-default-storage-class.md b/Standards/scs-0211-v1-kaas-default-storage-class.md index 7e224a9e3..ea755813a 100644 --- a/Standards/scs-0211-v1-kaas-default-storage-class.md +++ b/Standards/scs-0211-v1-kaas-default-storage-class.md @@ -14,7 +14,8 @@ description: | ## Introduction Cluster consumers can request persistent storage via [`PersistentVolumeClaims`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) which is provisioned automatically by cloud-provided automation. -Storage requirements may vary across use cases, so there is the concept of `StorageClasses`. `StorageClasses` define some set of storage properties. So, consumers can choose one of these depending on the use case. +Storage requirements may vary across use cases, so there is the concept of `StorageClasses`. `StorageClasses` define some set of storage properties. +So, consumers can choose one of these depending on the use case. [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) @@ -24,27 +25,27 @@ While often times, consumers will choose a `StorageClass` explicitly, usually, t This document attempts to define the properties this default `StorageClass` should have. -## Decision +## Standard The default `StorageClass` is made default using the `storageclass.kubernetes.io/is-default-class` annotation, following [Kubernetes upstream](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/). Hence, standardizing its name is not required for the intents of this standard. ### Required non-performance-related properties -- `ReadWriteOnce` must be a supported [access mode](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) -- volume must be protected against data loss due to hardware failures of a single disk or host -- volume must not be bound to the lifecycle of a Kubernetes Node +- `ReadWriteOnce` MUST be a supported [access mode](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) +- volume MUST be protected against data loss due to hardware failures of a single disk or host +- volume MUST not be bound to the lifecycle of a Kubernetes node Hence, -- ...volume must not be backed by local storage on the Kubernetes Node VM itself -- ...volume may be backed by some kind of redundant storage within an AZ, across hosts -- ...volume may be backed by some kind of redundant storage across AZ's +- ...volume MUST not be backed by local storage on the Kubernetes node VM itself +- ...volume MAY be backed by some kind of redundant storage within an AZ, across hosts +- ...volume MAY be backed by some kind of redundant storage across AZ's ### Required performance-related properties - _NO_ fixed guarantees regarding latency/bandwidth/IOPS/... -Generally, customers should be able to expect low-tier performance without pricing surprises. +Generally, customers SHOULD be able to expect low-tier performance without pricing surprises. ## Related Documents diff --git a/Standards/scs-0302-v1-domain-manager-role.md b/Standards/scs-0302-v1-domain-manager-role.md index 9a7eb8b7e..2edaece3d 100644 --- a/Standards/scs-0302-v1-domain-manager-role.md +++ b/Standards/scs-0302-v1-domain-manager-role.md @@ -15,18 +15,18 @@ To avoid conflict with the unscoped `admin` role in OpenStack we want to refer t The following special terms are used throughout this standard document: -| Term | Meaning | -|---|---| -| RBAC | Role-Based Access Control[^1] established by OpenStack Keystone | -| project | OpenStack project as per Keystone RBAC | -| user | OpenStack user as per Keystone RBAC | -| group | OpenStack group as per Keystone RBAC | -| role | OpenStack role as per Keystone RBAC | -| domain | OpenStack domain as per Keystone RBAC | -| IAM | identity and access management | +| Term | Meaning | +|---------------|--------------------------------------------------------------------------| +| RBAC | Role-Based Access Control[^1] established by OpenStack Keystone | +| project | OpenStack project as per Keystone RBAC | +| user | OpenStack user as per Keystone RBAC | +| group | OpenStack group as per Keystone RBAC | +| role | OpenStack role as per Keystone RBAC | +| domain | OpenStack domain as per Keystone RBAC | +| IAM | identity and access management | | IAM resources | projects, users, groups, roles, domains as managed by OpenStack Keystone | -| CSP | Cloud Service Provider, provider managing the OpenStack infrastructure | -| cloud admin | OpenStack user belonging to the CSP that possesses the `admin` role | +| CSP | Cloud Service Provider, provider managing the OpenStack infrastructure | +| cloud admin | OpenStack user belonging to the CSP that possesses the `admin` role | [^1]: [OpenStack Documentation: Role-Based Access Control Overview](https://static.opendev.org/docs/patrole/latest/rbac-overview.html) @@ -44,7 +44,7 @@ Omitting the provisioning of any Domain Manager users (i.e. not assigning the ne In the default configuration of Keystone, only users with the `admin` role may manage the IAM resources such as projects, groups and users and their relation through role assignments. The `admin` role in OpenStack Keystone is not properly scoped when assigned within a domain or project only as due to hard-coded architectural limitations in OpenStack, a user with the `admin` role may escalate their privileges outside of their assigned project or domain boundaries. -Thus, it is not possible to properly give customers a self-service functionality in regards to project, group and user management with the default configuration. +Thus, it is not possible to properly give customers a self-service functionality in regard to project, group and user management with the default configuration. To address this, this standard defines a new Domain Manager role in conjunction with appropriate Keystone API policy adjustments to establish a standardized extension to the default Keystone configuration allowing for IAM self-service capabilities for customers within domains. @@ -55,7 +55,7 @@ To address this, this standard defines a new Domain Manager role in conjunction 3. The cloud admin creates one or more users within each of the applicable domains and assigns the Domain Manager role to them. These users represent the Domain Managers of the corresponding domain. 4. The customer uses the Domain Manager users to manage (create, update, delete) users, projects, groups and corresponding role assignments within their domain. -## Design Considerations +## Context - the Domain Manager role MUST support managing projects, groups and users within a specific domain - the Domain Manager role MUST be properly scoped to a domain, it MUST NOT gain access to resources outside of its owning domain @@ -93,9 +93,9 @@ This means that by creating a new role and extending Keystone's API policy confi [^4]: [OpenStack Documentation: Administering Applications that use oslo.policy](https://docs.openstack.org/oslo.policy/latest/admin/index.html) -## Open questions +### Open questions -### Limitations +#### Limitations The approach described in this standard imposes the following limitations: @@ -250,7 +250,7 @@ Decision: - the standard should not strictly limit the roles a Domain Manager can assign/revoke to/from other users within a domain to the "member" role - the standard should allow CSPs to define one or more roles for Domain Managers to manage -- whether or not this includes the Domain Manager role itself is not to be predefined by the standard and should be up to the CSP to decide instead +- whether this includes the Domain Manager role itself is not to be predefined by the standard and should be up to the CSP to decide instead - the standard should only strictly prohibit adding the "admin" role to the list of roles manageable by Domain Managers Rationale: