Streamline syntax of flavor-name tooling (#455)

Streamline syntax of flavor-name tooling

Following issues are addressed:

-  provide a cli tool with a streamlined and stable syntax
-  get rid of the importlib hack
-  get rid of excessive globals
-  fix a few bugs (and probably introduce new ones)
-  refactor to make more Pythonic (and hopefully more accessible)

Thx @martinmo for all the helpful remarks in the review!

---

Signed-off-by: Matthias Büchse <matthias.buechse@cloudandheat.com>

Tests/iaas/flavor-naming/README.md

@@ -0,0 +1,228 @@
+# SCS flavor name tooling
+
+This directory contains three basic tools:
+
+- `cli.py`, a command-line interface for syntax-checking flavor names, turning them into prose descriptions
+  or yaml documents, and interactively constructing flavor names;
+- `flavor-form.py`, a CGI script for a web form that offers roughly the same features,
+- `flavor-names-openstack.py`, a script that connects to an OpenStack environment and checks the syntax
+  of all publicly available flavors.
+
+Besides, it contains the scripts `flavor-name-check.py` and `flavor-name-describe.py`, which have been
+superseded by `cli.py`, and it contains `flavor-form-server.py`, which is a most simple webserver that can
+be used to test the `flavor-form.py` CGI script.
+
+## cli.py
+
+The command-line syntax of this script is best shown via the built-in help function:
+
+```console
+$ ./cli.py --help
+Usage: cli.py [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  -d, --debug
+  -v, --verbose
+  --help         Show this message and exit.
+
+Commands:
+  input  Interactively constructs a flavor name.
+  parse  Validates flavor names, optionally turns into prose/yaml.
+```
+
+### Parse
+
+```console
+ ./cli.py parse --help
+Usage: cli.py parse [OPTIONS] {v1|v1/v2|v2/v1|v2|v3|latest} [NAME]...
+
+  Validates flavor names, optionally turns into prose/yaml.
+
+  The first argument selects the version of the flavor naming standard upon
+  which to base the syntax validation. With 'v1/v2', flavor names of both
+  kinds are accepted, but warnings are emitted for v2, and similarly with
+  'v2/v1', where warnings are emitted for v1.
+
+Options:
+  -o, --output [none|prose|yaml]  select output format (default: none)
+  --help                          Show this message and exit.
+```
+
+Here is an example invocation:
+
+```console
+$ ./cli.py parse --output prose v1/v2 SCS-16T-64-3x10s_bms_hwv_i3h_GNa-64_ib 
+WARNING: Name is merely tolerated v2: SCS-16T-64-3x10s_bms_hwv_i3h_GNa-64_ib
+SCS flavor with 16 High Perf Intel Ice Lake SMT Threads with 64.0 GiB RAM on Bare Metal System with HW virt and SSD 3x10GB root volume and Pass-Through GPU nVidia Ampere (w/ 64 CU/EU/SM) and Infiniband
+```
+
+And one example with a wrong name:
+
+```console
+$ ./cli.py parse --output prose v3 SCS-2T-4_ic
+Extra characters: _ic: SCS-2T-4_ic
+```
+
+Noteworthy: if `--verbose` is used, the output format `none` will produce an output for every name (even
+those that are valid):
+
+```console
+$ ./cli.py --verbose parse v3 SCS-16T-64 SC-2T-4 SCS-2-4
+OK: SCS-16T-64
+NOT an SCS flavor: SC-2T-4
+Failed to parse main part: SCS-2-4
+```
+
+### Input
+
+```console
+$ ./cli.py input --help
+Usage: cli.py input [OPTIONS]
+
+  Interactively constructs a flavor name.
+
+Options:
+  --help  Show this message and exit.
+```
+
+Example invocation:
+
+```console
+$ ./cli.py input
+CPU-RAM
+ #vCPUs: 16
+ CPU type Options:
+  L: LowPerf vCPUs
+  V: vCPUs
+  T: SMT Threads
+  C: Dedicated Cores
+ CPU type: T
+ ?Insec SMT: 
+ ##GiB RAM: 64
+ ?no ECC: 
+ ?RAM Over: 
+Disk
+ #.NrDisks: 3
+ #.GB Disk: 10
+ Disk type Options:
+  : (unspecified)
+  n: Networked
+  h: Local HDD
+  s: SSD
+  p: HiPerf NVMe
+ Disk type: s
+Hypervisor
+ Hypervisor Options:
+  kvm: KVM
+  xen: Xen
+  hyv: Hyper-V
+  vmw: VMware
+  bms: Bare Metal System
+ Hypervisor: bms
+Hardware/NestedVirtualization
+ ?HardwareVirt: y
+CPUBrand
+ CPU Vendor Options:
+  i: Intel
+  z: AMD
+  a: ARM
+  r: RISC-V
+ CPU Vendor: i
+ #.CPU Gen Options:
+  : (unspecified)
+  0: Unspec/Pre-Skylake
+  1: Skylake
+  2: Cascade Lake
+  3: Ice Lake
+  4: Sapphire Rapids
+ #.CPU Gen: 3
+ Performance Options:
+  : Std Perf
+  h: High Perf
+  hh: Very High Perf
+  hhh: Very Very High Perf
+ Performance: h
+GPU
+ Type Options:
+  g: vGPU
+  G: Pass-Through GPU
+ Type: G
+ Brand Options:
+  N: nVidia
+  A: AMD
+  I: Intel
+ Brand: N
+ Gen Options:
+  : (unspecified)
+  f: Fermi
+  k: Kepler
+  m: Maxwell
+  p: Pascal
+  v: Volta
+  t: Turing
+  a: Ampere
+  l: AdaLovelace
+ Gen: a
+ #.CU/EU/SM: 64
+ Performance Options:
+  : Std Perf
+  h: High Perf
+  hh: Very High Perf
+  hhh: Very Very High Perf
+ Performance: 
+Infiniband
+ ?IB: y
+SCS-16T-64-3x10s_bms_i3h_GNa-64_ib
+```
+
+Where the final line is the constructed flavor name, and everything above is the interactive session.
+
+## flavor-form.py
+
+This CGI script produces a small web form that can be used to construct (and deconstruct) flavor names. It
+can be tested using `flavor-form-server.py` like so:
+
+```console
+$ ./flavor-form-server.py 
+The form is accessible at http://0.0.0.0:8000/cgi-bin/flavor-form.py
+^C
+Keyboard interrupt received, exiting.
+```
+
+## flavor-names-openstack.py
+
+As with `cli.py`, we have a look at the built-in help:
+
+```console
+$ ./flavor-names-openstack.py -h
+Usage: flavor-names-openstack.py [options]
+Options: [-c/--os-cloud OS_CLOUD] sets cloud environment (default from OS_CLOUD env)
+ [-C/--mand mand.yaml] overrides the list of mandatory flavor names
+ [-1/--v1prefer] prefer v1 flavor names (but still tolerates v2
+ [-o/--accept-old-mandatory] prefer v2 flavor names, but v1 ones can fulfill mand list
+ [-2/--v2plus] only accepts v2 flavor names, old ones result in errors
+ [-3/--v3] differentiate b/w mand and recommended flavors
+ [-v/--verbose] [-q/--quiet] control verbosity of output
+This tool retrieves the list of flavors from the OpenStack cloud OS_CLOUD
+ and checks for the presence of the mandatory SCS flavors (read from mand.yaml)
+ and reports inconsistencies, errors etc. It returns 0 on success.
+```
+
+Example invocation:
+
+```console
+$ OS_CLOUD=wave1-compliance ./flavor-names-openstack.py
+wave1-compliance:
+  OtherFlavorSummary:
+    TotalAmount: 0
+  SCSFlavorSummary:
+    FlavorsWithWarnings: 0
+    MandatoryFlavorsMissing: 0
+    MandatoryFlavorsPresent: 26
+    OptionalFlavorsValid: 37
+    OptionalFlavorsWrong: 0
+    TotalAmount: 63
+  TotalSummary:
+    Errors: 0
+    Warnings: 0
+```

Tests/iaas/flavor-naming/check-yaml.py

@@ -8,21 +8,17 @@
 
 Also check that the flavor properties encoded in the YAML match those encoded in the name.
 """
-import importlib
 import os
 import os.path
 import sys
 
 import yaml
 
-flavor_name_check = importlib.import_module("flavor-name-check")
-flavor_name_check.disallow_old = True
+from flavor_names import parser_v2, flavorname_to_dict
 
 
 REQUIRED_FIELDS = ['name-v1', 'name-v2', 'name', 'cpus', 'ram', 'cpu-type']
 DEFAULTS = {'disk0-type': 'network'}
-CPUTYPE_KEY = {'L': 'crowded-core', 'V': 'shared-core', 'T': 'dedicated-thread', 'C': 'dedicated-core'}
-DISKTYPE_KEY = {'n': 'network', 'h': 'hdd', 's': 'ssd', 'p': 'nvme'}
 
 
 class Undefined:
@@ -52,24 +48,20 @@ def check_spec(self, flavor_spec):
             return
         name = flavor_spec['name']
         name_v2 = flavor_spec['name-v2']
-        parsed = flavor_name_check.parsename(name_v2)
-        if not parsed:
+        try:
+            flavorname = parser_v2(name_v2)
+        except Exception:
+            flavorname = None
+        if not flavorname:
             self.emit(f"flavor {name}: name-v2 '{name_v2}' could not be parsed")
             return
-        cpu, disk, hype, hvirt, cpubrand, gpu, ibd = parsed
         undefined = Undefined()
-        expected = {
-            'cpus': cpu.cpus,
-            'cpu-type': CPUTYPE_KEY[cpu.cputype],
-            'ram': cpu.ram,
-            'name-v1': flavor_name_check.new_to_old(name_v2),
-            'disk': undefined,
-        }
-        if disk.parsed:
-            if disk.nrdisks != 1:
-                self.emit(f"flavor '{name}': name-v2 using multiple disks")
-            expected['disk'] = disk.disksize
-            expected['disk0-type'] = DISKTYPE_KEY[disk.disktype or 'n']
+        expected = flavorname_to_dict(flavorname)
+        # add explicit undefined so the final for-loop catches spurious entries
+        for key in flavor_spec:
+            if key == 'name':
+                continue
+            expected.setdefault(key, undefined)
         for key, exp_val in expected.items():
             val = flavor_spec.get(key, DEFAULTS.get(key, undefined))
             if val != exp_val: