High-level design and roadmap for Honegumi

[sgbaird]

What tools to use, how to structure the repo, target scope, etc.

Primary threads:

• Interface
• Testing
• Backend
• Exposing as a PyScaffold extension
• Toggleable features

cc @jduerholt

[sgbaird]

Interface

• per-configuration output also yields an open in Colab badge
• Programmatic usage: selection of the number of parameters and outputs of certain types (i.e., instead of static display on docs, a dynamic generation via a notebook)
• guidelines/instructions for contributing new template generators for other platforms (BoFire, Olympus, Atlas, Chaos)
• when external data is involved, comments should include links to the corresponding data template (e.g., csv file)
• in addition to the runnable script, the output should be included. For example, error messages, warnings, other info
• Copy to clipboard

[sgbaird]

Testing

• pytorch-like unit tests and crossed out invalid configurations
• unit tests should not only check for lack of errors, but also functionality/performance. For example, X% better than a baseline. This requires performance metrics tailored to each configuration, where sometimes multiple metrics will be involved.
• Maybe these performance-based unit tests should be embedded into the notebooks themselves with statistics or visualizations to compare performance (i.e., really getting the teaching aspect in)
• template outputs a notebook-style .py format (e.g., nbformat, jupytext) that gets converted to a notebook, run with outputs, and saved
• will some notebooks need to be run as a script?
• if CUDA is selected, note instructions for switching to GPU
• pytest-xdist to run GitHub actions tests in parallel

[sgbaird]

Backend

• Jinja2 for template generation
• Abstracting the optimization template generation to make it modular and extensible
• Expanding to include other APIs or platforms
• asynchronous optimization is a tough one to implement
• Support xlsx and csv for existing data, featurizers, and non-automated experiments options
• Auto-create an environment.lock.yml file and push it to the repo (for reproducibility when APIs change)
• add tutorial notebook(s) for high-level package usage of honegumi and ensure that it can run to completion via GitHub actions
• Warnings and guidelines about dataset size limits could be included. Maybe fallback to simpler methods if the computational burden is too large
• Scripts should get converted to notebook formats. Where to put them? (docs, notebooks, etc.) and should it be a flat or nested organization?
• Consider applying black formatting and other nicety formatting options during script generation

[sgbaird]

Exposing as a PyScaffold extension

Perhaps as a separate package.

• put the notebooks into the notebooks folder, unit test(s) associated with that notebook into the tests folder, tables into the data folder (if applicable), make a copy of dependencies (loose and strict), etc.
• Should there be an option to add new tutorials to an existing repo? (I.e., separate optimization tasks/scripts)

[sgbaird]

Toggleable Features

Objective	Single	Multi

Model	GPEI	SAASBO