Some suggestions to AML

[scarlett2018]

As the title, design to be discussed, welcome for feedbacks.

[seanytak]

Hi, I'm currently trying to use NNI with AML and wanted to share some design thoughts based on my experience thus far.

Pain Points:

1. As far as I know, NNI only works by invoking the RESTful server using nnictl create --config <config-file>. When using AML as a training platform, unless the compute platform has additional setup to allow inbound and outbound traffic for the ports used by NNI (which doesn't appear to be available for configuration on compute platforms made through AML only on a regular Azure VM instance) I can't seem to access the WebUI dashboard. Thus, the only method I've found thus far is to obtain the results of the NNI experiment through hitting the API endpoint at /api/v1/nni/trial-jobs and parsing the finalMetricData.

Proposed Design Thoughts

1. NNI should be able to submit an AML experiment for each trial that is run so that the results and info about the trial can have a 1-1 mapping with the AML interface when it appears in the list of experiments ran.
2. NNI should submit trials as child runs of a parent experiment on AML's interface. Currently this is the way Hyperdrive works and it would be nice if NNI appeared to work as natively as possible here in a similar fashion.

As an additional thought, are there plans to allow developers to use the underlying tuning algorithms, NAS algorithms, etc. through a Python API instead of only through the command line tool? This could allow training platform developers to adopt NNI natively into their own platform instead of having NNI configured to work on a plethora of services which may incur technical debt as services evolve.