Cloud executor abstract template

[venkatBala]

The cloud executors largely follow a similar pattern where the function to be executed along with its args and kwargs are pickled and uploaded to a remote object storage. Following this an artifact is built locally (docker image or zip file) that is uploaded to a remote cloud registry or an object storage respectively. The deployed artifact is of the format that contains all the necessary dependencies for the function to be executed remotely. The output of the function execution is always a pickled result object that is to be retrieved, unpickled and returned by the executor. A remote object store is again used for this purpose.

It is apparent that the steps are largely the same across all cloud executors, and can be summarized as follows:

• Pickle function, args, kwargs
• Upload pickled function, args, kwargs to remote object store
• Create the python execution script
• Create deployment artifact (docker image/ zip file)
• Upload deployment artifact to remote (registry/object store)
• Use a cloud service to run the deployment artifact and get the pickled result object
• Retrieve the result pickle file from the remote object store, unpickle it locally and return the result object

The steps presented here are largely cloud agnostic and can be abstracted so that implementation of the cloud executors becomes more streamlined. There are several suggestions to this, which are as follows

• Cloud executor utilities: This aims to provide a custom toolkit that can be leveraged during cloud executor implementation. Examples of this can be a uniform way of retrieving cloud credentials necessary for the different cloud providers similar to boto3.Session
• CloudExecutor base class: A cloud executor specific base class can be created to cater to the cloud executor framework. This base class can be derived from the BaseExecutor class in order to maintain the executor inheritance hierarchy and can be tuned with abstract methods specific to the different steps laid out earlier.

Lastly, current the deployment artifacts are being built locally i.e on the covalent server. This entails that the software necessary to build the artifacts need to be present on the covalent server. A way around this requirement would to be offload artifact builds to the cloud provider itself via leveraging their native build services such as AWS CodeBuild, azure container build service. This has the added benefit
of being tightly integrated with the cloud provider as well as giving a performance boost that would be much needed when concerned
with large images.

@wjcunningham7 @santoshkumarradha @scottwn @FyzHsn @AlejandroEsquivel

[scottwn]

Now that we have the steps well defined, I think we can work on writing a cloud executor class that contains this logic. The code in the AWS executors is very similar. Most of it can be shared.