Improve Third-Party Deployment Plugins Reliability and Compatibility

[DimedS]

Overview

Part of #4317. This issue focuses on addressing user challenges with Kedro's third-party connection plugins for deployment. We aim to ensure that these plugins remain reliable and compatible with the latest Kedro versions while offering a flexible approach to managing platform-specific features.

User Insights and Challenges

• Kubeflow Plugin: Users are exploring ways to use the Kubeflow plugin for compiling and deploying Kedro pipelines but face issues with outdated functionality.
• Databricks Integration: Some users prefer interacting with the Databricks API manually via Asset Bundles to achieve greater flexibility, rather than using the Kedro-Databricks plugin.
• Argo Workflows: Translating Kedro pipelines into Argo Workflows manifests is a significant challenge.
• SageMaker Plugin: The Kedro-SageMaker plugin was not used due to incompatibility with Kedro 0.19, limiting its application for certain projects.

Each connection plugin includes two key functionalities:

• Converting Kedro pipelines for different platforms
• Handling API interactions to upload and execute the converted pipelines

Goals

Ensure Long-Term Compatibility: Maintain and update third-party plugins regularly to align with the latest Kedro releases.

Possible Solutions

• Split Plugin Functionality: Divide each plugin's functionality into two parts:
  • Pipeline Conversion: Kedro maintainers will support and maintain the part responsible for converting Kedro pipelines to various platforms.
  • API Interactions: The community will handle the API interaction component, responsible for uploading and running the converted pipelines on specific platforms.
• Resource Optimization: By splitting responsibilities, Kedro maintainers can focus on the core functionality (pipeline conversion), saving resources while ensuring essential compatibility, and empowering the community to maintain API-specific integrations.

Summary of Tech Design - Dec 11, 2024

The priority for Kedro maintainers should focus on improving the deployment experience for three main cloud platforms: Sagemaker, AzureML, and Google VertexAI.

• Short-term goal: Ensure these plugins are compatible with the latest version of Kedro through collaboration with GetInData and participation of Kedro maintainers in development.
• Long-term priorities:
  • Explore the overall deployment experience on cloud platforms to better guide users and inform future development of Kedro and its plugins.
  • Implement changes in Kedro to simplify and strengthen the development of deployment plugins.
  • Investigate @deepyaman's idea to create a universal deployment product (asset) that Kedro can leverage, potentially eliminating the need for individual deployment plugins.

Next Steps

• Update the kedro-sagemaker plugin to ensure compatibility with the latest version of Kedro

[DimedS]

Summarised the current state of deployment plugins, highlighting the node grouping functionality:

https://www.notion.so/mckinsey/1423043f5dcf80b7a6cdf21901ff55c2?v=7958cdee2674405b82de8dd318d9fa1f

[DimedS]

For those who don’t have access to Notion:

[DimedS]

Summary of Tech Design - Dec 11, 2024

Video

Discussed

• Reviewed the current state of deployment plugins (Notion Link).
• Summarized the outcomes of the conversation with GetInData on December 9.

Decided

The priority for Kedro maintainers should focus on improving the deployment experience for three main cloud platforms: Sagemaker, AzureML, and Google VertexAI.

• Short-term goal: Ensure these plugins are compatible with the latest version of Kedro through collaboration with GetInData and participation of Kedro maintainers in development.
• Long-term priorities:
  • Explore the overall deployment experience on cloud platforms to better guide users and inform future development of Kedro and its plugins.
  • Implement changes in Kedro to simplify and strengthen the development of deployment plugins.
  • Investigate @deepyaman's idea to create a universal deployment product (asset) that Kedro can leverage, potentially eliminating the need for individual deployment plugins.

Next Steps

• Coordinate with GetInData to provide Kedro maintainers access to plugin repositories.
• Update the kedro-sagemaker plugin to ensure compatibility with the latest version of Kedro

Feel free to share your thoughts here and suggest any corrections if there's anything inaccurate in the summary.