feat: Use python package if available #15
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a first iteration at installing and running the pip installable packages instead of using docker when theyre available, otherwise continue using docker. The main challenge with running airbyte sources with meltano is that they require docker to be available which makes the normal deployment routes more difficult (i.e. needed docker in docker support).
I found this was the simplest approach since the alternative options require installing the source python package at meltano install time which is hard to do. I attempted to do that with a custom install script but it was hacker than this approach and would require us to determine which sources are pip installable and update the hub which is a pain.
It simply checks if the package is pip installable via the registry endpoint that airbyte provides and installs it in a new virtualenv if it is.