You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like to figure out a way if I can inject some Python code at the beginning of my DAG.
The idea is that I want to use the python_callable field on a PythonOperator to call my code, however my code is going to be like 5-10 lines of code and I cannot really represent it with lambda (or I would like not to, cause it's going to be ugly!)
Currently I see two options but I would like to get your feedback please :) Thanks!
Define a plugin that creates a new operator that inherits from PythonOperator and will do the magic. Then, Boundary Layer will load the YAML file I config in the plugin so the output DAG.py is valid. And, I'll also have to include the implementation itself (not the YAML) in the DAGs folder in airflow itself.
A better option(which I'm not sure if it is supported) will be to have some way to paste code at the beginning of the DAG.py file. This I guess can be done using a pre (or post) processor. However, I'm not sure if Boundary Layer supports something like that.
While exploring these options, I preferred to file a bug and see what you guys think about it.
Thanks!
The text was updated successfully, but these errors were encountered:
I would like to figure out a way if I can inject some Python code at the beginning of my DAG.
The idea is that I want to use the
python_callable
field on aPythonOperator
to call my code, however my code is going to be like 5-10 lines of code and I cannot really represent it with lambda (or I would like not to, cause it's going to be ugly!)Currently I see two options but I would like to get your feedback please :) Thanks!
PythonOperator
and will do the magic. Then, Boundary Layer will load the YAML file I config in the plugin so the output DAG.py is valid. And, I'll also have to include the implementation itself (not the YAML) in the DAGs folder in airflow itself.While exploring these options, I preferred to file a bug and see what you guys think about it.
Thanks!
The text was updated successfully, but these errors were encountered: