You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have found that under the following conditions, an error occurs:
A DAG uses a generator with at least two elements
The generator node has downstream dependencies
A resource is created inside the generator workflow
At least one node in the generator workflow both requires the resource and has no downstream dependencies that do not require the resource.
Under these conditions, boundary-layer by default inserts a sentinel node downstream of the node(s) matching these conditions, in order to propagate errors past the resource-destroy step. However, inside the generator, the sentinel node that is created does not have <<item_name>> appended to its task_id. This causes errors when that node is connected to the generator's downstream dependencies, because the Airflow set_upstream() method will be called repeatedly (once per generator element) for the (sentinel_node, generator_downstream_dependency) pairs. Airflow is proactive about alerting on this, which is nice (although in theory it's not really a problem...).
Anyway this is a bug that we should fix.
The text was updated successfully, but these errors were encountered:
We have found that under the following conditions, an error occurs:
resource
is created inside the generator workflowUnder these conditions,
boundary-layer
by default inserts asentinel
node downstream of the node(s) matching these conditions, in order to propagate errors past the resource-destroy step. However, inside the generator, thesentinel
node that is created does not have<<item_name>>
appended to itstask_id
. This causes errors when that node is connected to the generator's downstream dependencies, because the Airflowset_upstream()
method will be called repeatedly (once per generator element) for the(sentinel_node, generator_downstream_dependency)
pairs. Airflow is proactive about alerting on this, which is nice (although in theory it's not really a problem...).Anyway this is a bug that we should fix.
The text was updated successfully, but these errors were encountered: