Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource sentinel nodes inside generators use incorrect task id's #29

Open
mchalek opened this issue May 2, 2019 · 0 comments
Open

Comments

@mchalek
Copy link
Member

mchalek commented May 2, 2019

We have found that under the following conditions, an error occurs:

  • A DAG uses a generator with at least two elements
  • The generator node has downstream dependencies
  • A resource is created inside the generator workflow
  • At least one node in the generator workflow both requires the resource and has no downstream dependencies that do not require the resource.

Under these conditions, boundary-layer by default inserts a sentinel node downstream of the node(s) matching these conditions, in order to propagate errors past the resource-destroy step. However, inside the generator, the sentinel node that is created does not have <<item_name>> appended to its task_id. This causes errors when that node is connected to the generator's downstream dependencies, because the Airflow set_upstream() method will be called repeatedly (once per generator element) for the (sentinel_node, generator_downstream_dependency) pairs. Airflow is proactive about alerting on this, which is nice (although in theory it's not really a problem...).

Anyway this is a bug that we should fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant