Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using preprocessors to cast into custom type #31

Open
y2k-shubham opened this issue May 13, 2019 · 1 comment
Open

Using preprocessors to cast into custom type #31

y2k-shubham opened this issue May 13, 2019 · 1 comment

Comments

@y2k-shubham
Copy link

I took hint from existing date_string_to_datetime preprocessor and wrote my own preprocessor that converts input string argument into an Enum.
.
While the conversion worked, during dag-build phase (Python code-generation), I ended up hitting this block and got Cannot format value {}: no handler for type {}.
.

  • Was this expected or am i doing something wrong?
  • Is there a reason why this is restricted?
  • Is there a workaround?
@mchalek
Copy link
Member

mchalek commented May 16, 2019

oh this is interesting @y2k-shubham . The issue is that Enum is not a supported type for the code that formats the arguments that get passed to operators in the generated DAG code. The formatter is here. As you can see, we have specific handlers for the various types of values that are typically encountered (e.g. strings, datetime, dict, etc), and we just have not ever encountered Enum before.

There are two possible solutions to this:

  1. Rather than generating an Enum, perhaps you can achieve your goal using a <<verbatim string>> (which will be pasted directly into the python dag)?
  2. We can add an Enum formatter to the method I linked up there ^^. This will require a core change and I'd want to make sure it does not introduce any unexpected problems, but in principle I don't see anything wrong with it.

Could you try out solution (1) to see if that works for you? If not, I'm spread a bit thin right now so I can't commit to adding (2) on any near-term timeline, but I'd be open to a contribution if you wanted to do it yourself (and I could help with code pointers etc).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants