Anyone using the Flink wrapper?

[dominikriemer]

Hi,
in streampipes-extensions, we have a few modules that provide pipeline elements using the Apache Flink wrapper. This was a very protoypical implementation we haven't touched for a while. The main drawback is that pipeline elements need to be explicitely developed for Flink and cannot run in the default standalone mode.

While refactoring the API, I discovered that these pipeline elements are not supposed to work anymore since we switched to Spring Boot 3 which requires Java 17, while Flink supports Java 11 as maximum version.

So I guess nobody is using Flink components? In this case, I'd propose to migrate a few Flink-based pipeline elements we would like to keep to the default implementation and remove the extensions modules afterwards. We can still keep the current implementation of the Flink runtime in case somebody wants to work on this.

What do you think?

[bossenti]

Do we want to collect feedback on this from users@streampipes... as well?

In this case, I'd propose to migrate a few Flink-based pipeline elements we would like to keep to the default implementation and remove the extensions modules afterwards.

What do you mean hereby? How would such a migration look like?

[dominikriemer]

There are few pipeline elements such as the MeasurementUnitConverter where we could just port the application logic to the default way of implementing pipeline elements using the standalone wrapper. Another one would be the Elasticsearch sink, which also is only available for Flink at this time.
So migration in this sense is just porting the code ;-)

[bossenti]

Ah okay, now I got you 👍🏼

Sounds like a good idea 🙂

[obermeier]

Hi, in streampipes-extensions, we have a few modules that provide pipeline elements using the Apache Flink wrapper. This was a very protoypical implementation we haven't touched for a while. The main drawback is that pipeline elements need to be explicitely developed for Flink and cannot run in the default standalone mode.

While refactoring the API, I discovered that these pipeline elements are not supposed to work anymore since we switched to Spring Boot 3 which requires Java 17, while Flink supports Java 11 as maximum version.

So I guess nobody is using Flink components? In this case, I'd propose to migrate a few Flink-based pipeline elements we would like to keep to the default implementation and remove the extensions modules afterwards. We can still keep the current implementation of the Flink runtime in case somebody wants to work on this.

What do you think?

I also think that the existing modules are better located in a "normal" module since they do not use many flink features.

But I would not give up the Flink integration complety (Maven-Flink-Archetype).
For me Flink is the most important integration component when it comes to aggregations.
Especialy for window operations.

[dominikriemer]

Hi @obermeier thanks for the feedback! I'd definitely leave the Flink wrapper and archetype in the codebase so that someone can pick up development in case we find someone who's interested.
Just out of curiosity, how do you currently use Flink components? I didn't manage to get them working with Java 17. If you are interested, could you have a look at the current implementation in the #1632 branch and see if it works since I wasn't even able to start this in the minicluster?

[obermeier]

Hi @dominikriemer ,

thank you!
I use Flink for some aggregations e.g for visualistation of facility data,
clculation KPIs like the Mean Time Between Failures ...

I currenty use Java 8/11 for this jobs and hope that Flike will support Java 17 in the future ;)

[dominikriemer]

@obermeier FDA looks like a cool project ;-)

So I'll create an issue to migrate a few Flink processors to the default runtime and then integrate the remaining Flink modules into a single new module streampipes-pipeline-elements-experimental-flink and leave the wrapper as-is!