Action Plan OPC-UA Adapter 🚀

[bossenti]

During the last time, we had a lot of discussions about weaknesses and potential features of our existing OPC UA adapter. To address those, I'd like to propose an action plan for this adapter that we can try to complete within the current release (0.93.0).
I browsed through the existing issues and discussions and came up with the following overview.

1) Migrate existing OPC UA adapter and sink to a common module
This is a first refactoring task which allows us later on to share features between the adapter and sink (e.g., authentication) more easily and achieve consistency (as we already did for influx). This should include:

• the actual refactoring to a shared module (Move OPC-UA processor and sink into a single module #1648)
• Unify the labels for adapter & sink (Unify the labels for OPC UA adapter & sink  #1664)
• remove nodeId & namespaceIndex from SpOocUaConfig
• add description to README how adapters can be integrated that build upon the OPC UA adapter

2) Pull & Subscription handling
We support two query options for OPC UA: pull-based and subscription. While we definitely want to keep both, their implementation in a shared OPC UA adapter introduces a lot of complexity and makes the code hard to read and understand. Therefore, we should add some separation here.

In addition there is one issue with the subscription mode and our current implementation. Let's say your adapter is connected two x nodes, then updates for all nodes can result in up to x events. This might not always be the desired behavior. Thus we could introduce a buffering mechanism and allow the user to activate the buffering within the configuration menu.

3) Support OPC UA timestamps
Currently, we don't support the timestamps that are provided by the OPC UA server for every node (server and source timestamp).
This can be achieved by providing a configuration field where a user can select which timestamp to use: server, source or StreamPipes. See also: #886.
To be clarified: Do both timestamps on OPC-UA side always exist? (Potential-Fallback: SP timestamp)

• Add option timestamp option to OPC UA adapter #889

4) Fully support OPC UA types
Our OPC UA adapter can only handle number-based identifier types. This is not compatible with the OPC UA specification (https://reference.opcfoundation.org/Core/Part6/v104/docs/5.2.2). Therefore, we should aspire to get more feature-complete here.

• Improve OPC UA Schema Detection to Support All Identifier Types #1579
• Support array types for OPC UA #1693

5) Remove Requirement of NodeId (& namespace)
If we compare to tools like UaExpert, there is no need to state an initial node id or namespace.
We should try to get the same behavior and reduce the configuration effort for users thereby.
Furthermore, this should address needs raised here: #888 & ttps://github.com//discussions/890

This requires a asynchronous browsing of available nodes (see 8).

6) Connection Caching
We could cache the connections to an OPC UA server so that they can be shared for multiple adapter instances.
See #888

7) Support encrypted connection
As of now, we only support no authentication and authentication via username/password.
Encrypted authentication is currently not supported.
See #1449

8) Load available nodes on demand
The current adapter tries to load all existing nodes in an OPC UA server recursively which can cause performance problems for larger servers. Therefore, we should introduce partial loading that allows to brows through the existing nodes iteratively.
See #895

• Support asynchronous browsing in OPC-UA adapter #1662
• OPC UA NullPointer Exception when Node Description is Missing #1713

9) Add integration test
To improve the robustness of our OPC UA implementation, we could introduce an integration test.
How could possible options look like? Starting a server in a docker container?

10) Handling null values
This is not specific to OPC UA, but I think it's time to handle null/optional values properly in StreamPipes.
References: #1596 & #1485

11) Celebrate new OPC UA adapter
Once we have implemented the desired improvements, we should promote & celebrate the new OPC UA adapter in form of a detailed blog post.

• Write blog post to demonstrate OPC-UA adapter #1551

In the midterm we could also think about having a look at the companion specifications and support some aspects there.

This collection is intended as a starting point for further discussion. What do you all think about the different aspects?
If not already there, let's create issues to complete the tasks and add them here or start discussions (as individual threads per topic below) about the individual steps.

cc: @Praveensteinb @vogler75 @curichan @ohlr @dominikriemer @tenthe

[dominikriemer]

Hi,
I've started to work on asynchronous browsing. A screenshot of the current prototype is below. Users can browse through all nodes, see details on every node and nodes can be selected before navigating to the next step of the StreamPipes Connect wizard.

Do you have any feedback/wishes? Especially I'd be interested in the metadata that should be shown when selecting a node - what metadata would be interesting/important to have in this section?

[vogler75]

Hi Dominik, Looks good! For the DataType it would be good to see the data-type instead of this NodeId. DisplayName would also be good - if you haven’t used this in the tree ;-) regards, Andreas

…

Am 09.06.2023 um 09:04 schrieb Dominik Riemer ***@***.***>: Hi, I've started to work on asynchronous browsing. A screenshot of the current prototype is below. Users can browse through all nodes, see details on every node and nodes can be selected before navigating to the next step of the StreamPipes Connect wizard. Do you have any feedback/wishes? Especially I'd be interested in the metadata that should be shown when selecting a node - what metadata would be interesting/important to have in this section? <https://user-images.githubusercontent.com/33908476/244617252-5bfb8bfb-b247-4612-83d9-cce4d0baec2f.png> — Reply to this email directly, view it on GitHub <#1650 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGEZIGDYE45QNTRTM4Q6BKTXKLDJTANCNFSM6AAAAAAYYVVBUY>. You are receiving this because you were mentioned.

[tenthe]

Hi @vogler75,
thank you very much for your reply.
As discussed with @curichan, we also want to support all identifier types (see #1579) in the future.
Do you know if the display name is always available regardless of the identifier type?

[tenthe]

Just a quick addition. My comment referred to the data type.
So how can you display the data type independently of the identifier type.
Sorry for the confusion.

[curichan]

Hi @tenthe,
https://reference.opcfoundation.org/Core/Part3/v105/docs/5.2.5 would imply that the provision of a display name is optional.

[curichan]

Hi @dominikriemer,
many thanks for this! I think this will be a huge improvement. It may be worth showing the current value of the variable as well.
Also the timestamp of the last update of the node would be useful. Sometimes there are "plausible" values reported, but they are "old". Seeing the timestamp of the current value would help to see whether a node is "stale" or active.

[curichan]

Hi all,
not sure this really belongs here, but I also noticed that Streampipes has a problem dealing with node values of type (Double) array:

The adapter creation page reads the node content correctly:

The result of reading this variable is shown correctly as an array with a single element. However, there is currently no way to "extract" this one element as an individual value. So, trying to use the array values in further processing, like storing in the local data lake or displaying the value in a visual in a dashboard or the data explorer is not possible. No error messages are shown. The values are simply not stored or displayed. I think it'd be useful to add a possibility to edit arrays in the "configure fields" step of adapter creation. It should be possible to store (may be not display) arrays in the local data lake as they are. But it should also be possible to configure the array field such that individual elements of the array are extracted, stored and displayed as individual values.

[tenthe]

Hi thanks a lot for reporting this.
In the current version it should be possible to us a pipeline element for arrays to transform the array into a basic value.
However, I like the idea of adding such a feature. But we need to make sure that it works as expected for all cases.
How many elements can be part of the array in your case? Is it always one?
Cheers,
Philipp

[curichan]

Hi @tenthe,
the workaround using a pipeline element hadn't occured to me. Thanks for that! Turns out in my case the number of elements in the array can change during runtime as it depends on the production process currently active in the machine feeding the OPC UA server. So a more general way of dealing with such a situation might be called for...
Cheers,
Christian.

[tenthe]

Ok thanks, in this case we need to discuss how we can achieve this.
Since the event schema of a stream created by the adapter is static, this means that we cannot dynamically change the amount of properties. In the case when there is only one entry in the array we could map it to a double property in the event.
What exactly is the data type of the field? On the screenshot it looks like it is Double or is it Double Array?

In case it is Double Array we could represent it as an array of doubles in the event. However, this will not solve the problem for the data lake and the data explorer 🤔

[curichan]

Hi @tenthe,
here a more complete screenshot:

My understanding is that in this particular case the data type is set to "Double", but there is a node property "is_array" which is "true". So I assume that OPC UA allows any data type to be associated with arrays, not just Double. So may be the way forward would be to represent the value of a node as an array in the event and really delegate dealing with arrays to pipeline elements?

[tenthe]

Yes, I think that's a good idea. I'll add an issue for this to the task above.
If arrays are a common case, we can also think about including a visualization for array values in the Data Explorer. But that is another discussion.