Outgoing Webhooks

[USBWAG]

Hello everyone

Have any of you tried outbound webhooks?

I'm currently working on version 9.4.0 and 11.0.2 for testing purposes, both of which are running without any problems, but now that I'm configuring the webhooks I'm running into problems

Locally I'm developing a python backend that should receive the PUT / GET /PUSH events, and it works so far

Can i only define 1 webhook? So, for example, only POST? Without GET and PUT? I haven't been able to do it at the moment

Locally it is only HTTP development, and when I try to send on an https connection I encounter errors.

Do the outgoing webhooks only support http destinations?
And which certificates is he trying to verify for the https interface?

The behavior of the whole thing is the same on both versions.

[USBWAG]

Hello everyone,
It's done, unfortunately I had to specify this variable on the server where Butler is running that it can send on HTTPS
NODE_TLS_REJECT_UNAUTHORIZED = 0
Otherwise I could not put the whole thing into operation.

Therefore my question to the users, do you always need 3 "webhooks" or would one kind not be enough?
Would it make sense to extend the whole thing with HTTPS authentication? (SSL Certs)
Or would a header authentication be better?
What do you think?

And would then directly simplify an adjustment of the config file in this syntax with the approach here

webhookNotification:
enable: true
reloadTaskFailure: true
reloadTaskAborted: true
serviceMonitor: true
httpmethods:
POST: true
PUT: true
GET: true
baseURL: "https://thebutlerbackend.com/"
ssl: true
certfile: "/path/to/your/certificate.crt"
keyfile: "/path/to/your/private_key.key"
webhooks:
- description: 'This is The Outgoing POST notification'
path: "/BUTLER_POST"
- description: 'This is The Outgoing PUT notification'
path: "/BUTLER_PUT"
- description: 'This is The Outgoing GET notification'
path: "/BUTLER_GET"

What do you think?

I would also like to support the Refactor part of the Webhook's if you like

[mountaindude]

Hey, thanks for your insights and feedback.

The very latest Butler uses a newer Node.js version, which changed some things around TLS certificates. That could be (part of) the reason you are seeing this.
Or... the webhook code in Butler is pretty old and to be honest I am not sure it's verified to work with https webhooks. Need to check that.

Re your question about whether 3 webhooks are always needed the answer is no.
The reason they are there in the template config file is simply to show that it's possible to use either of POST, PUT and GET methods in the outgoing webhook.
You can have 1 GET hook, or 5 POST hooks. Or any combination.

I'll look into the TLS part.
If nothing else it should be possible to state that self-signed certificates should be allowed. Should also be possible to enable/disable the individual hook types (for reload failed/aborted, service monitoring etc).

[mountaindude]

@USBWAG

I took a look at Butler's code for outgoing webhooks, and my suspicion was correct:
That code was created long ago for a use-case where TLS (i.e. https) was not needed.
There is no handling of self-signed certificates, ignoring certificate errors or similar.
That explains why things worked when you set NODE_TLS_REJECT_UNAUTHORIZED = 0 on the server where Butler is running.

Adding support for self-signed certificates to outgoing webhooks is certainly possible, but won't happen overnight, I am afraid.
Hard to give an exact date, but realistically I'd say a few weeks.

The work that needs to be done is more or less:

• Add entries to the YAML config file to handle custom certificates that should be used when doing outgoing webhook calls
• Add the new entries to the config file verification/assert code.
• Add code to use those certificates if they are enabled in the config file

For reference, most of the webhook related code is found here.

[mountaindude]

@USBWAG By the way, I realised that the docs at butler.ptarmiganlabs.com are not very complete when it comes to describing possible destinations for reload failure/aborted alerts.

FYI, I created a ticket to track this: #1078

[mountaindude]

Alright, I am thinking along these lines for adding support for outgoing webhooks calls to handle self-signed certificates in a good way.

• You can use https already today if the https server uses a proper certificate that is backed by a public CA. Just use https and things should work.
• If using self-signed certificates on the server the webhook will call, Butler must be told about that certificate's CA. Otherwise there will be a certificate warning.
• Add an option for disregarding certificate warnings. Useful during testing, for example.

As different outgoing webhooks may have their own requirements with respect to certificates etc, it makes sense to use separate cert configs for each webhook.

That would give us something like this in the config file:

  # Settings for notifications and messages sent using outgoing webhook
  webhookNotification:
    enable: true
    reloadTaskFailure:
      rateLimit: 15                   # Min seconds between outgoing webhook calls for a given taskID. Defaults to 5 minutes.
      webhooks:
        - description: 'This outgoing webhook is used to...'
          webhookURL: https://10.11.12.13:8765/butler_get              # outgoing webhook that Butler will call
          httpMethod: GET                   # GET/POST/PUT. 
          cert:
            enable: true                    # Set to true to use a custom CA certificate when calling the webhookURL
            rejectUnauthorized: true       # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
            certCA: /path/to/ca-cert.pem       # Path to the CA certificate file

[mountaindude]

@USBWAG

11.2.0 was just released, available here.

It adds some improvements around webhooks in general, including the option to use self-created TLS certificates.
More info on the doc site: https://butler.ptarmiganlabs.com/docs/concepts/failed-reloads/webhook/

[USBWAG]

Hey @mountaindude thank you so much for the quick customization,
I will be happy to test it this week and give you feedback

Thank you so much for your hard and great work with all the Butler tools

[mountaindude]

No worries, those updates were long overdue.

I am putting final touches to a new version today, so it may be worth waiting a few more hours...
The new version does not add any new features, but instead focus on verifying that the config file contains all required settings, and also the correctness of those settings (to the extent possible, which means ensuring that there are numbers where numbers are needed etc).

[mountaindude]

...and version 12.1.0 is out: https://github.com/ptarmiganlabs/butler/releases

[USBWAG]

Hello @mountaindude, thank you for your adjustments so quickly, last week I was no longer able to test it.

But this week,
Unfortunately I can't get version 12.1.0 to run, the config file seems to be correct, but the changes in Assert_config_file.js don't seem to query correctly:

We don't use a New Relict connection, so I never configured that part.

even if I leave it blank:

If I use fake data I get a little further, but then it says there is something wrong with the signl4 settings (we don't use that either)

The problem also exists with version 12.2.0 and the adapted config file

Currently I tried with the Windows binary.

Was it meant that i have to fill everything out now?

[mountaindude]

Ah yes... Recent Butler versions are MUCH more demanding when it comes to the config file.

A bit of background might help explaining things.

Over time lots of different features have been added to Butler, most of those features also brought new things and/or changes to the config file. Over time the config file has become a somewhat of a beast...

In order to establish a solid foundation from which to restart, the decision was made to make all config items mandatory, even for features that are not enabled.

I am the first to agree that this makes upgrading Butler harder.
But it will also make it easier to compare your config file with the documentation at butler.ptarmiganlabs.com.

Right now your only option is really to bring your config file up to speed, which is what you've already done - good work there.

The error messages show pretty clearly what's missing in the config file.
But... the Signl4 settings you mentioned were in fact also missing from the sample/template config file that ships with Butler, as well as on the doc site. Duh.
The doc site has been updated and Butler 12.2.1 is being built just now.

Finally, the last error (...static[0]) is actually a bug.
The config-checking code that runs at startup checks if the array at Butler.incidentTool.newRelic.reloadTaskFailure.destination.log.attribute.static consists of strings.
Which is wrong - that array should consist of objects, each with name and value properties:

The workaround is to leave that array empty, i.e. delete whatever is in it.
A fix for this bug is coming in 12.3.0 later tonight or tomorrow.

Looking forward

Going forward, my feeling is that the startup check of the config file should be a bit more relaxed.
If some Butler feature is not enabled, maybe the associated settings in the config file should not be checked on startup...
Maybe. Will have to think about that one.

Also, adding a new command line option --skip-config-verification makes it possible to skip the startup check altogether.
There's then a risk that Butler will crash or not work properly later on if something is invalid in the config file, but that option could still be useful. Tracked in #1114.
Butler SOS already has this, so it's a reasonable addition.

[USBWAG]

Great work , now i was able to test it, works wonderfully in version 12.3.0, I just tested it :)

To be honest, I think the fact that Yaml is now more strict is a good thing, as it avoids adjustment errors.

And the license feature works wonderfully without any problems

[mountaindude]

Good to hear!

…

On Thu, 25 Apr 2024, 10:51 Raffael W, ***@***.***> wrote: Great work , now i was able to test it, works wonderfully in version 12.3.0, I just tested it :) To be honest, I think the fact that Yaml is now more strict is a good thing, as it avoids adjustment errors. And the license feature works wonderfully without any problems — Reply to this email directly, view it on GitHub <#1040 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH3JDSNUXIOCQ34FLP6F7TY7C7YHAVCNFSM6AAAAABFJHOHI6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TEMRSGU4DG> . You are receiving this because you modified the open/close state.Message ID: ***@***.*** com>