Push stats instead of pulling

[Hutch79]

Hey there,

First of all I love this project!
Finally, a monitoring solution which is very quick to set up, thanks for the work so far!

Now to my suggestion.
The current setup works (as far as I understand it) that the host reaches out to the agent and pulls the stats from it.
My suggestion would be to reverse this process, so the agent pushes the stats to the Host via an API call.

This would have two benefits.

1. Different Tokens/Key's. Currently, every Agent is set up with the same SSH-Key. With an API every Agent could have a different token which decreases the risk of accidental leaking. I guess individual SSH-Keys would also be possible with the current setup.
2. Only the Host would need to be accessible via a Port. Currently, every Agent needs a port, so the Host can access it. Iv we reverse this setup, only the Host needs an open port.
   The API/Push approach would also allow for custom Agent's to be created more easily.

Anyway, I'm curious where this project goes!

[henrygd]

This has been brought up previously in #16 and #28, but this seems like a better place to discuss it.

It's on the radar for the future but not something I'm planning to jump into immediately.

For those having trouble with the current setup, it may be possible to connect using services like Cloudflare tunnel, WireGuard, or Tailscale.

[Hutch79]

I understand that this is not priority number one.
First everything must be stable as is, after that reworking existing stuff can happen.

I had connecting trough VPN's or overlay networks like Tailscale or Netbird in my mind, but my networking knowledge is not good enough to implement such a setup. Also, an open port is not a dealbreaker for me personally, but for some it might be.

[TheAnachronism]

Really hope this can be done at some point. With monitoring many VMs on a hypervisor, or systems that don't have public addresses, this would be tremendous.

[gedw99]

I am using NATS JetStream embedded in each server to do this.

NATS is push based and does not need public Ips and has its own security also so it can do multi tenant.

its works very well

[tuxdotrs]

Hey @henrygd! Are you open to pull requests? If so, I'd like to work on this.

[henrygd]

Sure, you can take a stab at it if you want.

On the agent side we'll probably want a ticker that sends the stats every 60 seconds. You can get the stats via the gatherStats method on the agent.

On the hub side I'm thinking maybe an mTLS endpoint is the way to go as that takes care of auth without needing to implement / manage tokens. The endpoint would receive the stats with system id and add the records.

The agent side could define the cert / key as env vars. If those env vars exist, use the ticker to push stats rather than starting the SSH server.

Then the other tricky part is that we'll need to exclude the systems that receive stats this way from the hub's current batch update that sends SSH requests for stats every 60 sec. Off the top of my head, I'm not sure of the best way to do this. Maybe just a bool db field on the system and the user can switch the mode?

Let me know if you have any better ideas.

Edit: We might also want to check with the hub first that the system is not paused before getting / sending stats.

[gedw99]

nats can be embedded in the existing agent and hub . It’s designed for this in golang projects. I typically run each agent as a nats leaf so it’s self autonomous.

You can also setup multi data center systems so the hub can see all your servers across both premise and cloud. On premise is possible because nats is push based. You don’t need yailscale or cf tunnel etc ..

nats can send the stats in real time if you want . That’s what it’s designed for. It can also have a TTL .

nats has built in security at auth and authz . This is fully controlled by nats itself . Revocation is instant across the whole hub .

You can pause and restart stars sending also as needed.

If you’re doing this with NATS Jetstream then I can help .

It can also be used to update agent binaries via the hub btw.

It can also manage dockers that become network partitioned. So if it’s networking becomes flakey for whatever reason . This works both ways too . So you can get stats from the agent , when the docker eventually recovers and you can update the agent itself when it eventually comes back on line .

lots of possibilities with nats :)

The nats cli can see and manage things across the whole network.

[henrygd]

I have no experience with it but will definitely look further into it when I have time.