slurm-web-agent-uwsgi not communicating with slurmrestd

[rseaman2016]

Hey there! I am running into an issue where when navigating to slurm-web, the cluster list is empty:

Upon further investigation, slurmrestd does not seem to be receiving requests from slurm-web-agent-uwsgi, but can receive requests when running the slurmrestd troubleshooting steps as the slurm user

When run locally as slurm user:

slurmrestd[806485]: operations_router: [/run/slurmrestd/slurmrestd.socket->socket:[26252175] (fd 8)] GET /slurmdb/v0.0.39/config
slurmrestd[806485]: rest_auth/local: _auth_socket: _auth_socket: [/run/slurmrestd/slurmrestd.socket->socket:[26252175] (fd 8)] accepted user socket connection with uid:202 gid:202 pid:806573
slurmrestd[806485]: slurmrestd: operations_router: [/run/slurmrestd/slurmrestd.socket->socket:[26258018] (fd 8)] GET /slurmdb/v0.0.39/config
slurmrestd[806485]: operations_router: [/run/slurmrestd/slurmrestd.socket->socket:[26258018] (fd 8)] GET /slurmdb/v0.0.39/config

Drop-in file /etc/systemd/system/slurmrestd.service.d/slurm-web.conf:

[Service]
# Unset vendor unit ExecStart to avoid cumulative definition
ExecStart=
Environment=
# Disable slurm user security check
Environment=SLURMRESTD_SECURITY=disable_user_check
ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS unix:/run/slurmrestd/slurmrestd.socket
RuntimeDirectory=slurmrestd
RuntimeDirectoryMode=0755
User=slurm
Group=slurm

Service file for slurm-web-agent:

[Unit]
Description=uWSGI instance for Slurm-web agent
After=network.target

[Service]
User=slurm
Group=slurm
RuntimeDirectory=slurm-web-agent
ExecStart=/usr/sbin/uwsgi --ini /usr/share/slurm-web/wsgi/agent/slurm-web-agent.ini

[Install]
WantedBy=multi-user.target

I am running slurm-web-agent-3.2.0-1.el9.noarch and slurm-ohpc-slurmrestd-23.11.6-310.ohpc.5.1.x86_64. Any pointers would be appreciated.

Thanks!

[rezib]

Hello @rseaman2016, I convert your issue #380 into a discussion as it is more a request for help than an actual issue.

Can you look into the logs of the agent service? Maybe your user does not have any permission on your cluster?

[rseaman2016]

Hi @rezib, that did the trick. The logs pointed to an invalid JWT token signature.

I'm guessing what happened is that since I have the slurm-web gateway on a separate machine that doesn't have the slurm user, the key didn't have the slurm user set as a valid audience (I'm assuming here, I'm not super familiar with JWTs).

I fixed this by adding a slurm user on the gateway, then rerunning slurm-web-gen-jwt-key, and copying the key over to the host running the slurm-web agent.

Thanks for your help!

[rezib]

Yes, when agent and gateway are deployed on separate hosts, the JWT signing key must be the same on both.