Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Readiness probe error not handled due to php-fpm-healthcheck timeout #41

Open
xriser opened this issue Oct 19, 2020 · 7 comments
Open
Labels
bug Something isn't working
Milestone

Comments

@xriser
Copy link

xriser commented Oct 19, 2020

When php-fpm stuck and didn't respond, php-fpm-healthckeck can't handle this, and timeouts to connect to the port.

root@deployment-php-5bddfd7964-7ctmf:/app# php-fpm-healthcheck -v Trying to connect to php-fpm via: localhost:9000/status ^C
So, php-fpm-healthcheck doesn't return fail status in a reasonable time.

Since K8s doesn't handle timeouts healthcheck, it doesn't work. Because Kubernetes remove the pod from service only when heathcheck returning fail event.
Readiness probe errored: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Readiness probe errored, but pod didn't remove from services.

Is there any way to setup a reasonable timeout to connect to the PHP port for the php-fpm-healthcheck?

@chrispage1
Copy link

I'm getting the same - makes it a little useless when PHP-FPM stops responding

@chrispage1
Copy link

chrispage1 commented Dec 16, 2020

Just to update with this one for anyone else searching, I managed to resolve using BusyBox timeout command in my Kubernetes liveness probe...

livenessProbe:
  exec:
    command:
     - timeout
     - "1"
     - php-fpm-healthcheck
     - --listen-queue=15 # fails if there are more than 10 processes waiting in the fpm queue

This means that if php-fpm-healthcheck doesn't return success within 1 second it'll assume its dead.

@xriser
Copy link
Author

xriser commented Dec 17, 2020

Just to update with this one for anyone else searching, I managed to resolve using BusyBox timeout command in my Kubernetes liveness probe...

livenessProbe:
  exec:
    command:
     - timeout
     - "1"
     - php-fpm-healthcheck
     - --listen-queue=15 # fails if there are more than 10 processes waiting in the fpm queue

This means that if php-fpm-healthcheck doesn't return success within 1 second it'll assume its dead.

Did the same before. It works
livenessProbe: exec: command: - /usr/bin/timeout - "3" - /usr/local/bin/php-fpm-healthcheck - --listen-queue=100 initialDelaySeconds: 300

@chrispage1
Copy link

@xriser do you find PHP-FPM becomes unresponsive often? I'm having problems where the node will restart about 10 times a day due to PHP-FPM not responding

@xriser
Copy link
Author

xriser commented Dec 20, 2020

@xriser do you find PHP-FPM becomes unresponsive often? I'm having problems where the node will restart about 10 times a day due to PHP-FPM not responding

no, it depends on the application. If the app doesn't free php child they start to grow and running out of active child limits.

@wandersonwhcr wandersonwhcr added the bug Something isn't working label Jan 5, 2022
@wandersonwhcr wandersonwhcr added this to the v0.6.0 milestone Jan 5, 2022
@wandersonwhcr
Copy link
Collaborator

If we implement a --timeout feature, we can also close #28 .

@wandersonwhcr
Copy link
Collaborator

wandersonwhcr commented Jan 17, 2022

Is this doc relevant?

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes

But yes, I think we must implement a --timeout feature, too.

image

@wandersonwhcr wandersonwhcr mentioned this issue Jan 18, 2022
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants