504 Gateway timeout error with ALB

[kollol-chowdhury-incontact]

Hi Team,

We are getting a lot of 504 gateway timeout error when using Kong with AWS ALB.
We are hitting 25million requests in an hour and we can see that almost 2lakh request send 504 timeout error.

We are using Kong 2.0.3.
Has anyone ever faced this problem or help.

TIA

[kikito]

Hello,

2.0.3 is a bit old at this point so the support we can provide is very limited. Any other details that you could provide? Here's a list to give you examples:

• Are you using dbless config, or a database?
• What plugins are you using?
• How many Routes, Services and Plugin instances you have?
• Are you seeing any messages in the kong logs around when those errors happen?
• Do you have a huge number of loadbalancers?
• How often do you update the Routes? How often do you update your plugins?

Anything else along those lines that you think might be relevant

[kollol-chowdhury-incontact]

Hi @kikito ,
Thanks for the reply.

We also tried with Kong 2.7.1 and still facing the same issue.
Are you using dbless config, or a database?
Using dbless .
What plugins are you using?
We are using
How many Routes, Services and Plugin instances you have?
We have 25 services,routes and 17plugins(including the one which we have written) configured.
Are you seeing any messages in the kong logs around when those errors happen?
Nothing as such
Do you have a huge number of loadbalancers?
Using 2 load balancers(We have a global accelerator that routes traffic to the load balancers and each of the load balancer has separate target groups that it manages)
How often do you update the Routes? How often do you update your plugins?
Not very often.

We are using c5.2xlarge instances where our kong docker image is running.

We have idle timeout defined in ALB: The default value is 60s. We also increased it to 120 second. And in kong.conf we have set timeout to be 120s and when we increased alb idle timeout to 120s,increased Kong timeout to 180s.

The 504 error we are getting is in ALB and it seems like it doesn't reach upstream and timesout from Kong itself.
I have attached the configuration file. Please let us know if we are missing something.
kong - Copy.txt

[kollol-chowdhury-incontact]

Hi @kikito ,

Any update on this.

Thanks

[kikito]

Hello @kollol-chowdhury-incontact, sorry for not following up on this earlier.

You didn't include the list of Plugins you are using,

What plugins are you using?
We are using

But I found this list in the config file you provided:

          acd-cluster-routing, \
          incontrol-routing, \
          token-retrieve, \
          token-validate, \
          options-handler, \
          apifacade-transformer, \
          api-facade-services, \
          logging-services, \
          response-transformer, \
          request-transformer, \
          mock-services, \
          legacyID-validate, \
          swa-redirect, \
		  unified-login

Of that list, I only recognize some. You also mention including the one which we have written so I assume at least some of those are custom plugins, and others are community-provided. Those are our prime suspects when there are errors in a Kong app.

The support that we can provide for someone using custom plugins on a free plan is very limited. I can give you some general advice:

• Set up a staging environment configured as similarly as possible to production. Set it on a network and make sure that it doesn't leak traffic to production servers or upstreams (so you don't end up charging your real users for traffic generated during tests for example)
• Either split your inbound traffic into production and staging or generate fake traffic similar to the production system in staging. The objective is replicating the errors you are seeing in production on a non-production system.
• Once you have a staging environment that replicates production (including the errors you are seeing), start deactivating custom plugins one by one. If you are lucky it will became obvious which plugin is responsible for the errors (you will have to ignore other errors produced by deactivating the plugin). It might be a combination of plugins instead of just one.
• Once you have identified which plugins are producing errors, check the source code. In particular, pay attention to errors that might be being masked by not checking return values. For example, this code:

tcpsock:connect(host, port)

-- do other things with tcpsock assuming it has connected

Is masking a possible error since tcpsock:connect can fail. The return values have to be inspected and treated:

local ok, err = tcpsock:connect(host, port)
if not ok then
  kong.log.err("could not connect to the socket: ", err)
  return
end

-- do other things with tcpsock

Hopefully you should start seeing errors pop up in your logs when you get those 504s you mention.

• One last thing you can try is just adding debug lines to your custom plugins. That way you might be able to correlate the errors with steps that are missing only when there's errors.

If you need more (and faster) support, please consider upgrading to Kong Enterprise.

[dndx]

Hello, could you try with the latest Kong release (3.0.1)?

[darw-kong]

Hey, @kollol-chowdhury-incontact , my team and I have a similar issue, did you manage to solve it?

[RazanakotoMandresy]

Please , Did anyone know how to solve it , i'm using postgres and it's on a load balancer