Can not connect to local database on Ubuntu

[nazShg48]

I get timeout error or connection to server at "one of the host names that i use" port 5432 failed: Connection refused when trying to connect to a postgresql that I have locally on my OS, Ubuntu 18.04.6 LTS, from a lambda that is running in sam.

In pg_hba.conf I set 0.0.0.0/0 for all IPv4 connection just to ensure that it is not an issue.
Database is working fine as I can use it from other applications that I am running on my local machine, but they are not using sam.

I used all possible combinations for the database host: 172.17.0.1 which is address of docker0, localhost, 127.0.0.1, host.docker.internal
And commands to run aws sam: sam local start-api --docker-network host, sam local start-api, sam local start-api --container-host localhost --container-host-interface 0.0.0.0 --docker-network host, sam local start-api --container-host localhost --container-host-interface 0.0.0.0 and some others.

Result is the same, either timeout error or connection to server at {one of the host addresses} port 5432 failed

SAM CLI, version 1.56.1

[sriram-mv]

@nazShg48 are you able to connect to your service running on your host from another docker container? Trying to triage if this is environment specific or directly related to interpolation of options supplied to AWS SAM CLI.

[nazShg48]

@sriram-mv not sure I understand what I should check. Are you asking me to run my lambda in aws sam and then try and access it from another service that is running in pure docker container (not in aws sam) ?

[nazShg48]

Any idea?

[sriram-mv]

Apologies for the late response, my question is if you are able to access services running within a docker container from the host directly without the use of SAM CLI. example: running postgres on a container and accessing the service from the host.

[nazShg48]

I haven't tried it since I need to connect from SAM to local PG instance which is not in docker, but running locally. I and my teammate tried to use SAM on his machine, with Ubuntu 22.04 LTS , same problem. I am pretty sure it is universal problem.

[nazShg48]

Also I was able to run dockerized (non SAM) backend server and connect to it from my machine.

[nazShg48]

Any updates here?

[nazShg48]

Bump

[aniruddh-purohit]

Yeh, same problem. Trying to connect mysql running on localhost from sam local invoke on Ubuntu 22.04. None of the solutions is working.

[linhabc]

Any idea? I got same problem on ubuntu 20.

[johanmynhardt]

I'm having similar pains.

Scenario:

• Windows 11/WSL2 using the Ubuntu distro.
• Natively on a Linux host as well (Pop!_OS 22.04 LTS x86_64)
• An ssm tunnel to an RDS instance using aws ssm start-session --target <instanceId> --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters ...
  • port on docker host: 3330
• I can verify that using docker run with the --network=host option works, eg:
  • docker run --rm -it --network=host mysql mysql -u <user> -h 127.0.0.1 -P 3330
• I can also verify that the nodejs20.x image by itself is able to use --network=host correctly:
  • docker run --rm --network=host public.ecr.aws/lambda/nodejs:20-rapid-x86_64 bash
  • while that is running, I get the container name using docker ps
  • docker inspect --format='{{range $k, $v := .NetworkSettings.Networks}}{{$k}}{{end}}' container_name returns host or bridge (bridge by default if --network is left out)
  • attempting to interact from within the running container, to 127.0.0.1:3330 suggests that it is able to communicate in the established session.

However, when running sam local invoke --docker-network=host ... and do docker inspect ..., it remains as bridge and the connection attempt during invocation fails.

[nascyimento]

Same problem here

[nazShg48]

So far I was not able to fix the issue, but there is a workaround.
Here are a steps to make it work locally on Ubuntu (I use PG, but it will probably work with any other database).
Replace placeholders "< >" with a values that you want. After that use the same value when you face a same placeholder.

1. Create new docker network:
   docker network create <network_name>
2. Create docker volume for PG data storage:
   docker volume create <volume_name>
3. Build PG docker image.
   My Dockerfile_pg looks like:

# Use the official PostgreSQL image as the base
FROM postgres:14.11

# Set environment variables
ENV POSTGRES_PASSWORD=mysecretpassword

# Expose the PostgreSQL port
EXPOSE 5432

docker build -t <tag_name> -f Dockerfile_pg .
4. Run docker container with the DB within created docker network:

docker run --name <container_name> \
--network=<network_name> \
-v <volume_name>:/var/lib/postgresql/data \
-d <tag_name>

5. To connect to this docker container with a database, you have to get it's IP, use following command:
   docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container_name>

after that you can pass this IP to your database URL when you are trying to connect to it

6. And the last step is to run AWS SAM within created docker network
   sam build && sam local start-api --docker-network <network_name>

[GuillaumeDesforges]

I have a similar issue.

I am trying to run sam local invoke like so, but it can't connect to local Postgres:

sam local invoke -e event.json --docker-network host

Notice --docker-network host above.

Postgres runs on my host using a simple Docker compose.

services:
  database:
    image: postgres
    network_mode: host
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
     - ./tmp/compose/database/data:/var/lib/postgresql/data

Notice network_mode: host above.

I can connect to the database running this command on my machine:

psql --host=127.0.0.1 --user=postgres

Then I try to connect from another container with

docker run --rm --network host -it postgres psql --host=127.0.0.1 --user=postgres --password

and it works.

My guess is sam local invoke does not apply --docker-network host properly.

[GuillaumeDesforges]

Interestingly, down the line I found that the CLI argument sets a docker_network_id

aws-sam-cli/samcli/local/docker/manager.py

Line 95 in 5712c7f

container.network_id = self.docker_network_id

So I tried using the id instead, but that gave me
container cannot be disconnected from host network or connected to host network.

This error led me to #669, and https://github.com/aws/aws-sam-cli/pull/715/files.
Setting docker_network to host should be handled, but it was removed
369aa72#diff-89004c015f06af81ec7bf99d97b6827403738d2df8a288d05459b37706edfccaL209-L211

[GuillaumeDesforges]

I tried to run with the patch

diff --git a/samcli/local/docker/container.py b/samcli/local/docker/container.py
index a34e96c7..2244ffc9 100644
--- a/samcli/local/docker/container.py
+++ b/samcli/local/docker/container.py
@@ -233,6 +233,10 @@ class Container:
             # Ex: 128m => 128MB
             kwargs["mem_limit"] = "{}m".format(self._memory_limit_mb)
 
+
+        if self.network_id == "host":
+            kwargs["network_mode"] = self.network_id
+
         if self._extra_hosts:
             kwargs["extra_hosts"] = self._extra_hosts

But I got Error: "host" network_mode is incompatible with port_bindings

EDIT: seems to come from

aws-sam-cli/samcli/local/docker/container.py

Lines 219 to 227 in 5712c7f

	kwargs["ports"] = {self.RAPID_PORT_CONTAINER: (self._container_host_interface, self.rapid_port_host)}
	if self._exposed_ports:
	kwargs["ports"].update(
	{
	container_port: (self._container_host_interface, host_port)
	for container_port, host_port in self._exposed_ports.items()
	}
	)

which binds at least one port, so of course network mode can't be set to host.

[GuillaumeDesforges]

Here goes nothing 🙂

#7278

[GuillaumeDesforges]

See workaround here: #7278 (comment)

[hawflau]

Thanks @GuillaumeDesforges for providing the solution. In short, create a docker network and connect your local database through the network.

Closing as a solution is provided. Feel free to open a new issue if you have any further question.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.