Bug: sam local start-api fails to initialise lambda function containers within podman/toolbox container

[AidanOB]

Description:

When attempting to run sam local start-api from within podman/toolbox on Fedora 40 Kinoite, the lambda function containers fail to initialise. Command hangs for over an hour (at least) without initialising containers. On one occassion, when left overnight it initialised the container and was ready to be called.

To test, utilised AWS Lambda powertools tutorial template found here: https://docs.powertools.aws.dev/lambda/python/latest/tutorial/#code-example

Output Initializing the lambda functions containers.

Steps to reproduce:

Fedora 40 Kinoite -> Toolbox

toolbox create sam-test
toolbox enter sam-test
cd temp
sudo ./sam-installation/install
export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
sam build && sam local start-api --debug

Observed result:

[user@toolbox hello-world-tutorial]$ sam build && sam local start-api --debug
Starting Build use cache                                                                                                              
Manifest is not changed for (HelloWorldFunction), running incremental build                                                           
Building codeuri: /var/home/user/hello-world-tutorial/hello_world runtime: python3.12 metadata: {} architecture: x86_64 
functions: HelloWorldFunction                                                                                                         
 Running PythonPipBuilder:CopySource                                                                                                  
 Running PythonPipBuilder:CopySource                                                                                                  

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2024-08-01 13:57:06,971 | Config file location: /var/home/user/hello-world-tutorial/samconfig.toml                      
2024-08-01 13:57:06,973 | Loading configuration values from [default.['local', 'start-api'].parameters] (env.command_name.section) in 
config file at '/var/home/user/hello-world-tutorial/samconfig.toml'...                                                  
2024-08-01 13:57:06,974 | Configuration values successfully loaded.                                                                   
2024-08-01 13:57:06,975 | Configuration values are: {'stack_name': 'hello-world-tutorial', 'warm_containers': 'EAGER'}                
2024-08-01 13:57:06,978 | Using SAM Template at /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml        
2024-08-01 13:57:07,041 | Using config file: samconfig.toml, config environment: default                                              
2024-08-01 13:57:07,042 | Expand command line arguments to:                                                                           
2024-08-01 13:57:07,042 | --template_file=/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml              
--host=127.0.0.1 --port=3000 --static_dir=public --layer_cache_basedir=/home/user/.aws-sam/layers-pkg --warm_containers=EAGER   
--container_host=localhost --container_host_interface=127.0.0.1                                                                       
2024-08-01 13:57:07,097 | local start-api command is called                                                                           
2024-08-01 13:57:08,114 | No Parameters detected in the template                                                                      
2024-08-01 13:57:08,159 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction
is HelloWorldFunction                                                                                                                 
2024-08-01 13:57:08,160 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the      
resource logical id as the resource id                                                                                                
2024-08-01 13:57:08,162 | 0 stacks found in the template                                                                              
2024-08-01 13:57:08,163 | No Parameters detected in the template                                                                      
2024-08-01 13:57:08,185 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction
is HelloWorldFunction                                                                                                                 
2024-08-01 13:57:08,186 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the      
resource logical id as the resource id                                                                                                
2024-08-01 13:57:08,187 | 2 resources found in the stack                                                                              
2024-08-01 13:57:08,188 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'                   
2024-08-01 13:57:08,189 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                                   
/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml                                                        
2024-08-01 13:57:08,191 | watch resource /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml               
2024-08-01 13:57:08,192 | Create Observer for resource /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml 
with recursive True                                                                                                                   
2024-08-01 13:57:08,193 | watch resource /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml's parent      
/var/home/user/hello-world-tutorial/.aws-sam/build                                                                      
2024-08-01 13:57:08,194 | Create Observer for resource /var/home/user/hello-world-tutorial/.aws-sam/build with recursive
False                                                                                                                                 
2024-08-01 13:57:08,200 | Initializing the lambda functions containers.                                                               
2024-08-01 13:57:08,202 | Async execution started                                                                                     
2024-08-01 13:57:08,203 | Invoking function functools.partial(<function                                                               
InvokeContext._initialize_all_functions_containers.<locals>.initialize_function_container at 0x7f23bff07920>,                         
Function(function_id='HelloWorldFunction', name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.12',        
memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None,                             
codeuri='/var/home/user/hello-world-tutorial/.aws-sam/build/HelloWorldFunction', environment=None, rolearn=None,        
layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId':                       
'ServerlessRestApi'}}}, metadata={'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None,                  
architectures=['x86_64'], function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP
function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))                     
2024-08-01 13:57:08,208 | Waiting for async results 

Expected result:

Building image..........................
2024-08-01 13:32:44 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000
 * 2024-08-01 13:32:44 Press CTRL+C to quit

Expected Lambda to initialise and be available.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Fedora 40 Kinoite
{
"version": "1.120.0",
"system": {
"python": "3.11.8",
"os": "Linux-6.9.10-200.fc40.x86_64-x86_64-with-glibc2.39"
},
"additional_dependencies": {
"docker_engine": "5.1.2",
"aws_cdk": "Not available",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}

[hnnasit]

Hi @AidanOB, thanks for reporting the issue and providing details. I will try to reproduce the issue.

[hnnasit]

Can you check if sam local invoke is able to initialize the containers for you and invoke the lambdas?

[AidanOB]

Unfortunately the containers don't initialise either. Regular output is just no response on the terminal.

With the -debug flag set:

2024-08-06 10:49:37,750 | Config file location: /var/home/user/hello-world-tutorial/samconfig.toml                   
2024-08-06 10:49:37,753 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in 
config file at '/var/home/user/hello-world-tutorial/samconfig.toml'...                                               
2024-08-06 10:49:37,754 | Configuration values successfully loaded.                                                                
2024-08-06 10:49:37,754 | Configuration values are: {'stack_name': 'hello-world-tutorial'}                                         
2024-08-06 10:49:37,757 | Using SAM Template at /var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml     
2024-08-06 10:49:37,834 | Using config file: samconfig.toml, config environment: default                                           
2024-08-06 10:49:37,835 | Expand command line arguments to:                                                                        
2024-08-06 10:49:37,835 | --template_file=/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml --no_event
--layer_cache_basedir=/home/user/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1         
2024-08-06 10:49:37,836 | local invoke command is called                                                                           
2024-08-06 10:49:39,126 | No Parameters detected in the template                                                                   
2024-08-06 10:49:39,168 | Sam customer defined id is more priority than other IDs. Customer defined id for resource                
HelloWorldFunction is HelloWorldFunction                                                                                           
2024-08-06 10:49:39,170 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the   
resource logical id as the resource id                                                                                             
2024-08-06 10:49:39,173 | 0 stacks found in the template                                                                           
2024-08-06 10:49:39,174 | No Parameters detected in the template                                                                   
2024-08-06 10:49:39,204 | Sam customer defined id is more priority than other IDs. Customer defined id for resource                
HelloWorldFunction is HelloWorldFunction                                                                                           
2024-08-06 10:49:39,206 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the   
resource logical id as the resource id                                                                                             
2024-08-06 10:49:39,208 | 2 resources found in the stack                                                                           
2024-08-06 10:49:39,209 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'                
2024-08-06 10:49:39,211 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                                
/var/home/user/hello-world-tutorial/.aws-sam/build/template.yaml

[AidanOB]

Further behaviour I have discovered, but intermittently (so unable to reliably reproduce) is that the lambdas will initialise if I do other things with toolbox/podman. Sometimes making or entering a new toolbox will trigger the initialisation of the lambdas. Or making another container. But this has only happened once or twice and not exactly sure what I did at the time that caused the lambdas in the other container to suddenly initialise.

[gergo-dryrun]

I'm experiencing something similar.
The sam commands (either sam local start-api or sam local invoke) simply hang indefinitely, until I would run something like podman run hello-world in a separate shell.
Somehow this allows the sam process to continue and everything works as expected.

Environment details:
podman version 5.2.0
macOS Sonoma Version 14.5
SAM CLI, version 1.121.0

[jorupp]

Thanks @gergo-dryrun - running podman run hello-world in another shell worked!

[chiting]

Attempt to fix this in #7760

[Conrad2134]

Attempt to fix this in #7760

Thank you! The changes there fixed the issue for me. Looking forward to seeing that released!

[gminova]

@gergo-dryrun You sir deserve more likes! Thank you!!! Now only if we know why your solution works would be even better 😄 🚀

[chiting]

@gminova It's because the event SAM cli is reading from blocks until there's an event of type image triggered, which happens when you run a pod or pull an image.

[gminova]

@chiting Thank you. Yes, I saw the reason described in your PR, hopefully you can get your fix merged, however I wonder why sam is working seamlessly with docker and does not have the same problem whereas with podman this issue happens. Perhaps there could be some fix podman could do instead of sam?

[chiting]

@gminova I agree with you, there seem to be some differences around events and it would be good for podman to close the gap with how docker behaves.

Regardless of the above, I still believe SAM cli shouldn't use blocking calls in the main thread (unless it's by design, which I could be missing). The fact it works with docker seems to be server side implementation specific and I think the tool shouldn't rely on it to work.

[gminova]

@chiting Hey, I was able to fix my issue locally by removing my machine and init it again and ensuring the podman info is using version 5.3 where they have a fix, please see issue and discussion here: containers/podman#24839 (comment)

I think this issue can be closed @AidanOB