Bug: SAM does not write Layer build results to build.toml unless its table already exists

[davidjb]

Description:

Starting with an empty environment (e.g. no .aws-sam directory or no .aws-sam/build.toml file), attempting to build a layer by itself:

sam build --cached --use-container MyLayer

does not result in the layer's build results being written to .aws-sam/build.toml. In fact, this file does not even get created.

This means that subsequent calls to the same build command will not use the cache, despite cached files being created in .aws-sam/cache and that a problem such as #5507 cannot be worked around.

Edit: currently, a Layer build will only update its entry within .aws-sam/build.toml (e.g. SAM won't create this file just for building a Layer), essentially meaning that in order for a Layer to be cached, it must first at least have been attempted to be cached. A workaround for this situation is to force the creation of .aws-sam/build.toml, building in whatever way is necessary to generate the Layer's entry in this file (or otherwise manually creating the TOML would work too). In my case, it was quickest to just do:

sam build --cached MyFunction                 # this fails to build MyLayer because of issue #5507

# now, .aws-build/build.toml exists so we can carry on
sam build --cached --use-container MyLayer    # this caches, updating build.toml
sam build --cached MyFunction                 # this succeeds now

Steps to reproduce:

1. Remove any .aws-sam directory or .aws-sam/build.toml file
2. Define a Layer and Function such as this in template.yaml:

   MyLayer:
     Type: AWS::Serverless::LayerVersion
     Properties:
       ContentUri: layers/mylayer/
       CompatibleRuntimes:
         - provided.al2
     Metadata:
       BuildMethod: makefile
   

MyFunction:
Type: AWS::Serverless::Function
Metadata:
BuildMethod: go1.x
Properties:
CodeUri: functions/myfunction
Handler: bootstrap
Runtime: provided.al2
Layers:
- !Ref MyLayer

```make
build-MyLayer:
     mkdir "$(ARTIFACTS_DIR)/bin"
     echo 'dummy' > "$(ARTIFACTS_DIR)/bin/dumy"

3. Build that layer with: sam build --cached --use-container MyLayer
4. Observe that no .aws-sam/build.toml file is created.
5. Try and re-run sam build command and layer will not have been cached.

Observed result:

$ sam build --debug --cached --use-container MyLayer
2023-07-21 04:44:17,571 | No config file found in this directory.
2023-07-21 04:44:17,574 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/home/circleci/project2/samconfig.toml'
2023-07-21 04:44:17,575 | Config file location: /home/circleci/project2/samconfig.toml
2023-07-21 04:44:17,576 | Config file '/home/circleci/project2/samconfig.toml' does not exist
2023-07-21 04:44:17,578 | Using SAM Template at /home/circleci/project2/template.yml
2023-07-21 04:44:17,593 | Using config file: samconfig.toml, config environment: default
2023-07-21 04:44:17,594 | Expand command line arguments to:
2023-07-21 04:44:17,594 | --template_file=/home/circleci/project2/template.yml --cached --use_container --resource_logical_id=MyLayer --mount_with=READ --build_dir=.aws-sam/build
--cache_dir=.aws-sam/cache
2023-07-21 04:44:17,633 | 'build' command is called
2023-07-21 04:44:17,634 | Starting Build use cache
2023-07-21 04:44:17,635 | Starting Build inside a container
2023-07-21 04:44:17,637 | No Parameters detected in the template
2023-07-21 04:44:17,653 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:17,654 | 0 stacks found in the template
2023-07-21 04:44:17,654 | No Parameters detected in the template
2023-07-21 04:44:17,668 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:17,669 | 1 resources found in the stack
2023-07-21 04:44:17,669 | --base-dir is not presented, adjusting uri layers/mylayer/ relative to /home/circleci/project2/template.yml
2023-07-21 04:44:17,672 | 1 resources found in the stack
2023-07-21 04:44:17,674 | Instantiating build definitions
2023-07-21 04:44:17,674 | No previous build graph found, generating new one
2023-07-21 04:44:17,675 | Unique Layer build definition found, adding as new (Layer Build Definition: LayerBuildDefinition(MyLayer, /home/circleci/project2/layers/mylayer, ,
ef3c3c4f-ca7b-4153-9bbd-6e33b801fbd7, makefile, ['provided.al2'], x86_64, {}), Layer: <samcli.lib.providers.provider.LayerVersion object at 0x7f4979f77fd0>)
2023-07-21 04:44:17,676 | Running cached build for runtime makefile for following resources (<bound method LayerBuildDefinition.get_resource_full_paths of
<samcli.lib.build.build_graph.LayerBuildDefinition object at 0x7f4979e08220>>)
2023-07-21 04:44:17,678 | Cache is invalid, running build and copying resources for following layers (MyLayer)
2023-07-21 04:44:17,679 | Building layer 'MyLayer'
2023-07-21 04:44:17,680 | For container layer build, first compatible runtime is chosen as build target for container.

Fetching public.ecr.aws/sam/build-provided.al2:latest-x86_64 Docker container image......
2023-07-21 04:44:17,788 | Mounting /home/circleci/project2/layers/mylayer as /tmp/samcli/source:ro,delegated, inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Running workflow 'CustomMakeBuilder'
Running CustomMakeBuilder:CopySource
Copying source file (/tmp/samcli/source/Makefile) to destination (/tmp/samcli/scratch/Makefile)
CustomMakeBuilder:CopySource succeeded
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /tmp/samcli/artifacts
executing Make: ['make', '--makefile', '/tmp/samcli/source/Makefile', 'build-MyLayer']
mkdir "/tmp/samcli/artifacts/bin"
echo 'dummy' > "/tmp/samcli/artifacts/bin/dumy"
CustomMakeBuilder:MakeBuild succeeded
2023-07-21 04:44:18,104 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
2023-07-21 04:44:18,105 | Build inside container was successful. Copying artifacts from container to host
2023-07-21 04:44:18,280 | Copying from container: /tmp/samcli/artifacts/. -> /home/circleci/project2/.aws-sam/build/MyLayer
2023-07-21 04:44:18,325 | Build inside container succeeded
2023-07-21 04:44:18,326 | Instantiating build definitions
2023-07-21 04:44:18,327 | No previous build graph found, generating new one
2023-07-21 04:44:18,328 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:18,329 | 1 resources found in the stack

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2023-07-21 04:44:18,330 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-21 04:44:18,332 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-21 04:44:18,332 | Unable to find Click Context for getting session_id.

$ ls .aws-sam/build.toml                                                                                               ls: .aws-sam/build.toml: No such file or directory

Expected result:

Layer to be cached and recorded, available for subsequent runs of sam build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.93.0",
  "system": {
    "python": "3.10.6",
    "os": "Linux-5.19.0-1022-aws-x86_64-with-glibc2.35"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.24",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[mildaniel]

Thanks for reporting. I was able to reproduce this and we welcome contributions if you'd like to help fix this issue!

[moelasmar]

@davidjb , the steps you mentioned as a workaround does not work on my side. My findings is SAM CLI does not write to build.toml file even if exist in case if SAM CLI was invoked to build specific resources:

aws-sam-cli/samcli/lib/build/app_builder.py

Line 264 in 46c374f

build_graph.clean_redundant_definitions_and_update(not self._is_building_specific_resource)

aws-sam-cli/samcli/lib/build/build_graph.py

Lines 320 to 321 in d841801

	if persist:
	self._atomic_write()

aws-sam-cli/samcli/lib/build/build_graph.py

Lines 442 to 479 in d841801

	def _write(self) -> None:
	"""
	Writes build definition details into build.toml file, which would be used by the next build.
	build.toml file will contain the same information as build graph,
	function details will only be preserved as function names
	layer details will only be preserved as layer names
	"""
	# convert build definition list into toml table
	function_build_definitions_table = tomlkit.table()
	for function_build_definition in self._function_build_definitions:
	build_definition_as_table = _function_build_definition_to_toml_table(function_build_definition)
	function_build_definitions_table.add(function_build_definition.uuid, build_definition_as_table)
	layer_build_definitions_table = tomlkit.table()
	for layer_build_definition in self._layer_build_definitions:
	build_definition_as_table = _layer_build_definition_to_toml_table(layer_build_definition)
	layer_build_definitions_table.add(layer_build_definition.uuid, build_definition_as_table)
	# create toml document and add build definitions
	document = tomlkit.document()
	document.add(tomlkit.comment("This file is auto generated by SAM CLI build command"))
	# we need to cast `Table` to `Item` because of tomlkit#135.
	document.add(BuildGraph.FUNCTION_BUILD_DEFINITIONS, cast(tomlkit.items.Item, function_build_definitions_table))
	document.add(BuildGraph.LAYER_BUILD_DEFINITIONS, cast(tomlkit.items.Item, layer_build_definitions_table))
	if not self._filepath.exists():
	open(self._filepath, "a+").close() # pylint: disable=consider-using-with
	self._filepath.write_text(tomlkit.dumps(document))
	def _atomic_write(self) -> None:
	"""
	Performs the _write() method with a global lock acquired
	It makes sure no other thread accesses build.toml when a write is happening
	"""
	with BuildGraph.__toml_lock:
	self._write()

I will check with the team if this was done on purpose.

[moelasmar]

I checked this issue with the team, and we confirmed that SAM CLI does not create the build.toml file when ASM CLI build specific resources, because we do not create the build graph in this case, and so we will not be able to create the build.toml file.

The work around for this issue now is to first run sam build --cached which will create the build.toml file, and then the subsequent sam build --cached <<specific resource>> commands executions will check the cached build of this resource, and skipped it if it is already cached.

[davidjb]

@moelasmar So the issue is trying to reach a point where a sam build --cached can succeed in the first place, especially given other issues such as #5280. More specifically, a situation where certain resources need to be built in containers and others cannot currently be built in containers, meaning it's effectively impossible without hacks and workarounds for certain setups.

It may be possible to change configurations and hack things up until sam build yields an entry in build.toml, but I'm thinking that a more reliable workaround is to generate a fake resource entry into build.toml, such as:

[layer_build_definitions.00000000-0000-0000-0000-000000000000]
layer_name = "MyLayer"
codeuri = "/path/to/layers/mylayer"
build_method = "makefile"
compatible_runtimes = ["provided.al2"]
architecture = "arm64"
source_hash = "fake"
manifest_hash = ""
layer = "MyLayer"

Once this exists, then running sam build --cached <<specific resource>> updates source_hash correctly after that. Provided the UUID is any fake value, this appears to work.

Could a similar strategy be added into sam itself? For a LayerVersion in particular, there are no dependencies or build graphs, just that resource itself, so outputting the layer's config alone would be sufficient to solve this problem.