[Bug]: The submission_environment_dependencies.txt file does not get staged when running with Flink runner on Dataproc

[liferoad]

What happened?

In some cases, "submission_environment_dependencies.txt" might not be staged.

#32752 added a workaround to ignore the error for a missing artifact, but we should rootcause why it didn't get staged.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

Do we crash? Seems like we should just print sth:

beam/sdks/python/container/boot.go

Line 416 in 2049e6b

bufLogger.Printf(ctx, "couldn't fetch the submission environment dependencies: %v", err)

[liferoad]

Flink on Dataproc returns this:
failed to retrieve staged files: failed to retrieve /tmp/staged in 3 attempts: failed to retrieve chunk for /tmp/staged/submission_environment_dependencies.txt
And the job then failed.

[tvalentyn]

ok, then the problem concerns the area of materialization of staged artifacts - we have a file that is being added to a manifest, but then not available when we try to materialize it.

It should either not be staged (and not included in the manifest), or be available in the staging location.

[tvalentyn]

Workaround: supply --experiments=disable_logging_submission_environment

beam/sdks/python/apache_beam/runners/portability/stager.py

Line 378 in 4cf99ea

'disable_logging_submission_environment')):

[liferoad]

Thanks, @tvalentyn. Let me investigate this later when I have time.

[zendesk-kjaanson]

I also ran into this bug when trying to run Beam 2.59.0 using PortableRunner on Kubernetes with Apache Flink Operator. When looking into task manager pods then the submission_environment_dependencies.txt file at /tmp/staging seems to be there but completely empty. All task-managers spawned will simply sit idle after logging this error.

[liferoad]

You can disable this by using --experiments disable_logging_submission_environment