"Supporting" buildkit/buildx

[flowchartsman]

Overview

As buildkit features like parallel builds, secrets and cache mounts become more popular, container testing frameworks like testcontainers-go should work to support them. This is something that is gaining traction for the original implementation, but has not come to pass yet.

There was an [issue](for this) with testcontainers-go once upon a time, but it evolved into the regular docker API build implementation and the original idea was dropped. This discussion seeks to resume that effort, since it would be pretty great for testcontainers-go to support buildkit, no matter how provisional, and I believe it could conceivably be staged to increase support over time.

The problems facing implementation are that buildkit does not have an HTTP api that I'm aware of. It's all over a GRPC communication with the daemon over unix socket. Compounding this, as far as I've been able to determine, Docker Desktop has their own closed GRPC API separate from the original implementation. Poking at the client on my macbooks has revealed it talking to /Users/<me>/.docker/run/docker-cli-api.sock, but it will not answer the standard moby tools like buildctl.

This doesn't have to be the end of the world, though, as most, if not all, features can be hooked via interaction with the CLI, and this should be more than enough for the initial option to simply build and tag images with buildkit instead of plain Docker, falling back to the API for the rest of the container lifecycle. Obviously, more advanced features would be better, but I think this can be done iteratively.

Phase 1

Implement an initial (and optional) BuildkitProvider type as an interface

type BuildkitProvider interface {
	CreateContainer(context.Context, ContainerRequest) (Container, error) 
}

And provide only one, implementation:

type CLIBuildkitProvider struct{}
func NewCLIBuildkitProvider() *CLIBuildkitProvider{}

This can remain opaque until such time as more options are needed and researched (such as those provided by buildx), so long as they are germane to testcontainers usage.

CLIBuildkitProvider could then be attached similarly to other options:

DockerProviderOptions struct {
        defaultBridgeNetworkName string
        buildkitProvider BuildkitProvider
        *GenericProviderOptions
}
//...
func WithBuildkitProvider(buildkitProvider BuildkitProvider) BuildkitOption {
	return BuildkitOption{
		buildkitProvider: buildkitProvider,
	}
}
//...
type BuildkitProviderOption struct {
	buildkitProvider BuildkitProvider
}

func (o BuildkitProviderOption) ApplyDockerTo(opts *DockerProviderOptions) {
	opts.buildkitProvider = o.buildkitProvider
}

If set in DockerProviderOptions, the BuildkitProvider would be delegated to to build the image. In the case of the initial CLIBuildkitProvider, this would involve calling os.Exec to run docker build with DOCKER_BUILDKIT=1 set in the environment. The CLI will also be used to label the image so that it can be claimed by the reaper.

Phase 2

Later work could include providing both

type OnDemandBuildkitProvider struct{
        //options such as retained cache location
}

as well as

type RemoteBuildkitProvider struct {
        //addr for GRPC connection
}

Taking advantage of the moby/buildkit client library to interface with the daemon over GRPC

Phase 3

• Alternate architectures via buildx, if this is somehow necessary.
• Extension to other container providers, if applicable.
• Compatibility with Docker Desktop, if such a thing is even possible.

Thoughts?

[flowchartsman]

Also, if anyone sees a need for things such as Buildah, Buildpacks, img, Kaniko, etc then the interface could be abstracted to something like

type ContainerBuilder interface {
	CreateContainer(context.Context, ContainerRequest) (Container, error) 
}

Instead.

[kiview]

I think this is a significant opportunity to create an abstraction layer. In Testcontainers for Java, we currently have a similar situation with regard to Jib integration:
https://github.com/testcontainers/testcontainers-java/pulls?q=jib

In the Testcontainers for Java architecture, this is currently planned to be achieved through the DockerClient abstraction:
https://github.com/testcontainers/testcontainers-java/pull/5623/files#diff-386dcb32939c2014f17035d347ee4c5197dcc21903d9f8cbf3008a908e47a1c3

[flowchartsman]

So, after looking into Jib, I'm further convinced that a ContainerBuilder abstraction is the right call, since this seems to be a particular niche at the moment.

Is there any reason why the Java library has opted for an impl of DockerClient as opposed to something like ContainerBuilder? From the Jib Site it definitely looks like it isn't tied to Docker, in fact, Jib explicitly mentions that it builds "without a Docker daemon".

I only ask because multi-language projects like this tend to hew (at least roughly) to the abstractions of the other so that polyglots can feel more at home with an "ecosystem". Is that the case here? Because, if not, it's a pretty strong case for ContainerBuilder

[flowchartsman]

Also: good news, it looks like Docker Desktop might respond to a client instance after all. I was able to create a container and run it with -v/Users/<me>/.docker/run/docker.sock.raw:/var/run/docker.sock and I installed the buildkit plugin and it responded to docker buildx du at least.. So that's a good sign. In theory, the client commands to issue buildkit image requests can be reverse-engineered from that.

[mdelapenya]

For some reason, I've not seen this discussion before. Sorry.

I'll go deep into it next week to understand it better.

[gheorghestrimtu]

Any updates on this work?
I am particularly interested in being able to use additional contexts, or build contexts for different stages of the build. Which seems to be supported only by BuildKit.