I’m trying to build multiarch docker images. I’d like to sand it down as much as possible. In a better world, that would mean just one simple command. It’s not.

Here’s the first error I got from my simple script:

podman build --platform linux/amd64 . -t "${TAG}-amd64"
podman build --platform linux/arm64/v8 . -t "${TAG}-arm64"
podman manifest create "$TAG" "${TAG}-amd64" "${TAG}-arm64"

Error: setting up to read manifest and configuration from "docker://account.dkr.ecr.us-east-1.amazonaws.com/image:tag": reading manifest docker://account.dkr.ecr.us-east-1.amazonaws.com/image:tag: manifest unknown: Requested image not found

This didn’t work, and it turned out the reason was quite simple if obtuse – podman manifest wants to build from the real repositories. As I hadn’t pushed those images, it couldn’t find them on the remote repository.

I spent some time searching for a solution to build images locally, then build them into a manifest, and then finally tag them. I found a couple of things that should work, but didn’t:

podman manifest add MANIFEST containers-storage:image:tag
reference "[overlay@/home/ted/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/image:tag" does not resolve to an image ID: identifier is not an image

I don’t know why this didn’t work. According to the docs on transports, containers-storage is the transport we can use to inspect local images. This is somewhat consistent in behavior:

podman build image:tag
podman tag 
podman inspect image:tag
...
podman  inspect containers-storage:repo/image:tag
...
podman inspect containers-storage:image:tag
Error: no such object: "containers-storage:image:tag
podman inspect containers-storage:localhost/image:tag
...

Containers-storage somewhat works, but you have to supply a hostname, which is “localhost” for otherwise unspecified images.

Another angle I tried is building both in a singular tag with the manifest flag. This seems like it should work.

podman build --platform linux/amd64,linux/arm64/v8 . --manifest image:tag

This actually worked – I didn’t realize it at first, but it built both architectures:

podman manifest inspect image:tag
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 2444,
            "digest": "sha256:ea95462b074c650e6c477f8bf88bcfa0b6a021de7c550e2faca25c7f833bdc5f",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 2444,
            "digest": "sha256:f1eb75a71b89b3655b845acd79076bc8d640d3db8fb0f24367748fb50b2e6001",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            }
        }
    ]
}

However, when I pushed my image, the wrong format was downloaded on my k8s nodes:

podman push image:tag

Containers:
  loadtest:
    Container ID:  containerd://5d157712c742aa63220c34eb2b5213b0cf580a50c5768406ff434910700a2638
    Image:         image:tag
    Image ID:      image:tag@sha256:d0345fbc0ec7c38fdcbedfb90e7b21986e2e9642856e7e2a62a0591d68d48f85

A significant amount of consternation later, I realized that because I was using podman push, the image was being resolved first, and then just the one architecture was pushed (but with tag for the whole . What I needed to do instead was podman manifest push, which pushed the whole manifest and all sub-images.