TIL How to build and host a container on GCP VM instance

2025-01-18

I had a project that required updating a legacy web app (built with dash) hosted on GCP Cloud Run. We needed to update replace the authentication layer by placing the app behind a VPN, which requires a static external IP address.

Unfortunately, static IP addresses aren’t simply assigned to Cloud Run deployments – a more involved setup using a load balancer is required for this. Since this was a relatively small app that isn’t expected to have much traffic, I elected to try take a shorter route to satisfy these requirements: host the image on a small Compute VM instance with a static IP.

There are a few variables I define that I’ll reference throughout the post:

PROJECT_ID=my-gcp-project
REGION=us-central1
ZONE=us-central1-a
REPOSITORY_NAME=my-legacy-web-app
CONTAINER_NAME=my-container

INSTANCE_NAME=my-web-app-instance
MACHINE_TYPE=e2-small
ADDRESS_NAME=my-address

REPO=${REGION}-docker.pkg.dev
COMMIT_SHA=$(git rev-parse --short HEAD)

CONTAINER_REPO=${REPO}/${PROJECT_ID}/${REPOSITORY_NAME}
CONTAINER_NAME_REMOTE=${CONTAINER_REPO}/${CONTAINER_NAME}

Firstly we need to build the docker image. To avoid platform issues with building things locally, I used gcloud builds submit to build and store an image in the Artifacts storage:

gcloud builds submit \
    --tag $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA)
gcloud container images add-tag \
    $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA) \
    $(CONTAINER_NAME_REMOTE):latest \
    --quiet

This kicks off a job that builds and pushes the Docker image from my local code repository to a private repo on in my GCP project. It will also assign the latest tag to the built image.

Among the public machine images you can use on VMs, there’s “container optimised systems” (COS), which come with minimal dependencies and come equipped with docker – seems like a perfect tool for the job!

Creating the static IP and the instance:

gcloud compute addresses create ${ADDRESS_NAME} --region=${REGION}
gcloud compute instances create ${INSTANCE_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --address=${ADDRESS_NAME} \
    --machine-type=${MACHINE_TYPE} \
    --image-project=cos-cloud \
    --image-family=cos-117-lts

So it should be as simple as ssh-ing to the instance, pulling and running the image right?

Unfortunately not. Attempting to do this will result in a “permission denied” error, despite the compute service account having all the correct permissions for pulling from artifact storage… So what gives?

In a nutshell, the docker installation on these images come equipped with credential helpers. If you have a look in the ~/.docker/config.json on the instance:

{
        "auths": {},
        "credHelpers": {
                "asia.gcr.io": "gcr",
                "eu.gcr.io": "gcr",
                "gcr.io": "gcr",
                "marketplace.gcr.io": "gcr",
                "us.gcr.io": "gcr"
        }
}

All we need is to add our private artifact repo to this list of cred helpers:

cat > ~/.docker/config.json << 'EOF'
{
        "auths": {},
        "credHelpers": {
                "asia.gcr.io": "gcr",
                "eu.gcr.io": "gcr",
                "gcr.io": "gcr",
                "marketplace.gcr.io": "gcr",
                "us.gcr.io": "gcr",
                "$REPO": "gcr"
        }
}
EOF

I’d like to put this into a startup script, so that I can deploy the latest docker image, or a particular commit. To do this, we can add environment variables to the instance creation, and read them in a provided startup script with

TAG=abcd1234
gcloud compute instances create ${INSTANCE_NAME} \
    --project=${PROJECT_ID} \
    --zone=${ZONE} \
    --address=${ADDRESS_NAME} \
    --machine-type=${MACHINE_TYPE} \
    --image-project=cos-cloud \
    --image-family=cos-117-lts \
    --metadata TAG=${TAG:-latest},REPO=${REPO} \
    --metadata-from-file startup-script=./startup.sh

Using the startup script:

#!/bin/bash

TAG=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/TAG" -H "Metadata-Flavor: Google")
echo "TAG=${TAG}"
export TAG

REPO=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/TAG" -H "Metadata-Flavor: Google")
echo "REPO=${REPO}"
export REPO

# Define the writable location for the Docker configuration
DOCKER_CONFIG_DIR="/home/$(whoami)/.docker"

# Create the Docker configuration directory
mkdir -p "$DOCKER_CONFIG_DIR"

# Write the config.json file
cat > "$DOCKER_CONFIG_DIR/config.json" << 'EOF'
{
        "auths": {},
        "credHelpers": {
                "asia.gcr.io": "gcr",
                "eu.gcr.io": "gcr",
                "gcr.io": "gcr",
                "marketplace.gcr.io": "gcr",
                "us.gcr.io": "gcr",
                "$REPO": "gcr"
        }
}
EOF

# Ensure proper permissions for the Docker configuration
chmod 600 "$DOCKER_CONFIG_DIR/config.json"

# Set the environment variable to let Docker know about the custom configuration
export DOCKER_CONFIG="$DOCKER_CONFIG_DIR"
echo "export DOCKER_CONFIG=\"${DOCKER_CONFIG_DIR}\"" >> /etc/profile

# Start Docker service (if not already running)
systemctl start docker

# Start image
docker run -d -p 8080:8080 "${REPO}/my-gcp-project/my-legacy-web-app/my-container:${TAG}"

Putting this altogether in a Makefile:

SHELL := /usr/bin/env bash
.SHELLFLAGS := -eu -o pipefail -c

CONTAINER_NAME = my-container

PROJECT_ID := my-gcp-project
REGION := us-central1
ZONE := us-central1-a
REPOSITORY_NAME := my-legacy-web-app

REPO := $(REGION)-docker.pkg.dev
COMMIT_SHA := $(shell git rev-parse --short HEAD)
INSTANCE_NAME = "my-instance"
MACHINE_TYPE := "e2-medium"
STARTUP_SCRIPT := ./startup.sh
ADDRESS_NAME := my-address

CONTAINER_REPO := $(REPO)/$(PROJECT_ID)/$(REPOSITORY_NAME)
CONTAINER_NAME_REMOTE := $(CONTAINER_REPO)/$(CONTAINER_NAME)

.PHONY: create-address list-tags build-remote-container deploy clean-instance

create-address:
    @if ! gcloud compute addresses describe $(ADDRESS_NAME) --region=$(REGION) > /dev/null 2>&1; then \
        echo "Creating static IP address: $(ADDRESS_NAME)"; \
        gcloud compute addresses create $(ADDRESS_NAME) --region=$(REGION); \
    else \
        echo "Static IP address $(ADDRESS_NAME) already exists."; \
    fi

build-remote-container: _check-clean
    @gcloud builds submit \
        --tag $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA)
    @gcloud container images add-tag \
        $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA) \
        $(CONTAINER_NAME_REMOTE):latest \
        --quiet

clean-instance:
    @if gcloud compute instances list --filter="name=$(INSTANCE_NAME)" --format="value(name)" | grep -q $(INSTANCE_NAME); then \
        echo "Instance '$(INSTANCE_NAME)' exists. Deleting..."; \
        gcloud compute instances delete $(INSTANCE_NAME) --zone=$(ZONE) --quiet; \
    else \
        echo "Instance '$(INSTANCE_NAME)' does not exist."; \
    fi

list-tags:
    @gcloud container images list-tags $(CONTAINER_NAME_REMOTE) \
        --filter 'tags:*'\
        --format="table(timestamp, digest, tags)"

tag ?= latest
# Deploy custom tag with `make deploy tag=<TAG>`
deploy: _check-image-exists clean-instance
    @echo "Creating instance '$(INSTANCE_NAME)' from tag '$(tag)'"; \
    gcloud compute instances create $(INSTANCE_NAME) \
        --project=$(PROJECT_ID) \
        --zone=$(ZONE) \
        --address=$(ADDRESS_NAME) \
        --machine-type=$(MACHINE_TYPE) \
        --image-project=cos-cloud \
        --image-family=cos-117-lts \
        --metadata TAG=$(tag),REPO=$(REPO) \
        --metadata-from-file startup-script=$(STARTUP_SCRIPT)


.PHONY: _check-image-exists _check-clean

_check-image-exists:
    @if gcloud container images list-tags $(CONTAINER_NAME_REMOTE) --filter="tags=$(tag)" --format="value(tags)" | grep -wq "$(tag)$$"; then \
        echo "Image with tag '$(tag)' exists in the repository."; \
    else \
        echo "Image with tag '$(tag)' does NOT exist."; \
        exit 1; \
    fi


_check-clean:
    @git diff --quiet || { echo "Error: Uncommitted changes found. Commit or stash changes before proceeding."; exit 1; }

Reply to this post by email ↪