TIL How to build and host a container on GCP VM instance
I had a project that required updating a legacy web app (built with dash) hosted on GCP Cloud Run. We needed to update replace the authentication layer by placing the app behind a VPN, which requires a static external IP address.
Unfortunately, static IP addresses aren’t simply assigned to Cloud Run deployments – a more involved setup using a load balancer is required for this. Since this was a relatively small app that isn’t expected to have much traffic, I elected to try take a shorter route to satisfy these requirements: host the image on a small Compute VM instance with a static IP.
There are a few variables I define that I’ll reference throughout the post:
1PROJECT_ID=my-gcp-project
2REGION=us-central1
3ZONE=us-central1-a
4REPOSITORY_NAME=my-legacy-web-app
5CONTAINER_NAME=my-container
6
7INSTANCE_NAME=my-web-app-instance
8MACHINE_TYPE=e2-small
9ADDRESS_NAME=my-address
10
11REPO=${REGION}-docker.pkg.dev
12COMMIT_SHA=$(git rev-parse --short HEAD)
13
14CONTAINER_REPO=${REPO}/${PROJECT_ID}/${REPOSITORY_NAME}
15CONTAINER_NAME_REMOTE=${CONTAINER_REPO}/${CONTAINER_NAME}Firstly we need to build the docker image. To avoid platform issues with building things locally,
I used gcloud builds submit to build and store an image in the Artifacts storage:
1gcloud builds submit \
2 --tag $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA)
3gcloud container images add-tag \
4 $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA) \
5 $(CONTAINER_NAME_REMOTE):latest \
6 --quietThis kicks off a job that builds and pushes the Docker image from my local code repository to a
private repo on in my GCP project. It will also assign the latest tag to the built image.
Among the public machine images you can use on VMs, there’s “container optimised systems” (COS), which come with minimal dependencies and come equipped with docker – seems like a perfect tool for the job!
Creating the static IP and the instance:
1gcloud compute addresses create ${ADDRESS_NAME} --region=${REGION}
2gcloud compute instances create ${INSTANCE_NAME} \
3 --project=${PROJECT_ID} \
4 --zone=${ZONE} \
5 --address=${ADDRESS_NAME} \
6 --machine-type=${MACHINE_TYPE} \
7 --image-project=cos-cloud \
8 --image-family=cos-117-ltsSo it should be as simple as ssh-ing to the instance, pulling and running the image right?
Unfortunately not. Attempting to do this will result in a “permission denied” error, despite the compute service account having all the correct permissions for pulling from artifact storage… So what gives?
In a nutshell, the docker installation on these images come equipped with credential helpers. If
you have a look in the ~/.docker/config.json on the instance:
1{
2 "auths": {},
3 "credHelpers": {
4 "asia.gcr.io": "gcr",
5 "eu.gcr.io": "gcr",
6 "gcr.io": "gcr",
7 "marketplace.gcr.io": "gcr",
8 "us.gcr.io": "gcr"
9 }
10}All we need is to add our private artifact repo to this list of cred helpers:
1cat > ~/.docker/config.json << 'EOF'
2{
3 "auths": {},
4 "credHelpers": {
5 "asia.gcr.io": "gcr",
6 "eu.gcr.io": "gcr",
7 "gcr.io": "gcr",
8 "marketplace.gcr.io": "gcr",
9 "us.gcr.io": "gcr",
10 "$REPO": "gcr"
11 }
12}
13EOFI’d like to put this into a startup script, so that I can deploy the latest docker image, or a particular commit. To do this, we can add environment variables to the instance creation, and read them in a provided startup script with
1TAG=abcd1234
2gcloud compute instances create ${INSTANCE_NAME} \
3 --project=${PROJECT_ID} \
4 --zone=${ZONE} \
5 --address=${ADDRESS_NAME} \
6 --machine-type=${MACHINE_TYPE} \
7 --image-project=cos-cloud \
8 --image-family=cos-117-lts \
9 --metadata TAG=${TAG:-latest},REPO=${REPO} \
10 --metadata-from-file startup-script=./startup.shUsing the startup script:
1#!/bin/bash
2
3TAG=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/TAG" -H "Metadata-Flavor: Google")
4echo "TAG=${TAG}"
5export TAG
6
7REPO=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/TAG" -H "Metadata-Flavor: Google")
8echo "REPO=${REPO}"
9export REPO
10
11# Define the writable location for the Docker configuration
12DOCKER_CONFIG_DIR="/home/$(whoami)/.docker"
13
14# Create the Docker configuration directory
15mkdir -p "$DOCKER_CONFIG_DIR"
16
17# Write the config.json file
18cat > "$DOCKER_CONFIG_DIR/config.json" << 'EOF'
19{
20 "auths": {},
21 "credHelpers": {
22 "asia.gcr.io": "gcr",
23 "eu.gcr.io": "gcr",
24 "gcr.io": "gcr",
25 "marketplace.gcr.io": "gcr",
26 "us.gcr.io": "gcr",
27 "$REPO": "gcr"
28 }
29}
30EOF
31
32# Ensure proper permissions for the Docker configuration
33chmod 600 "$DOCKER_CONFIG_DIR/config.json"
34
35# Set the environment variable to let Docker know about the custom configuration
36export DOCKER_CONFIG="$DOCKER_CONFIG_DIR"
37echo "export DOCKER_CONFIG=\"${DOCKER_CONFIG_DIR}\"" >> /etc/profile
38
39# Start Docker service (if not already running)
40systemctl start docker
41
42# Start image
43docker run -d -p 8080:8080 "${REPO}/my-gcp-project/my-legacy-web-app/my-container:${TAG}"Putting this altogether in a Makefile:
1SHELL := /usr/bin/env bash
2.SHELLFLAGS := -eu -o pipefail -c
3
4CONTAINER_NAME = my-container
5
6PROJECT_ID := my-gcp-project
7REGION := us-central1
8ZONE := us-central1-a
9REPOSITORY_NAME := my-legacy-web-app
10
11REPO := $(REGION)-docker.pkg.dev
12COMMIT_SHA := $(shell git rev-parse --short HEAD)
13INSTANCE_NAME = "my-instance"
14MACHINE_TYPE := "e2-medium"
15STARTUP_SCRIPT := ./startup.sh
16ADDRESS_NAME := my-address
17
18CONTAINER_REPO := $(REPO)/$(PROJECT_ID)/$(REPOSITORY_NAME)
19CONTAINER_NAME_REMOTE := $(CONTAINER_REPO)/$(CONTAINER_NAME)
20
21.PHONY: create-address list-tags build-remote-container deploy clean-instance
22
23create-address:
24 @if ! gcloud compute addresses describe $(ADDRESS_NAME) --region=$(REGION) > /dev/null 2>&1; then \
25 echo "Creating static IP address: $(ADDRESS_NAME)"; \
26 gcloud compute addresses create $(ADDRESS_NAME) --region=$(REGION); \
27 else \
28 echo "Static IP address $(ADDRESS_NAME) already exists."; \
29 fi
30
31build-remote-container: _check-clean
32 @gcloud builds submit \
33 --tag $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA)
34 @gcloud container images add-tag \
35 $(CONTAINER_NAME_REMOTE):$(COMMIT_SHA) \
36 $(CONTAINER_NAME_REMOTE):latest \
37 --quiet
38
39clean-instance:
40 @if gcloud compute instances list --filter="name=$(INSTANCE_NAME)" --format="value(name)" | grep -q $(INSTANCE_NAME); then \
41 echo "Instance '$(INSTANCE_NAME)' exists. Deleting..."; \
42 gcloud compute instances delete $(INSTANCE_NAME) --zone=$(ZONE) --quiet; \
43 else \
44 echo "Instance '$(INSTANCE_NAME)' does not exist."; \
45 fi
46
47list-tags:
48 @gcloud container images list-tags $(CONTAINER_NAME_REMOTE) \
49 --filter 'tags:*'\
50 --format="table(timestamp, digest, tags)"
51
52tag ?= latest
53# Deploy custom tag with `make deploy tag=<TAG>`
54deploy: _check-image-exists clean-instance
55 @echo "Creating instance '$(INSTANCE_NAME)' from tag '$(tag)'"; \
56 gcloud compute instances create $(INSTANCE_NAME) \
57 --project=$(PROJECT_ID) \
58 --zone=$(ZONE) \
59 --address=$(ADDRESS_NAME) \
60 --machine-type=$(MACHINE_TYPE) \
61 --image-project=cos-cloud \
62 --image-family=cos-117-lts \
63 --metadata TAG=$(tag),REPO=$(REPO) \
64 --metadata-from-file startup-script=$(STARTUP_SCRIPT)
65
66
67.PHONY: _check-image-exists _check-clean
68
69_check-image-exists:
70 @if gcloud container images list-tags $(CONTAINER_NAME_REMOTE) --filter="tags=$(tag)" --format="value(tags)" | grep -wq "$(tag)$$"; then \
71 echo "Image with tag '$(tag)' exists in the repository."; \
72 else \
73 echo "Image with tag '$(tag)' does NOT exist."; \
74 exit 1; \
75 fi
76
77
78_check-clean:
79 @git diff --quiet || { echo "Error: Uncommitted changes found. Commit or stash changes before proceeding."; exit 1; }