claude
4e09f27e4d
The poll is the fallback, not the mechanism. It is 30s on staging but 300s on production, so a green run could sit five minutes ahead of the deploy it claimed to have made -- the smoke job was absorbing that wait. Copied from easyDent verbatim, including the soft failure: an unset secret or an unreachable API logs and falls back to the poll rather than failing the build. That matters right now because production has no Watchtower HTTP API yet, so the production URL will do nothing until the infra stack there is updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WpTYCBLH58XzrM7n3xPJ5N
205 lines
8.6 KiB
YAML
205 lines
8.6 KiB
YAML
name: Build and Push Docker Images
|
|
|
|
# Branch-driven deploys — no yaml edits to switch environment:
|
|
# merge into `staging` -> tag :staging (staging Watchtower deploys)
|
|
# merge into `production` -> tag :production (production Watchtower deploys)
|
|
# `main` is the day-to-day work branch and deploys nothing.
|
|
on:
|
|
push:
|
|
branches:
|
|
- staging
|
|
- production
|
|
|
|
env:
|
|
GIT_HOST: docker-git.easysoft.ro
|
|
REGISTRY_HOST: registry.easysoft.ro
|
|
DOCKER_BUILDKIT: "1"
|
|
API_IMAGE: apps/myai-api
|
|
CV_MATCHER_API_IMAGE: apps/myai-cv-matcher-api
|
|
RAG_API_IMAGE: apps/myai-rag-api
|
|
EMAIL_API_IMAGE: apps/myai-email-api
|
|
WEB_IMAGE: apps/myai-web
|
|
CV_CLEANUP_JOB_IMAGE: apps/myai-cv-cleanup-job
|
|
CV_SEARCH_JOB_IMAGE: apps/myai-cv-search-job
|
|
PAGE_FETCHER_API_IMAGE: apps/myai-page-fetcher-api
|
|
IMAGE_TAG: ${{ github.ref_name }} # branch name == image tag (staging | production)
|
|
WEB_PORT: "5140" # host port the web container is published on
|
|
|
|
jobs:
|
|
build:
|
|
runs-on: host
|
|
|
|
steps:
|
|
- name: Checkout the pushed commit
|
|
env:
|
|
TOKEN: ${{ secrets.REPO_TOKEN }}
|
|
run: |
|
|
git clone "http://gelu:${TOKEN}@${GIT_HOST}:3000/${GITHUB_REPOSITORY}.git" .
|
|
git checkout "${{ github.sha }}"
|
|
|
|
- name: Login to registry
|
|
run: |
|
|
echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login "${REGISTRY_HOST}" \
|
|
-u "${{ secrets.REGISTRY_USER }}" \
|
|
--password-stdin
|
|
|
|
- name: Build API image
|
|
run: |
|
|
docker build -f Apis/api/Dockerfile -t "${REGISTRY_HOST}/${API_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Build CV Matcher API image
|
|
run: |
|
|
docker build -f Apis/cv-matcher-api/Dockerfile -t "${REGISTRY_HOST}/${CV_MATCHER_API_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Build RAG API image
|
|
run: |
|
|
docker build -f Apis/rag-api/Dockerfile -t "${REGISTRY_HOST}/${RAG_API_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Build Email API image
|
|
run: |
|
|
docker build -f Apis/email-api/Dockerfile -t "${REGISTRY_HOST}/${EMAIL_API_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Build Web image
|
|
run: |
|
|
docker build --build-arg GIT_SHA="${{ github.sha }}" \
|
|
-f web/Dockerfile -t "${REGISTRY_HOST}/${WEB_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Build CV cleanup job image
|
|
run: |
|
|
docker build -f Jobs/cv-cleanup-job/Dockerfile -t "${REGISTRY_HOST}/${CV_CLEANUP_JOB_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Build CV search job image
|
|
run: |
|
|
docker build -f Jobs/cv-search-job/Dockerfile -t "${REGISTRY_HOST}/${CV_SEARCH_JOB_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Build Page Fetcher API image
|
|
run: |
|
|
docker build -f Apis/page-fetcher-api/Dockerfile -t "${REGISTRY_HOST}/${PAGE_FETCHER_API_IMAGE}:${IMAGE_TAG}" .
|
|
|
|
- name: Push API image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${API_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Push CV Matcher API image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${CV_MATCHER_API_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Push RAG API image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${RAG_API_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Push Email API image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${EMAIL_API_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Push Web image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${WEB_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Push CV cleanup job image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${CV_CLEANUP_JOB_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Push CV search job image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${CV_SEARCH_JOB_IMAGE}:${IMAGE_TAG}"
|
|
|
|
- name: Push Page Fetcher API image
|
|
run: |
|
|
docker push "${REGISTRY_HOST}/${PAGE_FETCHER_API_IMAGE}:${IMAGE_TAG}"
|
|
|
|
# Watchtower's poll is the fallback, not the mechanism: 30s on staging but 300s on
|
|
# production, so without this a green run can sit five minutes ahead of the deploy it
|
|
# claims to have made. Copied from easyDent, including the soft failure -- an unset
|
|
# secret or an unreachable API must degrade to the poll, never fail the build.
|
|
- name: Trigger Watchtower redeploy
|
|
env:
|
|
URL_STAGING: ${{ secrets.WATCHTOWER_URL_STAGING }}
|
|
URL_PRODUCTION: ${{ secrets.WATCHTOWER_URL_PRODUCTION }}
|
|
TOKEN: ${{ secrets.WATCHTOWER_TOKEN }}
|
|
run: |
|
|
if [ "${IMAGE_TAG}" = "production" ]; then URL="${URL_PRODUCTION}"; else URL="${URL_STAGING}"; fi
|
|
if [ -n "${URL}" ] && [ -n "${TOKEN}" ]; then
|
|
echo "Triggering Watchtower at ${URL}"
|
|
curl -sf -m 30 -H "Authorization: Bearer ${TOKEN}" "${URL}" && echo " -> redeploy triggered" \
|
|
|| echo " -> trigger failed; Watchtower will still pick it up on the next poll"
|
|
else
|
|
echo "Watchtower push-trigger not configured (WATCHTOWER_* secrets unset); relying on the poll interval."
|
|
fi
|
|
|
|
- name: Reclaim disk space (keep recent build cache)
|
|
if: always()
|
|
run: |
|
|
docker image prune -f # dangling only (keep base images)
|
|
# Building and pushing an image proves nothing about what the host is running.
|
|
# Watchtower pulls asynchronously, and for a month it was pulling a tag nobody
|
|
# intended -- with every run green, because no step ever asked the deployed site
|
|
# what it was serving. This job asks.
|
|
#
|
|
# It polls the deploy host directly on the LAN rather than the public hostname:
|
|
# the runner sits inside the network, only easysoft.ro has a staging equivalent in
|
|
# public DNS, and going direct also takes Caddy and any CDN out of the answer.
|
|
smoke:
|
|
runs-on: host
|
|
needs: build
|
|
steps:
|
|
- name: Wait for the deploy host to serve this commit
|
|
run: |
|
|
case "${{ github.ref_name }}" in
|
|
staging) HOST=192.168.1.111 ;;
|
|
production) HOST=192.168.1.101 ;;
|
|
*) echo "::error::No deploy host mapped for '${{ github.ref_name }}'."; exit 1 ;;
|
|
esac
|
|
|
|
URL="http://${HOST}:${WEB_PORT}/version.json"
|
|
echo "Polling ${URL} for ${{ github.sha }}"
|
|
|
|
# 10 minutes: Watchtower's poke is fire-and-forget with a 30s fallback poll,
|
|
# and the container still has to start.
|
|
# ⚠️ Steps run under `bash -e -o pipefail`, so a polling loop has to be written
|
|
# defensively: the FIRST miss is the normal case, not an error.
|
|
# - `curl -sf | sed` fails the whole pipeline under pipefail while the old
|
|
# container is still up (404/connection refused), so `|| GOT=""` is required
|
|
# - `[ test ] && { ... }` returns non-zero when the test fails, which under -e
|
|
# aborts the step. Use `if`.
|
|
# Getting both wrong made the first run of this job fail in 20 seconds.
|
|
DEADLINE=$(( $(date +%s) + 600 ))
|
|
while :; do
|
|
GOT=$(curl -sf -m 15 "${URL}" 2>/dev/null | sed -n 's/.*"version":"\([^"]*\)".*/\1/p') || GOT=""
|
|
|
|
if [ "${GOT}" = "${{ github.sha }}" ]; then
|
|
echo "Serving ${GOT}."
|
|
break
|
|
fi
|
|
|
|
if [ "$(date +%s)" -ge "${DEADLINE}" ]; then
|
|
echo "::error::Timed out after 10m. ${HOST} is serving '${GOT:-nothing}', wanted ${{ github.sha }}."
|
|
echo "Either Watchtower never pulled the new image, the container failed to"
|
|
echo "start, or the stack's IMAGE_TAG does not match this branch."
|
|
exit 1
|
|
fi
|
|
echo " still serving '${GOT:-nothing}' ..."
|
|
sleep 15
|
|
done
|
|
|
|
- name: Check the site actually answers
|
|
run: |
|
|
case "${{ github.ref_name }}" in
|
|
staging) HOST=192.168.1.111 ;;
|
|
production) HOST=192.168.1.101 ;;
|
|
esac
|
|
# `-L` follows redirects and we assert on the FINAL code, because a 302 from `/`
|
|
# is a healthy answer for a site running in UnderConstruction mode -- it means the
|
|
# app is up and routing. Asserting a bare 200 failed jecreativ.ro's first
|
|
# production deploy for doing exactly what it was configured to do.
|
|
#
|
|
# `|| CODE=000` for the same reason as above: curl exiting non-zero on a
|
|
# connection failure must produce a reportable code, not kill the step before
|
|
# it can say what went wrong. (`-s` without `-f` already tolerates 4xx/5xx.)
|
|
CODE=$(curl -sL -o /dev/null -w '%{http_code}' -m 20 "http://${HOST}:${WEB_PORT}/") || CODE=000
|
|
if [ "${CODE}" != "200" ]; then
|
|
echo "::error::Home page returned ${CODE}."
|
|
exit 1
|
|
fi
|
|
echo "Home page 200."
|