Release and Deployment

This page traces the exact path a change takes from a merged pull request to running on omegaup.com: the GitHub Actions workflows that gate and build it, the five Docker images we ship, the two long-lived branches (main and release) that stand in for our two environments, and the GitOps hand-off that finally rolls the new images onto the Kubernetes cluster. Everything here lives in .github/workflows/ and stuff/docker/ in the omegaup/omegaup repo — when in doubt, read the YAML, because that is the source of truth and this page is just the map.

Two branches are the two environments

We don't tag semantic versions of the frontend and hand them to an ops team. Instead, two branches in the repo are the deployment surface, and their whole job is to mirror what is currently live:

• main holds the latest changes the review team has approved. Every merged PR lands here, and every landing rebuilds and redeploys sandbox. Sandbox is deliberately a lap ahead of production so that a bad change gets caught by us on sandbox.omegaup.com before real users on omegaup.com ever see it — the wiki's phrasing is that sandbox "gives us a buffer in case of errors in the latest changes," and that buffer is precisely the window in which we can revert a commit on main before it is ever promoted.
• release mirrors what is running in production. You never merge a PR into release by hand; a scheduled workflow merges main into it for you (see Scheduled promotion below). Because release only ever moves forward by absorbing already-reviewed, already-on-sandbox commits from main, production is by construction a strict subset of what sandbox has already survived.

This is why you should never commit directly to release, and why the promotion job uses a merge rather than a force-push: release must always be a real ancestor-plus of main, never a divergent history.

The whole path, end to end

Here is the full journey of a commit, which the rest of the page unpacks stage by stage:

flowchart TD
    PR[Pull Request] -->|CI workflow: php, javascript,<br/>lint, cypress×4, python,<br/>teaching-assistant| Review[Code review approves]
    Review -->|merge| Main[main branch]
    Main -->|push triggers<br/>build-containers.yml| BuildS[Build 5 images<br/>tag: sha + latest-main]
    BuildS -->|kustomize edit set image<br/>in omegaup/prod overlays/sandbox| Sandbox[sandbox.omegaup.com]
    Main -.->|daily-release / weekly-release<br/>merge main → release| Release[release branch]
    Release -->|push triggers<br/>build-containers.yml| BuildP[Build 5 images<br/>tag: sha + latest-release]
    BuildP -->|kustomize edit set image<br/>in omegaup/prod overlays/production| Prod[omegaup.com]

Stage 1 — CI must pass before merge

Before a PR can merge into main, it must be green on the CI workflow, which runs on every pull_request and on every push to main (and is also exposed as a reusable workflow_call). A concurrency group keyed on the PR number cancels any in-flight run when you push a new commit — cancel-in-progress: true — so you never burn runners double-testing a stale revision.

CI is not one check but a fan-out of jobs, and the gate before all of them is verify-action-hashes: it runs ./hack/gha-reversemap.sh verify-mapusage to confirm that every third-party GitHub Action is pinned to a full commit SHA rather than a mutable tag like @v4. That is a supply-chain defense — a tag can be re-pointed at malicious code under you, a 40-character SHA cannot — and it is why you'll see actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 everywhere instead of @v4 in our workflows.

The real test jobs then run in parallel:

Two details in the php job are worth internalizing because they are exactly what breaks when a PR touches the schema. First, before any test runs, CI downloads the Go gitserver binary — omegaup-gitserver.tar.xz from omegaup/gitserver release v1.9.13 — plus libinteractive.jar v2.0.27, because the PHP tests need a live gitserver to store problem git repositories against. This is a concrete reminder that the grader stack lives in other repos (omegaup/quark for the grader/runner/broadcaster, omegaup/gitserver for problem storage) and is consumed here as pinned release artifacts, never built from this monorepo.

Second, CI validates the database migrations as migrations, not just by running them: stuff/db-migrate.py validate checks the scripts under frontend/database/, then db-migrate.py migrate --databases=omegaup-test applies them to a fresh MySQL, and finally stuff/policy-tool.py validate and stuff/database_schema.py validate assert the resulting schema matches the checked-in policy. If you add a column but forget to regenerate the schema, this is the job that fails you — long before production.

Cypress runs in four shards

The Cypress suite is deliberately split into a fail-fast: false matrix of four shards so the ~25-minute wall-clock cost is paid in parallel instead of serially, and so one flaky spec doesn't cancel the others:

On failure, every shard uploads its screenshots and videos as run artifacts (guarded by if: always() && hashFiles(...)), and dumps docker logs for every running container into frontend/tests/runfiles/containers/ — so when a run goes red, the evidence is already attached to the workflow run and you don't have to reproduce locally to see what the browser saw.

Stage 2 — merge to main builds and deploys sandbox

The moment your reviewed PR merges, the push to main fires build-containers.yml. This is the workflow that turns source into shippable artifacts. It triggers on pushes to both main and release, and the branch it fires on decides which environment it targets — same build steps, different deploy target at the end.

The five images we build

The build step runs docker compose --file=docker-compose.k8s.yml build with DOCKER_BUILDKIT=1 and TAG=${{ github.sha }}, building five services declared in docker-compose.k8s.yml. Four of them are stage targets of a single multi-stage Dockerfile.frontend; the fifth has its own Dockerfile:

The heavy lifting happens in the shared build stage. It clones the repo at --branch=${BRANCH} (default release, overridden per-run by the --build-arg BRANCH=<branch> the workflow passes), then: builds the reKarel bundle (npm install && npx gulp && npm run build under frontend/www/rekarel), runs yarn build to produce the Webpack 5 assets, runs composer install --no-dev --classmap-authoritative for an optimized autoloader, writes a production config.php (OMEGAUP_ENVIRONMENT = 'production', cache implementation none, TEMPLATE_CACHE_DIR = /var/lib/omegaup/templates), and finally runs CompileTemplatesCmd.php to pre-compile the Twig templates into /var/lib/omegaup/templates. Compiling templates at build time is why production never pays the Twig-compile cost on the first request after a deploy.

Every image is tagged twice, pushed to two registries

After the build, the workflow logs into both registries and pushes each of the five images under two tags:

• ${{ github.sha }} — the immutable, exact-commit tag. This is the one deployments actually pin to, so a given production rollout is traceable to one specific commit and can never silently drift.
• latest-<branch> — a moving pointer (latest-main or latest-release) for humans and tooling that just want "the newest sandbox/prod build."

Both tag sets go to GitHub Container Registry (ghcr.io/omegaup/..., authenticated with the run's github.token) and to Docker Hub (omegaup/..., authenticated with the DOCKER_USERNAME / DOCKER_PASSWORD secrets). Publishing to two registries is redundancy on purpose: if one is down or rate-limiting during a deploy, the cluster can still pull from the other.

GitOps: the deploy is a commit to omegaup/prod

Here is the step that surprises people the first time: nothing in this workflow SSHes into a server or restarts a service. Instead, on a main push the final step (guarded by if: github.ref == 'refs/heads/main') clones the private omegaup/prod manifests repo, cds into k8s/omegaup/overlays/sandbox/frontend, and runs kustomize edit set image to rewrite all five image references to the new ${{ github.sha }} tag, patches the app.kubernetes.io/version label to the same SHA, then git commit + git push as omegaup-bot.

That commit is the deploy. A GitOps reconciler watching omegaup/prod notices the manifest changed and rolls the sandbox Deployment to the new images. To ship to production instead, the identical step runs behind if: github.ref == 'refs/heads/release' and edits overlays/production/frontend instead of overlays/sandbox/frontend. Same five kustomize edit set image lines, same SHA-pinning, different overlay directory — that single conditional is the entire difference between "deploy to sandbox" and "deploy to production."

Scheduled promotion to production

Production isn't deployed by a human clicking a button. Two scheduled workflows promote main into release, and once release moves, Stage 2 does the rest automatically:

• daily-release.yml — cron: '0 3 * * 0', i.e. Sundays at 03:00 UTC (20:00 PT). Despite the name it currently runs weekly, not daily; the file even carries a TODO(#1624): Make this daily once we have better coverage of the frontend, which is a candid note that we don't yet trust the automated suite enough to promote every day.
• weekly-release.yml — cron: '0 3 * * 1', i.e. Mondays at 03:00 UTC (Sunday 20:00 PT).

Both do the same core thing: they POST to the GitHub /repos/omegaup/omegaup/merges API with {"base":"release","head":"main"}, authenticated with the OMEGAUPBOT_RELEASE_TOKEN secret, to merge main into release. They parse the JSON response and hard-fail if it lacks a commit sha or comes back merged: false, so a silently failed promotion can't masquerade as success. Both can also be kicked off out-of-band via a repository_dispatch of type daily-release / weekly-release when someone needs to cut a release off-schedule.

The two workflows differ in exactly one guardrail, and it's an important one.

Backend services release on their own cadence

Everything above ships the frontend (the PHP/nginx/Vue app). The grader stack is separate code with separate versioning, and you can read its current pinned versions straight out of docker-compose.yml:

These are pinned semantic-version tags, bumped deliberately when a new grader/runner release is cut in omegaup/quark — not rebuilt on every frontend merge the way the five app images are. The frontend reaches the grader over HTTP at OMEGAUP_GRADER_URL (default https://localhost:21680), so a frontend deploy and a grader deploy are genuinely independent events. When you're diagnosing a production issue, this separation matters: a broken submission verdict is very likely a v1.9.35 backend concern in omegaup/quark, while a broken page render is a frontend-image concern in this repo.

Rolling back

Because a deploy is just a commit that pins an image SHA in omegaup/prod, a rollback is the same move in reverse: point the manifest back at the previous known-good ${{ github.sha }} tag and let the cluster reconcile. Every historical build is still in both registries under its immutable commit SHA, so there is always a concrete tag to roll back to — you are never rebuilding to recover, only re-pinning.

This is exactly what the sandbox "buffer" buys us in practice. Because main/sandbox always run ahead of release/production, a regression usually surfaces on sandbox.omegaup.com first, and reverting the offending commit on main (which rebuilds sandbox) keeps it from ever being promoted into release at the next scheduled window. The cheapest rollback is the one where the bad change never reaches production because sandbox caught it and .pause-release bought time.

Related Documentation

• Monitoring — New Relic, Prometheus, and the metrics to watch after a deploy
• Troubleshooting — common failures and how to read the logs
• Docker Setup — the local docker-compose.yml stack these images mirror
• Testing — the test suites that gate every merge in Stage 1