DexioView only
Sign inMake a copy

Dexio / how-we-build-dexio / app

How app.dexio.wiki runs, and how to build it

The Dexio app at app.dexio.wiki: the web app, sign-in, and the MCP endpoint agents connect to. The server is open source at https://github.com/dexio-wiki/dexio. The marketing site is a separate static build: astro-marketing-site. How agents authenticate to the MCP endpoint: remote-mcp-server-with-oauth.

The stack

Cost at on-demand prices in us-east-1: the server about $12 a month, the database about $14 with its storage, and the pipeline $6 to $12 a month at 500 releases. Disks, snapshots and the static IP add a few dollars.

Why one server

What would change it: moving the write lock and rate limits into Postgres. Two copies behind a load balancer, on ECS or an Auto Scaling group, become worth it after that, not before.

How to build it

Prerequisites: an AWS account bootstrapped for CDK, a Route 53 hosted zone, the app in a GitHub repository, and Docker.

  1. Build one image with three stages from one Dockerfile (deploy/Dockerfile):

    FROM public.ecr.aws/docker/library/python:3.12-slim@sha256:<digest> AS base
COPY deploy/requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir --require-hashes -r /tmp/requirements.txt
COPY pyproject.toml README.md ./
COPY src ./src
RUN pip install --no-cache-dir --no-deps .

FROM base AS test      # adds test dependencies and tests; CI runs the suite here
FROM base AS runtime   # what ships: non-root user, HEALTHCHECK on /healthz

    Pin the base image by digest and install from a lockfile export with hashes. What ships is then exactly what the tests ran against.

  2. Run it with Compose: the app plus Caddy (docker-compose.yml, Caddyfile):

    {$DEXIO_DOMAIN} {
	encode gzip
	reverse_proxy dexio:8080 {
		lb_try_duration 30s
		lb_try_interval 250ms
	}
}

    lb_try_duration makes Caddy hold requests for the second or two a container swap takes instead of answering 502. Mount data with bind mounts onto the data volume, not Docker named volumes, which live on the root disk and die with the instance. Keep Caddy's certificates on the volume too, or every replacement re-issues them and spends Let's Encrypt rate limits.

  3. Write the CDK stack:

    • Security group with 80 and 443 open. 80 stays open for the ACME challenge.
    • Instance role with AmazonSSMManagedInstanceCore, for Session Manager and Run Command.
    • Instance with IMDSv2 required, a pinned AMI id, and user_data_causes_replacement=True. Without that flag a changed boot script never runs, since cloud-init runs it once per instance.
    • The data volume as its own ec2.Volume with RemovalPolicy.RETAIN. The boot script attaches it, waiting for the outgoing instance to let go. Do not use a CfnVolumeAttachment (see Pitfalls).
    • An Elastic IP and a Route 53 A record with a 60-second TTL.
    • Keys in Secrets Manager, read by the boot script with the instance role, so nothing secret is in the template or the user data.
    • RDS Postgres: not public, ingress only from the server's security group, encrypted, 7-day backups with point-in-time restore, deletion protection, RemovalPolicy.SNAPSHOT. RDS wants a subnet group across two zones even for a single-zone database.
    • A private S3 bucket for files. Presigned URLs serve them from S3's own domain, so an uploaded file never runs with the app's cookies.
    • A Data Lifecycle Manager policy: daily snapshots of the data volume, 14 kept.
    • SES send permission conditioned on ses:FromAddress, so the role can send only from one address.

  4. Keep a release pointer outside the stack: an SSM parameter holding the S3 location of the release to run. A release bundle is the deploy files (compose file, Caddyfile, boot script) plus an IMAGE file naming the tested image by digest. The boot script and the release script both read the pointer. It must not be a stack resource: when it was, every infrastructure deploy set it back to whatever source that deploy happened to package.

  5. Write the release script and store it as an SSM Command document, so the stack and the pipeline run the same copy. In order:

    • take a lock with flock;
    • exit if cloud-init is still running, since a booting replacement reads the pointer itself;
    • if the pointer matches what runs, only refresh secrets, restarting if they changed;
    • download the bundle beside the live one, carry over .env, re-read the secrets;
    • pull the image by digest while the old container keeps serving;
    • move live aside, move the new one in, docker compose up -d --no-deps --force-recreate;
    • check /healthz through Caddy for up to 60 seconds (curl --resolve <domain>:443:127.0.0.1 https://<domain>/healthz);
    • on failure, put the previous release back and recreate it.

  6. Build the pipeline, three stages:

    • Source: a CodeStar connection to GitHub, branch main, triggered on push. CDK creates the connection pending; someone authorizes it once in the console.
    • TestAndBuild: CodeBuild on ARM small, privileged for Docker. Check the lockfile export matches the shipped requirements, build the test stage, run it against a throwaway Postgres container, build the runtime stage tagged with the commit, push it to ECR, zip the bundle with IMAGE, upload it to S3, and emit release.json.
    • Release: CodeBuild again. Save the current pointer, write the new one, run the release document with ssm send-command, poll until it finishes. On failure, write the old pointer back so a replacement instance boots what is actually running.

    Use CodePipeline V1 ($1 a month flat) with cross_account_keys=False, which avoids a $1 a month KMS key. Nothing in GitHub holds AWS credentials; the pipeline pulls from GitHub.

  7. Rotate secrets without a new instance: the release script rewrites the secret-derived lines of .env from Secrets Manager on every run. Change the key in the secret and run the release document again.

Verify

Rollback: run the pipeline on the previous commit, or point the parameter at the previous bundle and run the release document.

Pitfalls we hit