ELT (Airflow) configuration

The ELT DAGs (staging_dag, process_dag) run on Apache Airflow and share the same PostgreSQL database as the backend. They have no direct HTTP dependency on the backend at startup: parameters (source configuration, transform config, callback URLs) are passed in at trigger time, by the backend.

Deploying the DAGs

The Docker Compose setup builds a custom Airflow image (docker/Dockerfile.airflow) that bundles apps/elt/dags and the shared libs/data_manipulation package. For a platform deployment, deploy this image (or your own image built the same way) as your Airflow workers/scheduler.

The image is built on the official GDAL image rather than on apache/airflow, because the ingestion code shells out to ogr2ogr and needs GDAL >= 3.13, which no Debian release packages. Airflow is installed on top from apps/elt/uv.lock, and the official image's /entrypoint is reused. See the header of docker/Dockerfile.airflow for the details.

Key settings

Setting	Purpose
AIRFLOW_UID	User ID Airflow containers run as. Set in .env; make install-python writes your current UID automatically
AIRFLOW_STAGING_TIMEOUT_SECONDS	Timeout, in seconds, for the staging task execution (default: 600)
USE_ORG_SCHEMA	Schema of the final tables written by scheduled runs: the org short name when true, the shared data schema otherwise (default: false). Must match the backend USE_ORG_SCHEMA
AIRFLOW_VERSION	Airflow version installed by Dockerfile.airflow; keep in sync with apps/elt/pyproject.toml (default: 3.3.2)

The backend also needs to be pointed at the Airflow instance: see AIRFLOW_INTERNAL_URL, AIRFLOW_USERNAME and AIRFLOW_PASSWORD in the backend configuration.

Callbacks

Both DAGs accept success_callback_url / failure_callback_url parameters and call them on completion. These URLs point back at the backend (BACKEND_INTERNAL_URL), which is how the backend learns a run finished without polling Airflow continuously.

Source databases

The staging_dag reads external source databases (the Database import type) through Airflow Connections. See adding a source database for how to configure one.