obsforge

ObsForge: a metadata enrichment service for Rubin Observatory observations Learn more at https://obsforge.lsst.io

obsforge is developed with FastAPI and Safir.

Local Postgres and Redis

ObsForge local development uses Docker Compose for stateful services only. Run the FastAPI app and arq worker on the host so that reloads and debugging stay simple.

Start Postgres and Redis from this directory:

docker compose up -d postgres redis

Configure the host shell that will run ObsForge:

export OBSFORGE_DATABASE_URL=postgresql://obsforge@localhost/obsdb
export OBSFORGE_DATABASE_PASSWORD=INSECURE-PASSWORD
export OBSFORGE_ARQ_MODE=production
export OBSFORGE_ARQ_QUEUE_URL=redis://localhost:6379/0
export OBSFORGE_ALEMBIC_CONFIG_PATH=./alembic.ini

The FastAPI app can register jobs with only Postgres and Redis configured. To run the worker and populate ObsCore rows, also configure the Prompt Butler repository and the ObsCore exporter config:

export OBSFORGE_BUTLER_LABEL=prompt
export OBSFORGE_BUTLER_REPOSITORY=/path/to/prompt/butler
export OBSFORGE_OBSCORE_CONFIG=/path/to/prompt.yaml
export OBSFORGE_OBSCORE_DATASET_TYPE=preliminary_visit_image

The worker uses a remote Butler and requires a service token:

export OBSFORGE_BUTLER_ACCESS_TOKEN=...

To enable debug-level application logs, also set:

export OBSFORGE_LOG_LEVEL=DEBUG

Initialize the database and start the development server:

uv run obsforge init --reset
uv run uvicorn obsforge.main:app --reload

When testing queued enrichment, start the arq worker in a second shell with the same environment:

uv run arq obsforge.worker.main.WorkerSettings

Exercise the local API from another shell:

BASE=http://127.0.0.1:8000/obsforge

curl -sS "$BASE/" | jq .

PAYLOAD='{
  "instrument": "LSSTCam",
  "day_obs": 20260327,
  "visit": 20260327123456,
  "datasets": [
    {
      "dataset_type": "preliminary_visit_image",
      "id": "019ba0a6-0173-765f-bf27-56884ff9342a"
    }
  ],
  "timespan": {
    "begin": "2026-03-27T08:15:10Z",
    "end": "2026-03-27T08:15:45Z"
  }
}'

RESPONSE=$(
  curl -sS \
    -H "Content-Type: application/json" \
    -d "$PAYLOAD" \
    "$BASE/register"
)

echo "$RESPONSE" | jq .

JOB_ID=$(echo "$RESPONSE" | jq -r '.id')

curl -sS "$BASE/jobs/$JOB_ID" | jq .

The datasets entries are persisted in the job's registration_payload. During enrichment, the worker selects entries whose dataset_type matches OBSFORGE_OBSCORE_DATASET_TYPE, uses their UUIDs to constrain lsst.dax.obscore, and upserts the returned ObsCore records into ObsDB.

Registering the same instrument and visit pair again is idempotent:

curl -sS \
  -H "Content-Type: application/json" \
  -d "$PAYLOAD" \
  "$BASE/register" | jq .

To abort an enrichment job:

curl -i -sS -X DELETE "$BASE/jobs/$JOB_ID"

The abort command must run while the queued arq job still exists, otherwise it returns:

{"detail":"Queued job not found"}

Stop the services when done:

docker compose down

To discard local Postgres data and start over, also remove the Compose volume:

docker compose down --volumes

The Compose stack is for interactive local development. The tox test harness is managed separately and should keep owning its own test services.