Skip to content

About

A Kubernetes operator for managing Prefect servers and work pools

Resources

Stars

23 stars

Watchers

5 watching

Forks

Latest commit

 

History

307 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

prefect-operator

The Prefect Operator is an application that can manage instances of Prefect. It follows the Kubernetes operator pattern.

Custom Resource Definitions

PrefectServer

The PrefectServer CRD

PrefectWorkPool

The PrefectWorkPool CRD

PrefectWorkQueue

The PrefectWorkQueue CRD

PrefectDeployment

The PrefectDeployment CRD

PrefectAutomation

The PrefectAutomation CRD

Pod template overrides

Most pod settings, such as resources, node selectors, tolerations and extra containers, have their own fields. For anything else, you can patch the pods the operator builds with a pod template:

Resource Field Pods Container names
PrefectServer deploymentTemplate the server Deployment prefect-server, wait-for-database (init)
PrefectServer migrationJobTemplate the PostgreSQL migration Job prefect-server-migration, wait-for-database (init)
PrefectWorkPool deploymentTemplate the worker Deployment prefect-worker

A template is a partial pod template, with metadata and spec like a Deployment's spec.template, applied as a strategic merge patch. Lists of containers are matched by name, so you can change one setting of a container without restating the rest of it. The container names above are stable. Patch directives such as $patch: delete work too.

A patch merges into what the operator built, so a probe you patch keeps the operator's handler. To swap the handler, for example from httpGet to tcpSocket, replace the whole probe with $patch: replace. Setting httpGet: null instead only works with kubectl apply --server-side, because a plain kubectl apply drops the null.

deploymentTemplate:
  spec:
    containers:
      - name: prefect-server
        readinessProbe:
          $patch: replace
          tcpSocket:
            port: 4200
          periodSeconds: 10

Lists of strings such as args and command are replaced, not merged. On prefect-worker, overriding args replaces the whole list, including --pool <name>, so include every argument the worker needs.

The operator applies the other fields first and the template last, so the template wins. The exception is the fields the operator depends on, which it puts back if a template overrides them:

  • the pod labels the Deployment selects on
  • the api port of the prefect-server container
  • the prefect-data volume and its mounts
  • the environment variables the operator derives from the resource, such as the database settings and the worker's PREFECT_API_URL. Variables from settings are not protected.

The resource's DeploymentTemplateApplied (or MigrationJobTemplateApplied) condition describes the desired pod template after rendering, before Kubernetes writes or migration completion; it does not confirm that running pods use it. Its reason is ProtectedFieldsIgnored, and its message lists the fields that kept the operator's values, such as spec.containers[prefect-worker].env[PREFECT_API_URL]. The operator also emits a Warning event when that list changes. Otherwise the reason is TemplateApplied. The condition is removed when there is no template or template rendering fails. On a PrefectServer, a rendering failure clears both template conditions because the combined render did not complete. Check DeploymentReconciled and MigrationJobReconciled for reconciliation progress and errors.

For example, to run the server as a non-root user and give it more time before the first readiness check:

apiVersion: prefect.io/v1
kind: PrefectServer
metadata:
  name: prefect
spec:
  postgres:
    # ...
  deploymentTemplate:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        fsGroup: 1001
      containers:
        - name: prefect-server
          readinessProbe:
            initialDelaySeconds: 30
  migrationJobTemplate:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001

Pod template labels and annotations that other tools add to a Deployment, such as the annotation kubectl rollout restart sets, are left alone. The operator records which ones it set in the Deployment's prefect.io/pod-template-labels and prefect.io/pod-template-annotations annotations, so that it only removes those when they're deleted from the template.

The migration Job's name comes from a hash of its spec, so changing migrationJobTemplate runs the migrations again. They are safe to run again.

The template is stored as is, without a schema, so a misspelled field is only caught when the operator applies it. It then sets the resource's DeploymentReconciled (or MigrationJobReconciled) condition to False with the reason InvalidPodTemplate, emits a Warning event, and leaves the running pods as they are. It tries again when the template changes. A template that adds a container with the name of an existing one is rejected the same way.

Installing

Install the operator with its Helm chart:

helm repo add prefect-operator https://prefecthq.github.io/prefect-operator
helm repo update
helm install prefect-operator prefect-operator/prefect-operator --namespace prefect-system --create-namespace

The chart's README covers verifying the chart's signature, the chart's values, and uninstalling.

Development

Prerequisites

  • mise (curl https://mise.run | sh). It installs Go, kubectl, Helm, and the other tools pinned in .mise.toml:

    mise trust
    mise install

    The make targets run mise install for you. Python tools such as yamllint also need pipx (pip install --user pipx); see jdx/mise#2536.

  • Docker, to build the operator image.

  • A Kubernetes cluster, such as minikube. CI installs the chart on the Kubernetes versions listed in operator-lint-and-test.yaml.

Run the operator from your machine

Install the CRDs into the cluster in your current kube context, and then run the operator against that cluster:

make install run

Installing CRDs needs cluster-wide permissions. If make install fails with RBAC errors, use an account with cluster-admin.

In another shell, apply a sample resource from deploy/samples:

kubectl apply -f deploy/samples/v1_prefectserver_ephemeral.yaml

Deploy your build to a cluster

Build the operator image and push it to a registry that your cluster can pull from:

make docker-build docker-push IMG=<some-registry>/prefect-operator:<tag>

make deploy installs the chart with the image set in its values.yaml, and ignores IMG. To deploy your image, build the chart's dependencies and pass the image to Helm:

make helmbuild
helm install prefect-operator deploy/charts/prefect-operator \
  --namespace prefect-system --create-namespace \
  --set operator.image.repository=<some-registry>/prefect-operator \
  --set operator.image.tag=<tag>

Uninstall

Delete the resources you created, then the operator, then the CRDs:

kubectl delete -f deploy/samples/v1_prefectserver_ephemeral.yaml
make undeploy
make uninstall

Contributing

NOTE: Run make help for more information on all potential make targets

More information can be found via the Kubebuilder Documentation

License

The Prefect Operator is licensed under the Apache License, Version 2.0. See LICENSE.

About

A Kubernetes operator for managing Prefect servers and work pools

Resources

Stars

23 stars

Watchers

5 watching

Forks

Releases

Packages

Used by

Contributors

Languages