Skip to content
On this page

Concurrency Flow

This page illustrates how Pipelines-as-Code manages concurrent PipelineRun execution. When you set a concurrency limit on a Repository CR, Pipelines-as-Code queues incoming PipelineRuns and starts them only when capacity allows.

Flow diagram

flowchart LR
%% title: How the controller and the watcher queue PipelineRuns under a concurrency limit
%% autoplay
  Ctrl@{ img: "/icons/controller.svg", label: "Controller", w: 48, h: 48, constraint: "on" }
  Limit@{ img: "/icons/decision.svg", label: "concurrency_limit", w: 48, h: 48, constraint: "on" }
  Run@{ img: "/icons/pipeline.svg", label: "PipelineRun", w: 48, h: 48, constraint: "on" }
  Watch@{ img: "/icons/monitor.svg", label: "Watcher", w: 48, h: 48, constraint: "on" }
  Queue@{ img: "/icons/queue.svg", label: "Queue", w: 48, h: 48, constraint: "on" }
  Git@{ img: "/icons/repository.svg", label: "Git provider", w: 48, h: 48, constraint: "on" }
  Done@{ img: "/icons/success.svg", label: "completed", w: 48, h: 48, constraint: "on" }
  Ctrl --> Limit
  Limit none@-->|not set| Run
  Limit set@-->|set| Run
  Ctrl -->|status| Git
  Run --> Watch
  Watch --> Queue
  Queue -->|start| Run
  Watch -->|status| Git
  Watch --> Done

%% play: Queued
%% skip Done
%% Ctrl -> Limit : event
%% Limit -> Run @set : pending | Every matched PipelineRun is created pending with the state queued
%% Ctrl -> Git : queued | The controller reports them as queued and stores their order, sorted by name
%% Run -> Watch : queued
%% Watch -> Queue | The watcher adds them to the queue of their Repository
%% Queue -> Run : start | While fewer runs than the limit are running, the first one in the queue starts
%% Watch -> Git : running
%% play: Started
%% skip Queue Done
%% Ctrl -> Limit : event
%% Limit -> Run @none : started | Without a limit the PipelineRun starts right away with the state started
%% Ctrl -> Git : running
%% Run -> Watch | The watcher follows it and does nothing until it is done
%% play: Completed
%% skip Ctrl Limit
%% Run -> Watch : done
%% Watch -> Git : success | The final status goes to the Git provider first
%% Watch -> Queue | The run leaves the queue and frees its slot
%% wait 1000 | The next pending PipelineRun of the Repository takes the slot
%% Queue -> Run : next
%% Watch -> Done | The finished run gets the state completed

Inspecting the queue at runtime

Every concurrency bug is the same shape: what the watcher believes is running drifts away from what is really running in the cluster. When a repository’s queue looks stuck — nothing starting, or more running than the configured concurrency_limit — the fastest way to find out why is to ask the watcher directly instead of guessing from PipelineRun status.

The watcher can serve a read-only, JSON snapshot of its in-memory queues on /debug/queue, on the same port it already uses for its health probe.

This endpoint is disabled by default and must be explicitly enabled. It is unauthenticated: anything that can reach the watcher pod on its probe port (any workload in the cluster with network access to it, not just cluster API clients) can read the namespace, Repository name, and pending/running PipelineRun names for every repository the watcher knows about. Only turn it on for troubleshooting, and turn it back off afterward.

Enabling it

Set PAC_ENABLE_QUEUE_DEBUG=true on the pac-watcher container in the pipelines-as-code-watcher Deployment, then restart the pod:

kubectl -n pipelines-as-code set env deployment/pipelines-as-code-watcher \
  -c pac-watcher PAC_ENABLE_QUEUE_DEBUG=true

Unset it (or set it to false) to disable the endpoint again:

kubectl -n pipelines-as-code set env deployment/pipelines-as-code-watcher \
  -c pac-watcher PAC_ENABLE_QUEUE_DEBUG-

Querying it

The endpoint is not exposed by the watcher Service (which only serves metrics), so reach it through the pod itself. The API server’s pod proxy does not resolve named container ports, so use the numeric port (8080 by default, or whatever PAC_WATCHER_PORT is set to):

POD=$(kubectl -n pipelines-as-code get pods -l app.kubernetes.io/name=watcher -o jsonpath='{.items[0].metadata.name}')

# through the API server's pod proxy (what "kubectl proxy" exposes too)
kubectl get --raw "/api/v1/namespaces/pipelines-as-code/pods/http:${POD}:8080/proxy/debug/queue" | jq .

# or port-forward straight to the pod
kubectl -n pipelines-as-code port-forward "pod/${POD}" 8080:8080 &
curl -s localhost:8080/debug/queue | jq .

The response is a map keyed by namespace/repository:

{
  "my-namespace/my-repo": {
    "limit": 1,
    "running": ["my-namespace/pr-run-2"],
    "pending": ["my-namespace/pr-run-3", "my-namespace/pr-run-4"]
  }
}

A few things to look for:

  • len(running) should never exceed limit.
  • A repository key that never disappears after all of its PipelineRuns have finished means a queue slot was leaked, and the next run for that repository will never start until the watcher restarts.
  • pending staying non-empty while running is empty for longer than expected means the queue has stalled.

The endpoint answers 503 while the watcher is busy reconciling, because it gives up on the queue lock rather than block the work it is reporting on. That is normal under load — just ask again.