Concurrency Flow
This page illustrates how Pipelines-as-Code manages concurrent PipelineRun execution. When you set a concurrency limit on a Repository CR, Pipelines-as-Code queues incoming PipelineRuns and starts them only when capacity allows.
Flow diagram
flowchart LR
%% title: How the controller and the watcher queue PipelineRuns under a concurrency limit
%% autoplay
Ctrl@{ img: "/icons/controller.svg", label: "Controller", w: 48, h: 48, constraint: "on" }
Limit@{ img: "/icons/decision.svg", label: "concurrency_limit", w: 48, h: 48, constraint: "on" }
Run@{ img: "/icons/pipeline.svg", label: "PipelineRun", w: 48, h: 48, constraint: "on" }
Watch@{ img: "/icons/monitor.svg", label: "Watcher", w: 48, h: 48, constraint: "on" }
Queue@{ img: "/icons/queue.svg", label: "Queue", w: 48, h: 48, constraint: "on" }
Git@{ img: "/icons/repository.svg", label: "Git provider", w: 48, h: 48, constraint: "on" }
Done@{ img: "/icons/success.svg", label: "completed", w: 48, h: 48, constraint: "on" }
Ctrl --> Limit
Limit none@-->|not set| Run
Limit set@-->|set| Run
Ctrl -->|status| Git
Run --> Watch
Watch --> Queue
Queue -->|start| Run
Watch -->|status| Git
Watch --> Done
%% play: Queued
%% skip Done
%% Ctrl -> Limit : event
%% Limit -> Run @set : pending | Every matched PipelineRun is created pending with the state queued
%% Ctrl -> Git : queued | The controller reports them as queued and stores their order, sorted by name
%% Run -> Watch : queued
%% Watch -> Queue | The watcher adds them to the queue of their Repository
%% Queue -> Run : start | While fewer runs than the limit are running, the first one in the queue starts
%% Watch -> Git : running
%% play: Started
%% skip Queue Done
%% Ctrl -> Limit : event
%% Limit -> Run @none : started | Without a limit the PipelineRun starts right away with the state started
%% Ctrl -> Git : running
%% Run -> Watch | The watcher follows it and does nothing until it is done
%% play: Completed
%% skip Ctrl Limit
%% Run -> Watch : done
%% Watch -> Git : success | The final status goes to the Git provider first
%% Watch -> Queue | The run leaves the queue and frees its slot
%% wait 1000 | The next pending PipelineRun of the Repository takes the slot
%% Queue -> Run : next
%% Watch -> Done | The finished run gets the state completedInspecting the queue at runtime
Every concurrency bug is the same shape: what the watcher believes is running
drifts away from what is really running in the cluster. When a repository’s
queue looks stuck — nothing starting, or more running than the configured
concurrency_limit — the fastest way to find out why is to ask the watcher
directly instead of guessing from PipelineRun status.
The watcher can serve a read-only, JSON snapshot of its in-memory queues on
/debug/queue, on the same port it already uses for its health probe.
Enabling it
Set PAC_ENABLE_QUEUE_DEBUG=true on the pac-watcher container in the
pipelines-as-code-watcher Deployment, then restart the pod:
kubectl -n pipelines-as-code set env deployment/pipelines-as-code-watcher \
-c pac-watcher PAC_ENABLE_QUEUE_DEBUG=trueUnset it (or set it to false) to disable the endpoint again:
kubectl -n pipelines-as-code set env deployment/pipelines-as-code-watcher \
-c pac-watcher PAC_ENABLE_QUEUE_DEBUG-Querying it
The endpoint is not exposed by the watcher Service (which only serves
metrics), so reach it through the pod itself. The API server’s pod proxy does
not resolve named container ports, so use the numeric port (8080 by
default, or whatever PAC_WATCHER_PORT is set to):
POD=$(kubectl -n pipelines-as-code get pods -l app.kubernetes.io/name=watcher -o jsonpath='{.items[0].metadata.name}')
# through the API server's pod proxy (what "kubectl proxy" exposes too)
kubectl get --raw "/api/v1/namespaces/pipelines-as-code/pods/http:${POD}:8080/proxy/debug/queue" | jq .
# or port-forward straight to the pod
kubectl -n pipelines-as-code port-forward "pod/${POD}" 8080:8080 &
curl -s localhost:8080/debug/queue | jq .The response is a map keyed by namespace/repository:
{
"my-namespace/my-repo": {
"limit": 1,
"running": ["my-namespace/pr-run-2"],
"pending": ["my-namespace/pr-run-3", "my-namespace/pr-run-4"]
}
}A few things to look for:
len(running)should never exceedlimit.- A repository key that never disappears after all of its PipelineRuns have finished means a queue slot was leaked, and the next run for that repository will never start until the watcher restarts.
pendingstaying non-empty whilerunningis empty for longer than expected means the queue has stalled.
The endpoint answers 503 while the watcher is busy reconciling, because it
gives up on the queue lock rather than block the work it is reporting on. That
is normal under load — just ask again.