Pipeline Controller Metrics

The following pipeline metrics are available at controller-service on port 9090.

We expose several kinds of exporters, including Prometheus, Google Stackdriver, and many others. You can set them up using observability configuration.

Name Type Labels/Tags Status
tekton_pipelines_controller_pipelinerun_duration_seconds_[bucket, sum, count] Histogram/LastValue(Gauge) *pipeline=<pipeline_name>
*pipelinerun=<pipelinerun_name>
status=<status>
namespace=<pipelinerun-namespace>		 experimental
tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_[bucket, sum, count] Histogram/LastValue(Gauge) *pipeline=<pipeline_name>
*pipelinerun=<pipelinerun_name>
status=<status>
*task=<task_name>
*taskrun=<taskrun_name>
namespace=<pipelineruns-taskruns-namespace>
*reason=<reason>		 experimental
tekton_pipelines_controller_pipelinerun_count Counter status=<status>
*reason=<reason>		 deprecate
tekton_pipelines_controller_pipelinerun_total Counter status=<status> experimental
tekton_pipelines_controller_running_pipelineruns_count Gauge deprecate
tekton_pipelines_controller_running_pipelineruns Gauge experimental
tekton_pipelines_controller_taskrun_duration_seconds_[bucket, sum, count] Histogram/LastValue(Gauge) status=<status>
*task=<task_name>
*taskrun=<taskrun_name>
namespace=<pipelineruns-taskruns-namespace>
*reason=<reason>		 experimental
tekton_pipelines_controller_taskrun_count Counter status=<status>
*reason=<reason>		 deprecate
tekton_pipelines_controller_taskrun_total Counter status=<status> experimental
tekton_pipelines_controller_running_taskruns_count Gauge deprecate
tekton_pipelines_controller_running_taskruns Gauge experimental
tekton_pipelines_controller_running_taskruns_throttled_by_quota_count Gauge
namespace=<pipelinerun-namespace>		 deprecate
tekton_pipelines_controller_running_taskruns_throttled_by_node_count Gauge
namespace=<pipelinerun-namespace>		 deprecate
tekton_pipelines_controller_running_taskruns_throttled_by_quota Gauge
namespace=<pipelinerun-namespace>		 experimental
tekton_pipelines_controller_running_taskruns_throttled_by_node Gauge
namespace=<pipelinerun-namespace>		 experimental
tekton_pipelines_controller_client_latency_[bucket, sum, count] Histogram experimental

The Labels/Tag marked as “*” are optional. And there’s a choice between Histogram and LastValue(Gauge) for pipelinerun and taskrun duration metrics.

Configuring Metrics using config-observability configmap

A sample config-map has been provided as config-observability. By default, taskrun and pipelinerun metrics have these values:

    metrics.taskrun.level: "task"
    metrics.taskrun.duration-type: "histogram"
    metrics.pipelinerun.level: "pipeline"
    metrics.running-pipelinerun.level: ""
    metrics.pipelinerun.duration-type: "histogram"
    metrics.count.enable-reason: "false"

Following values are available in the configmap:

configmap data value description
metrics.taskrun.level taskrun Level of metrics is taskrun
metrics.taskrun.level task Level of metrics is task and taskrun label isn’t present in the metrics
metrics.taskrun.level namespace Level of metrics is namespace, and task and taskrun label isn’t present in the metrics
metrics.pipelinerun.level pipelinerun Level of metrics is pipelinerun
metrics.pipelinerun.level pipeline Level of metrics is pipeline and pipelinerun label isn’t present in the metrics
metrics.pipelinerun.level namespace Level of metrics is namespace, pipeline and pipelinerun label isn’t present in the metrics
metrics.running-pipelinerun.level pipelinerun Level of running-pipelinerun metrics is pipelinerun
metrics.running-pipelinerun.level pipeline Level of running-pipelinerun metrics is pipeline and pipelinerun label isn’t present in the metrics
metrics.running-pipelinerun.level namespace Level of running-pipelinerun metrics is namespace, pipeline and pipelinerun label isn’t present in the metrics
metrics.running-pipelinerun.level `` Level of running-pipelinerun metrics is cluster, namespace, pipeline and pipelinerun label isn’t present in the metrics.
metrics.taskrun.duration-type histogram tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds and tekton_pipelines_controller_taskrun_duration_seconds is of type histogram
metrics.taskrun.duration-type lastvalue tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds and tekton_pipelines_controller_taskrun_duration_seconds is of type gauge or lastvalue
metrics.pipelinerun.duration-type histogram tekton_pipelines_controller_pipelinerun_duration_seconds is of type histogram
metrics.pipelinerun.duration-type lastvalue tekton_pipelines_controller_pipelinerun_duration_seconds is of type gauge or lastvalue
metrics.count.enable-reason false Sets if the reason label should be included on count metrics
metrics.taskrun.throttle.enable-namespace false Sets if the namespace label should be included on the tekton_pipelines_controller_running_taskruns_throttled_by_quota metric

Histogram value isn’t available when pipelinerun or taskrun labels are selected. The Lastvalue or Gauge will be provided. Histogram would serve no purpose because it would generate a single bar. TaskRun and PipelineRun level metrics aren’t recommended because they lead to an unbounded cardinality which degrades the observability database.

To check that appropriate values have been applied in response to configmap changes, use the following commands:

kubectl port-forward -n tekton-pipelines service/tekton-pipelines-controller 9090

And then check that changes have been applied to metrics coming from http://127.0.0.1:9090/metrics