Migrate from Prometheus/Grafana to OpenTelemetry#
The Prometheus/Grafana based monitoring setup will soon be deprecated in favor of the OpenTelemetry setup. This section provides the details required to update/migrate your Prometheus configuration and Grafana dashboard to continue monitoring with OpenTelemetry. Refer to Opentelemetry setup for the new setup before proceeding further.
Refer to Prometheus/Grafana-only section for the soon to be deprecated setup.
Update Prometheus configuration#
With a Prometheus-only setup, you need to set up a
scrape_configs configuration or service discovery plugin to specify the targets for pulling metrics data. In the OpenTelemetry setup, each Pod pushes metrics to the OpenTelemetry Collector. The Prometheus configuration now only needs to scrape from the OpenTelemetry Collector to get all the data from OpenTelemetry-instrumented applications.
The new Prometheus configuration for the
otel-collector Collector hostname is:
scrape_configs: - job_name: 'otel-collector' scrape_interval: 500ms static_configs: - targets: ['otel-collector:8888'] # metrics from the collector itself - targets: ['otel-collector:8889'] # metrics collected from other applications
Update Grafana dashboard#
The OpenTelemetry Histogram provides quantile window buckets automatically (unlike the Prometheus Summary instrument). You need to manually configure the required quantile window. The quatile window metric will then be available as a separate time series metric.
In addition, the OpenTelemetry
Counter/UpDownCounter instruments do not add the
_total suffix to the base metric name.
To adapt Prometheus queries in Grafana:
Use the histogram_quantile function to query the average or desired quantile window time series data from Prometheus. For example, to view the 0.99 quantile of the
jina_receiving_request_secondsmetric over the last 10 minutes, use query
_totalprefix from the Counter/UpDownCounter metric names.
You can download a sample Grafana dashboard JSON file and import it into Grafana to get started with some pre-built graphs.
A list of available metrics is in the Flow Instrumentation section.