pushgateway指标聚合问题

一 问题现象

一个job有多个实例推送指标，但是从pushgateway上看这个job的instance字段，只显示一个实例的ip，而不是多个实例。导致在grafana上无法正常根据ip查看监控。

应用的prometheus的配置

management:
  metrics:
    tags:
      application: ${spring.application.name}
    export:
      prometheus:
        pushgateway:
          #pushgateway地址
          baseUrl:  ***:9091
          #推送周期
          pushRate: 15s
          #job定义名
          job: ${spring.application.name}
          #启用推送
          enabled: true
  ## actuator暴露出prometheus
  endpoints:
    web:
      exposure:
        include: prometheus
  endpoint:
    health:
      show-details: always

二 解决办法

增加grouping-key，避免多个不同的实例的指标被聚合。HOSTNAME这个环境变量是k8s自带的，并且每个实例都不一样。当时想找一个能标识pod ip的环境变量，用env命令搜了一下，并没有。

management:
  metrics:
    tags:
      application: ${spring.application.name}
    export:
      prometheus:
        pushgateway:
          #pushgateway地址
          baseUrl: ***:9091
          #推送周期
          pushRate: 15s
          #job定义名
          job: ${spring.application.name}
          #启用推送
          enabled: true
          grouping-key:
            hostname: ${HOSTNAME}
  ## actuator暴露出prometheus
  endpoints:
    web:
      exposure:
        include: prometheus,health
  endpoint:
    health:
      show-details: always

三 groupingkey作用分析

pushgateway指标上报的源码路径：io.prometheus.client.exporter.PushGateway#doRequest

没配置groupingkey

配置了groupingkey

由于是本机debug，HostName这个环境变量取不到，k8s上正常取值。

grouping-key 官方解释 https://github.com/prometheus/pushgateway

The default port the Pushgateway is listening to is 9091. The path looks like
 
      /metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}
 
<JOB_NAME> is used as the value of the job label, followed by any number of other label pairs (which might or might not include an instance label). The label set defined by the URL path is used as a grouping key. Any of those labels already set in the body of the request (as regular labels, e.g. name{job=“foo”} 42) will be overwritten to match the labels defined by the URL path!

pushgateway是用请求pushgateway的url的path作为grouping-key的。如果不配置任何grouping-key，pushgateway会把所有job字段一样的指标聚合到一起。