> For the complete documentation index, see [llms.txt](https://darren.gitbook.io/project/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://darren.gitbook.io/project/cha-jian-kuo-zhan/jian-kong/custommetrics-ji-prometheus-jian-kong-xi-tong.md). # Custom-Metrics及Prometheus监控系统 ## Custom Metrics概述 上篇文章中《K8s -- 通过自定义prometheus数据实现k8s hpa》讲到，自heapster被废弃以后，所有的指标数据都从API接口中获取，由此kubernetes将资源指标分为了两种： * Core metrics(核心指标)：由metrics-server提供API metrics.k8s.io，仅提供Node和Pod的CPU和内存使用情况。 * Custom Metrics(自定义指标)：由Prometheus Adapter提供API custom.metrics.k8s.io，由此可支持任意Prometheus采集到的指标。 想让k8s一些核心组件，比如HPA，获取核心指标以外的其它自定义指标，则必须部署一套prometheus监控系统，让prometheus采集其它各种指标，`但是prometheus采集到的metrics并不能直接给k8s用，因为两者数据格式不兼容，还需要另外一个组件(kube-state-metrics)，将prometheus的metrics 数据格式转换成k8s API接口能识别的格式，转换以后，因为是自定义API，所以还需要用Kubernetes aggregator在主API服务器中注册，以便直接通过/apis/来访问。` ## Custom Metrics 的部署流程 1. `node-exporter`：prometheus的agent端，收集Node级别的监控数据。 2. `prometheus`：监控服务端，从node-exporter拉数据并存储为时序数据。 3. `kube-state-metrics`： 将prometheus中可以用PromQL查询到的指标数据转换成k8s对应的数据格式，即\ 转换成【Custerom Metrics API】接口格式的数据，但是它不能聚合进apiserver中的功能。 4. `k8s-prometheus-adpater`：聚合apiserver，即提供了一个apiserver【cuester-metrics-api】，\ 自定义APIServer通常都要通过Kubernetes aggregator聚合到apiserver。 5. `grafana`：展示prometheus获取到的metrics。 6. 导入grafana模板。 ## 资源清单文件获取 从kubernetes源码树中的addons下获取 prometheus相关组件的资源清单文件：prometheus、node-exporter、kube-state-metrics。 从DirectXMan12项目获取 组件k8s-prometheus-adpater的清单文件。 grafana的配置在google一搜，很多项目都提供了，这里从heapster项目下载grafana资源清单文件。 下载之后，各组件归类存放到各目录： ``` $ ls grafana k8s-prometheus-adapter kube-state-metrics node_exporter prometheus ``` 规划所有组件部署的名称空间，默认是在kube-system，这里统一部署在monitoring ``` $ kubectl create namespace monitoring namespace/monitoring created ``` 并手动将清单文件中，资源所属名称空间改为monitoring 开始部署各组件\ 现在按上面写的顺序一一部署 部署node-exporter ``` $ ls node_exporter node-exporter-ds.yaml node-exporter-svc.yaml ``` 简单下看此组件部署的资源： ``` daemonset daemonset-name:prometheus-node-exporter container-name: prometheus-node-exporter hostnetwork：hostPort: 9100 image: prom/node-exporter:v0.16.0 Service: name: prometheus-node-exporter clusterIP: None ``` 应用到集群之上： ``` $ kubectl apply -f ./node_exporter daemonset.apps/prometheus-node-exporter created service/prometheus-node-exporter created $ kubectl get all -n monitoring NAME READY STATUS RESTARTS AGE pod/prometheus-node-exporter-d4wg7 1/1 Running 0 4m7s pod/prometheus-node-exporter-tqczz 1/1 Running 0 4m7s pod/prometheus-node-exporter-wcrh6 1/1 Running 0 4m7s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/prometheus-node-exporter ClusterIP None 9100/TCP 4m7s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-node-exporter 3 3 3 3 3 4m7s ``` 部署prometheus\ 从github下载的清单文件，用statefulset部署的，prometheus本身是有状态的应用，这里只部署一个副本，所以将statefulset改为deployment了， ``` $ ls prometheus prometheus-cfg.yaml prometheus-deploy.yaml prometheus-rbac.yaml prometheus-svc.yaml ``` 此组件部署的资源 ``` Deployment name:prometheus-server containers-name: prometheus image: prom/prometheus:v2.2.1 containerPort: 9090 Service name: prometheus type: NodePort nodePort: 30090<-->9090 ``` 应用： ``` $ kubectl apply -f ./prometheus configmap/prometheus-config created deployment.apps/prometheus-server created clusterrole.rbac.authorization.k8s.io/prometheus created serviceaccount/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created service/prometheus created ``` 对于prometheus，有几点说明： 1. 简单将原清单文件中的stateful改为了deployment，部署起来相对简单此，且只部署一个副本。 2. prometheus自带的UI监听在9090端口，使用到了NodePort，以便集群外访问。 3. prometheus使用的volume"prometheus-storage-volume"，存储所有它采集到的metrics，应该放于持久卷中。 等一会查看组件已正常运行： ``` $ kubectl get all -n prom NAME READY STATUS RESTARTS AGE pod/prometheus-node-exporter-d4wg7 1/1 Running 0 9m pod/prometheus-node-exporter-tqczz 1/1 Running 0 9m pod/prometheus-node-exporter-wcrh6 1/1 Running 0 9m pod/prometheus-server-5fcbdbcc6f-nt4wj 1/1 Running 0 2m24s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/prometheus NodePort 10.107.112.119 9090:30090/TCP 2m service/prometheus-node-exporter ClusterIP None 9100/TCP 9m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/prometheus-node-exporter 3 3 3 3 3 9m NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/prometheus-server 1 1 1 1 2m NAME DESIRED CURRENT READY AGE replicaset.apps/prometheus-server-5fcbdbcc6f 1 1 1 2m ``` 部署kube-state-metrics ``` $ ls kube-state-metrics kube-state-metrics-deploy.yaml kube-state-metrics-rbac.yaml kube-state-metrics-svc.yaml ``` 此组件部署的资源： ``` deploymet name: kube-state-metrics replicas: 1 image: gcr.io/google_containers/kube-state-metrics-amd64:v1.3.1 containerPort: 8080 service: name: kube-state-metrics port: 8080 ``` 应用: ``` $ kubectl apply -f ./kube-state-metrics deployment.apps/kube-state-metrics created serviceaccount/kube-state-metrics created clusterrole.rbac.authorization.k8s.io/kube-state-metrics created clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created service/kube-state-metrics created ``` 等一会查看： ``` $ kubectl get pod -n monitoring NAME READY STATUS RESTARTS AGE kube-state-metrics-667fb54645-xj8gr 1/1 Running 0 116s $ kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-state-metrics ClusterIP 10.104.171.60 8080/TCP 2m50s ``` 部署组件k8s-prometheus-adapter\ 最后一个核心组件，也是部署最麻烦的一个组件。\ 它是一个API服务器，提供了一个APIServer服务，名为 custom-metrics-apiserver，提供的API组： custom.metrics.k8s.io，它是自定义指标API（custom.metrics.k8s.io）的实现 查看资源清单文件： ``` $ ls k8s-prometheus-adapter custom-metrics-apiserver-auth-delegator-cluster-role-binding.yaml custom-metrics-apiserver-auth-reader-role-binding.yaml custom-metrics-apiserver-deployment.yaml custom-metrics-apiserver-resource-reader-cluster-role-binding.yaml custom-metrics-apiserver-service-account.yaml custom-metrics-apiserver-service.yaml custom-metrics-apiservice.yaml custom-metrics-cluster-role.yaml custom-metrics-config-map.yaml custom-metrics-resource-reader-cluster-role.yaml hpa-custom-metrics-cluster-role-binding.yaml ``` 此组件部署的资源： ``` deployment： name: custom-metrics-apiserver replicas: 1 containers-name： custom-metrics-apiserver image: directxman12/k8s-prometheus-adapter-amd64 ports: - containerPort: 6443 volumes: secret: secretName: cm-adapter-serving-certs Service name: custom-metrics-apiserver ports: - port: 443 targetPort: 6443 APIService name: custom-metrics-apiserver custom.metrics.k8s.io version: v1beta1 ``` 从上面该组件的deployment看出，它需要挂一个secret存储卷，secret名为"cm-adapter-serving-certs"，这个secret是一个证书，因此这里需要创建相应的证书和key，这个证书必须由k8s的kube-apiserver信任的CA签发，因此直接用k8s的CA签发。 1. 生成证书： ``` 私钥 $ (umask 077;openssl genrsa -out serving.key 2048) $ ls serving.key ``` 1. 证书请求： ``` $ openssl req -new -key serving.key -out serving.csr -subj "/CN=serving" $ ls serving.csr serving.key ``` 1. 签署证书： ``` $ openssl x509 -req -in serving.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out serving.crt -days 3650 Signature ok subject=/CN=serving Getting CA Private Key $ ls serving.crt serving.csr serving.key ``` 1. 创建secret： ``` $ kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key -n monitoring secret/cm-adapter-serving-certs created $ kubectl get secrets -n monitoring NAME TYPE DATA AGE cm-adapter-serving-certs Opaque 2 49s ``` 应用资源清单文件： ``` $ kubectl apply -f ./k8s-prometheus-adapter clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created deployment.apps/custom-metrics-apiserver created clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created serviceaccount/custom-metrics-apiserver created service/custom-metrics-apiserver created apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created configmap/adapter-config created clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created ``` 等一会查看： ``` $ kubectl get all -n monitoring |grep custom-metrics pod/custom-metrics-apiserver-746485c45d-9dnqn 1/1 Running 0 69s service/custom-metrics-apiserver ClusterIP 10.102.104.175 443/TCP 70s deployment.apps/custom-metrics-apiserver 1 1 1 1 70s replicaset.apps/custom-metrics-apiserver-746485c45d 1 1 1 71s ``` 最后查看所有的pod：四个组件的pod： ``` $ kubectl get pod -n monitoring -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE custom-metrics-apiserver-746485c45d-9dnqn 1/1 Running 0 116s 192.168.85.197 k8s-node01 kube-state-metrics-667fb54645-xj8gr 1/1 Running 0 63m 192.168.235.196 k8s-master prometheus-node-exporter-d4wg7 1/1 Running 0 175m 10.3.1.20 k8s-master prometheus-node-exporter-tqczz 1/1 Running 0 175m 10.3.1.21 k8s-node01 prometheus-node-exporter-wcrh6 1/1 Running 0 175m 10.3.1.25 k8s-node02 prometheus-server-5fcbdbcc6f-nt4wj 1/1 Running 0 89m 192.168.58.197 ``` 查看新创建的api群组： ``` $ kubectl api-versions ...... custom.metrics.k8s.io/v1beta1 metrics.k8s.io/v1beta1 ...... ``` 有了自定义指标api了，过一会就可以从接口获取到数据了： ``` curl localhost:8091/apis/custom.metrics.k8s.io/v1beta1 "apiVersion": "v1", "groupVersion": "custom.metrics.k8s.io/v1beta1", "resources": [ { "name": "namespaces/fs_reads_bytes", "singularName": "", "namespaced": false, "kind": "MetricValueList", "verbs": [ "get" ] }, ...... ``` 如此，说明自定义指标API已成功部署了，就可以借助于这些自定义指标的创建HPA了。 ## 部署grafana 既然部署了Prometheus，那么当然要部署Grafana展示Prometheus采集到的metrics数据。 查看grafana清单文件： ``` $ ls grafana grafana.yaml ``` 它就一个清单文件，部署成一个deploy和service，因为从heapster项目中复制过来的，配置grafana连接的是influxdb，因此需要改下，完整的grafana.yaml如下 ``` $ cat grafana/grafana.yaml apiVersion: apps/v1 kind: Deployment metadata: name: monitoring-grafana namespace: monitoring spec: replicas: 1 selector: matchLabels: task: monitoring k8s-app: grafana template: metadata: labels: task: monitoring k8s-app: grafana spec: containers: - name: grafana image: k8s.gcr.io/heapster-grafana-amd64:v5.0.4 ports: - containerPort: 3000 protocol: TCP volumeMounts: - mountPath: /etc/ssl/certs name: ca-certificates readOnly: true - mountPath: /var name: grafana-storage env: #- name: INFLUXDB_HOST # value: monitoring-influxdb - name: GF_SERVER_HTTP_PORT value: "3000" # The following env variables are required to make Grafana accessible via # the kubernetes api-server proxy. On production clusters, we recommend # removing these env variables, setup auth for grafana, and expose the grafana # service using a LoadBalancer or a public IP. - name: GF_AUTH_BASIC_ENABLED value: "false" - name: GF_AUTH_ANONYMOUS_ENABLED value: "true" - name: GF_AUTH_ANONYMOUS_ORG_ROLE value: Admin - name: GF_SERVER_ROOT_URL # If you're only using the API Server proxy, set this value instead: # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy value: / volumes: - name: ca-certificates hostPath: path: /etc/ssl/certs - name: grafana-storage emptyDir: {} --- apiVersion: v1 kind: Service metadata: labels: # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: 'true' kubernetes.io/name: monitoring-grafana name: monitoring-grafana namespace: monitoring spec: # In a production setup, we recommend accessing Grafana through an external Loadbalancer # or through a public IP. # type: LoadBalancer # You could also use NodePort to expose the service at a randomly-generated port # type: NodePort ports: - port: 80 targetPort: 3000 selector: k8s-app: grafana type: NodePort ``` 有三点要说明的是 1. 挂载的volume grafana-storage应该为持久卷，这里测试为挂载为emptyDir 2. grafana的svc使用了NodePort，便于集群之外访问。 3. 取消了环境变量INFLUXDB\_HOST。 应用并查看： ``` kubectl apply -f grafana/grafana.yaml $ kubectl get pod -n monitoring |grep grafana NAME READY STATUS RESTARTS AGE monitoring-grafana-7f99994bc4-mpmhz 1/1 Running 0 3m $ kubectl get svc -n monitoring |grep grafana monitoring-grafana NodePort 10.109.154.210 80:31337/TCP 6d18h ``` grafana已成功部署完，接下来，就可以用NodeIP + NodePort 这里是31337 打开grafana界面，接入Prometheus数据源，并下载grafana适用于k8s的grafana来查看各种指标数据了。\ Grafana使用\ ![](//upload-images.jianshu.io/upload_images/7378149-616fb28ad2c70023.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/756/format/webp)image.png\ ![](//upload-images.jianshu.io/upload_images/7378149-151a85d4897a0839.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/802/format/webp)image.png\ 进入Dashboards：\ ![](//upload-images.jianshu.io/upload_images/7378149-64554db47ed149a3.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/764/format/webp)image.png\ 在下面可以导入各种模板：\ ![](//upload-images.jianshu.io/upload_images/7378149-0f6dd73c23b170fb.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/946/format/webp)image.png\ 模板在哪找呢？在grafana官网 中搜索grafana模板，有很多适用于kubernetes prometheus的模板：\ ![](//upload-images.jianshu.io/upload_images/7378149-d3d08338cca48c73.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1000/format/webp)image.png\ 比如下面找到了1621号模板：\ ![](//upload-images.jianshu.io/upload_images/7378149-d66830f796c288ac.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1000/format/webp)image.png\ 按下面的方法导入：\ ![](//upload-images.jianshu.io/upload_images/7378149-bc09fcfcc1d9ff10.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1000/format/webp)image.png\ 最终展示：\ ![](//upload-images.jianshu.io/upload_images/7378149-5987967f86bf9475.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1000/format/webp)image.png Grafana之所以能够发现k8s集群中各Node、各Pod的详细使用信息，主要是因为prometheus部署时使用的配置文件，它这个配置是经过改造后适用于运行k8s集群之中，配置了很多Job、Service Discovery功能，可以自动发现集群各资源。\ \ 作者：沉沦2014\ 链接： 來源：简书\ 简书著作权归作者所有，任何形式的转载都请联系作者获得授权并注明出处。