距离上一次更新该文章已经过了 527 天,文章所描述的內容可能已经发生变化,请留意。
前言 Jaeger 是一个开源的端到端的分布式跟踪系统, 允许用户在复杂的分布式系统中监控和排查故障。
利用Rancher的安装方式 安装cert-manager 参考文档:
jaeger requires:https://www.jaegertracing.io/docs/1.49/operator/#prerequisite
cmctl(the tool which is to manage and verify cert-manager) install:https://cert-manager.io/v1.6-docs/usage/cmctl/#installation
cert-manager verify:https://cert-manager.io/v1.6-docs/installation/verify/
安装cert-manager:https://cert-manager.io/v1.6-docs/installation/kubectl/
默认静态安装cert-manager:https://cert-manager.io/v1.6-docs/installation/#default-static-install
1 kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.6.3/cert-manager.yaml
quay.io
国内速度较慢,可手动更换yaml文件中的默认镜像源1 quay.io -> quay.mirrors.ustc.edu.cn
1 cd /usr/local/bin && curl -L -o cmctl.tar.gz https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cmctl-linux-amd64.tar.gz && tar xzf cmctl-linux-amd64.tar.gz && chmod +x cmctl
1 2 3 The cert-manager API is ready
Rancher -> Apps Rancher -> Apps中可找到jaeger operator的install button,最好是安装在 observability
namespace下。
默认安装方式,数据存储在内存中。可按文档下方修改为es或其他方式存储。
在较新版本中,jaeger要求集群先安装cert-manager
,否则可能报错
1 2 3 4 5 helm install --namespace=observability --timeout=10m0s --values=/home/shell/helm/values-jaeger-operator-2.46.2.yaml --version=2.46.2 --wait=true jaeger-operator /home/shell/helm/jaeger-operator-2.46.2.tgz 2023-09-22T15 :15:47.650969451+08:00 creating 1 resource(s) Error : INSTALLATION FAILED: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "jaeger-operator-service-cert" namespace: "observability" from "": no matches for kind "Certificate" in version "cert-manager.io/v1" ensure CRDs are installed first, resource mapping not found for name: "selfsigned-issuer" namespace: "observability" from "": no matches for kind "Issuer" in version "cert-manager.io/v1" 2023-09-22T15 :15:48.112055095+08:00 ensure CRDs are installed first]
检查安装成功与否 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/jaeger-operator-metrics ClusterIP 10.43.51.191 <none> 8383/TCP 4h6m service/jaeger-operator-webhook-service ClusterIP 10.43.213.138 <none> 443/TCP 4h6m service/simplest-agent ClusterIP None <none> 5775/UDP,5778/TCP,6831/UDP,6832/UDP,14271/TCP 4h5m service/simplest-collector ClusterIP 10.43.147.4 <none> 9411/TCP,14250/TCP,14267/TCP,14268/TCP,14269/TCP,4317/TCP,4318/TCP 4h5m service/simplest-collector-headless ClusterIP None <none> 9411/TCP,14250/TCP,14267/TCP,14268/TCP,14269/TCP,4317/TCP,4318/TCP 4h5m service/simplest-query ClusterIP 10.43.22.27 <none> 16686/TCP,16685/TCP,16687/TCP 4h5m NAME READY STATUS RESTARTS AGE pod/jaeger-operator-5c4fdd77c6-gpkvc 1/1 Running 0 4h6m pod/simplest-7688c4cf5d-sgj8d 1/1 Running 0 4h5m NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/simplest-query <none> * 80 4h5m
安装好后,会默认创建一个指向80端口的ingress,这个是jaeger ui的端口,访问ip:80
即可访问到此ui,如下
Istio配套的安装方式 istio的安装包里是带有jaeger部署方式的,但是他提供的部署方式是all-in-one,即数据存储是内存,这对于生产环境来说是不合适的,因此,我们可以根据官方的
生产环境的部署方式进行部署,这个在istio的官方文档里也有说明
参考:
https://blog.csdn.net/Mrheiiow/article/details/131109127
https://istio.io/latest/zh/docs/ops/integrations/jaeger/#installation
根据官方文档,先安装cert-manager 参考上文
部署jaeger-operator 根据官方文档操作
1 2 3 4 5 6 7 kubectl create ns observability kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.45.0/jaeger-operator.yaml -n observability kubectl get deployment jaeger-operator -n observability NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE jaeger-operator 1 1 1 1 48s
部署jaeger 以下示例使用的存储是外部的ES,同时参照官方的配置,使用Elasticsearch rollover 的配置方式,然后外部ES是有账号密码的,所以可以使用Secrets Support 配
置账号密码,secrets的具体配置方式可以参见External Elasticsearch 的第四条说明kubectl create secret generic jaeger-secret --from-
literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 $ kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic $ kubectl get secrets -n observability NAME TYPE DATA AGE default-token-fc756 kubernetes.io/service-account-token 3 23h jaeger-operator-service-cert kubernetes.io/tls 3 23h jaeger-operator-token-dmrzc kubernetes.io/service-account-token 3 20h jaeger-secret Opaque 2 22h jaeger-token-n5xfx kubernetes.io/service-account-token 3 20h $ cat jaeger.yaml apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: jaeger namespace: observability spec: strategy: production collector: maxReplicas: 2 resources: limits: cpu: 100m memory: 128Mi storage: type : elasticsearch options: es: server-urls: http://192.168.8.105:9200 use-aliases: true esRollover: conditions: "{\"max_age\": \"2d\"}" readTTL: 168h schedule: "55 23 * * *" secretName: jaeger-secret query: serviceType: NodePort $ kubectl apply -f jaeger.yaml $ kubectl get svc,pod -n observability NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/jaeger-collector ClusterIP 10.233.19.93 <none> 9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP 20h service/jaeger-collector-headless ClusterIP None <none> 9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP 20h service/jaeger-operator-metrics ClusterIP 10.233.29.204 <none> 8443/TCP 20h service/jaeger-operator-webhook-service ClusterIP 10.233.28.228 <none> 443/TCP 20h service/jaeger-query NodePort 10.233.23.105 <none> 16686:32003/TCP,16685:32004/TCP 20h NAME READY STATUS RESTARTS AGE pod/jaeger-collector-c498bfb45-khtrq 1/1 Running 0 20h pod/jaeger-es-index-cleaner-28102555-t4v77 0/1 Completed 0 14h pod/jaeger-es-lookback-28102555-d98x8 0/1 Completed 0 14h pod/jaeger-es-rollover-28102555-2rxlw 0/1 Completed 0 14h pod/jaeger-es-rollover-create-mapping-k4x5r 0/1 Completed 0 20h pod/jaeger-operator-58d97648c5-gr2kx 2/2 Running 0 20h pod/jaeger-query-79754974c7-7gnk9 2/2 Running 0 20h pod/jaeger-spark-dependencies-28102555-dbnxt 0/1 Completed 0 14h
对接istio 到此为止,基本的部署已经完成了,已经可以使用jaeger-query提供的WEB-UI,但是此时却并没有istio相关的tracing信息,需要在istio的配置中定义jaeger-collector的地址,官方给出的方式是在安装(更新)istio的时候定义参数:https://istio.io/latest/zh/docs/ops/integrations/jaeger/#option-2-customizable-install
其实还有一个方法,通过修改configmap来实现:
istiod有一个cm,名字就叫istio,我们只需要在istio中添加或者修改即可
修改完了,并不代表已经能正常使用了,此时需要重启istiod,然后会发现,先前部署的服务的tracing信息还是无法通过jaeger看到,此时需要重
启被istio注入的服务,然后就能看到tracing信息了
对接kiali kiali的好几个菜单都能查看traces,而数据来源就是jaeger,如果我们使用istio自带的jaeger,那么不用任何配置,直接就能看
但现在我们是自己部署的jaejer,就跟istio自带的安装方式有一定的差异,所以我们需要配置一下,具体的方式也很简单,就是修改kiali的configmap
1 kubectl edit cm -n istio-system kiali
然后找到external_services
,在下面添加如下内容
1 2 3 4 5 tracing: url: http://192.168.8.104:32003 in_cluster_url: http://jaeger-query.observability:16685/jaeger