参考：https://www.cnsre.cn/posts/210902330007

前言

Service可能会有很多，如果每个资源都绑定一个 node port的话，主机则需要开放外围的端口进行服务调用，管理上会比较混乱。

比较优雅的方式是通过一个外部的负载均衡器，比如 nginx ，绑定固定的端口比如80，然后根据域名/服务名向后面的Service Ip转发，但是这里对问题在于：当有新服务加入的时候如何修改 Nginx 配置？

手动改或者 Rolling Update Nginx Pod 都是不现实的。

对于这个问题， k8s 给出的七层解决方案是：Ingress

ingress-nginx

ingress nginx 官方网站

ingress nginx 仓库地址

ingress-nginx v1.0 最新版本 v1.0

适用于 Kubernetes 版本 v1.19+ （包括 v1.19 ）

Kubernetes-v1.22+ 需要使用 ingress-nginx>=1.0，因为 networking.k8s.io/v1beta 已经移除

直接部署 ingress-nginx

直接部署比较简单，直接拉去 girhub 的文件就可以了，如果遇到长时间无响应，可以终止任务从新拉取。
拉取镜像部分，可以修改为一下的镜像地址

1
2
3
4
5
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0/deploy/static/provider/baremetal/deploy.yaml

sed -i 's@k8s.gcr.io/ingress-nginx/controller:v1.0.0\(.*\)@cnsre/ingress-nginx-controller:v1.0.0@' deploy.yaml
sed -i 's@k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0\(.*\)$@cnsre/ingress-nginx-kube-webhook-certgen:v1.0@' deploy.yaml
kubectl apply -f deploy.yaml

检查安装

Completed 状态的是正常的，可以忽略。

1
2
3
4
5
6
7
8
9
[root@master ~]# kubectl get po -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-pm6sw        0/1     Completed   0          22m
ingress-nginx-admission-patch-m8w94         0/1     Completed   0          22m
ingress-nginx-controller-7d4df87d89-272ft   1/1     Running     0          22m
[root@master ~]# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.96.88.139   <none>        80:30497/TCP,443:32581/TCP   22m
ingress-nginx-controller-admission   ClusterIP   10.96.193.26   <none>        443/TCP                      22m

创建应用yaml

1
vim tomcat.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
apiVersion: apps/v1 
kind: Deployment   
metadata:             
  name: tomcat-deployment     
  labels:       
    app: tomcat  
spec:          
  replicas: 2 
  selector:      
    matchLabels: 
      app: tomcat
  minReadySeconds: 1
  progressDeadlineSeconds: 60
  revisionHistoryLimit: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:        
    metadata:  
      labels:  
        app: tomcat
    spec:         
      containers:     
      - name: tomcat     
        image: wenlongxue/tomcat:tomcat-demo-62-8fe6052    
        imagePullPolicy: Always          
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "2Gi"
            cpu: "80m"
          limits: 
            memory: "2Gi" 
            cpu: "80m"
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 180
          periodSeconds: 5
          timeoutSeconds: 3
          successThreshold: 1
          failureThreshold: 30
---
apiVersion: v1
kind: Service
metadata:      
  name: tomcat-service
  labels:      
    app: tomcat 
spec:        
  selector:   
    app: tomcat  
  ports:
  - name: tomcat-port 
    protocol: TCP      
    port: 8080         
    targetPort: 8080   
  type: ClusterIP

部署 tomcat 应用

1
kubectl  apply  -f  tomcat.yaml

创建 ingress yaml

1
vim tomcat-ingress.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tomcat
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: tomcat.cnsre.cn
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: tomcat-service
            port:
              number: 8080

部署 tomcat ingress yaml

1
kubectl  apply  -f  tomcat-ingress.yaml

查看 ingress 对应节点的端口

1
2
3
4
5
kubectl get svc -n ingress-nginx

NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.96.88.139   <none>        80:30497/TCP,443:32581/TCP   54m
ingress-nginx-controller-admission   ClusterIP   10.96.193.26   <none>        443/TCP                      54m

添加 hosts

在 hosts 文件最后追加 ingress 节点的 IP 地址

1
54.xxx.xxx.xxx tomcat.cnsre.cn

然后在浏览器中访问 tomcat.cnsre.cn:30497。

使用 hostNetwork 的方式部署 ingress-nginx

每次部署 ingres-nginx 都随机一个 nodePort ，而使用 ingres-nginx 访问的时候也要以 域名:端口 的形式去访问如何直接使用域名去访问呢？下面介绍另外一种安装方式。

1
2
3
4
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0/deploy/static/provider/baremetal/deploy.yaml

sed -i 's@k8s.gcr.io/ingress-nginx/controller:v1.0.0\(.*\)@cnsre/ingress-nginx-controller:v1.0.0@' deploy.yaml
sed -i 's@k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0\(.*\)$@cnsre/ingress-nginx-kube-webhook-certgen:v1.0@' deploy.yaml

优化 ingress-nginx

使用 hostNetwork

默认 ingress-nginx 随机提供 nodeport 端口，开启 hostNetwork 启用80、443端口。
修改 Deployment 下面的 spec
参数如下：

1
2
3
4
5
6
7
8
9
10
...
    spec:
      hostNetwork: true # 新增
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: cnsre/ingress-nginx-controller:v1.0.0  # 更换镜像地址
          imagePullPolicy: IfNotPresent
          lifecycle:
...

修改负载均衡问题

把 kind: Deployment 改为 kind: DaemonSet 模式，这样每台 node 上都有 ingress-nginx-controller pod 副本。
参数如下：

1
2
3
4
5
6
7
8
9
...
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
#kind: Deployment   # 注释
kind: DaemonSet     # 新增
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.1
...

修改 ingressClass 问题

如果不关心 ingressClass 或者很多没有 ingressClass 配置的 ingress 对象，
添加参数 ingress-controller --watch-ingress-without-class=true 。

1
2
3
4
5
6
7
8
9
10
11
...
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --controller-class=k8s.io/ingress-nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
            - --watch-ingress-without-class=true  # 新增
...

部署检查 ingress

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 部署 
kubectl apply -f ingress-nginx.yaml
# 检查 pod 
[root@master ~]# kubectl  get  pods -n ingress-nginx  -o wide 
NAME                                   READY   STATUS      RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-gmnmp   0/1     Completed   0          84m   10.100.219.105   master   <none>           <none>
ingress-nginx-admission-patch-f5sgc    0/1     Completed   0          84m   10.100.219.106   master   <none>           <none>
ingress-nginx-controller-b62w7         1/1     Running     0          84m   10.0.10.51       master   <none>           <none>
ingress-nginx-controller-lsn7h         1/1     Running     0          84m   10.0.20.222      node1    <none>           <none>
# 检查端口
[root@master ~]# netstat  -pntl |grep 443 
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      31248/nginx: master 
[root@master ~]# netstat  -pntl |grep 80        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      31248/nginx: master

创建应用yaml

1
vim tomcat.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
apiVersion: apps/v1 
kind: Deployment   
metadata:             
  name: tomcat-deployment     
  labels:       
    app: tomcat  
spec:          
  replicas: 2 
  selector:      
    matchLabels: 
      app: tomcat
  minReadySeconds: 1
  progressDeadlineSeconds: 60
  revisionHistoryLimit: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:        
    metadata:  
      labels:  
        app: tomcat
    spec:         
      containers:     
      - name: tomcat     
        image: wenlongxue/tomcat:tomcat-demo-62-8fe6052    
        imagePullPolicy: Always          
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "2Gi"
            cpu: "80m"
          limits: 
            memory: "2Gi" 
            cpu: "80m"
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 180
          periodSeconds: 5
          timeoutSeconds: 3
          successThreshold: 1
          failureThreshold: 30
---
apiVersion: v1
kind: Service
metadata:      
  name: tomcat-service
  labels:      
    app: tomcat 
spec:        
  selector:   
    app: tomcat  
  ports:
  - name: tomcat-port 
    protocol: TCP      
    port: 8080         
    targetPort: 8080   
  type: ClusterIP

部署 tomcat 应用

1
kubectl  apply  -f  tomcat.yaml

创建 ingress yaml

1
vim tomcat-ingress.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tomcat
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: tomcat.cnsre.cn
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: tomcat-service
            port:
              number: 8080

部署 tomcat ingress yaml

1
kubectl  apply  -f  tomcat-ingress.yaml

添加 hosts

在 hosts 文件最后追加 ingress 节点的 IP 地址

1
54.xxx.xxx.xxx tomcat.cnsre.cn

然后在浏览器中访问 tomcat.cnsre.cn。

给 ingress-nginx 配置 HTTPS 访问

创建自签证书文件

1
openssl req -x509 -nodes -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginx/O=nginx"

创建后会生成两个文件

1
2
3
ll tls.*
-rw-r--r--. 1 root root 1127 9月   2 13:04 tls.crt
-rw-r--r--. 1 root root 1708 9月   2 13:04 tls.key

创建 secret

1
kubectl create secret tls tls-secret --key tls.key --cert tls.crt

修改 tomcat-ingress yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tomcat
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:                      # 新增
  - hosts:                  # 新增
    - tomcat.cnsre.cn       # 新增
    secretName: tls-secret  # 新增
  rules:
  - host: tomcat.cnsre.cn
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: tomcat-service
            port:
              number: 8080

修改完重新部署下

1
kubectl  apply  -f  tomcat-ingress.yaml

验证证书

访问tomcat.cnsre.cn