在 K8s 集群上部署 RabbitMQ 实战-六虎

作者：老Z，云原生爱好者，现在专注于云原生运维，KubeSphere Ambassador。

前言

常识点

定级：入门级
RabbitMQ 单节点装置布置
RabbitMQ 集群装置布置
GitOps 运维思想

演示服务器装备

主机名	IP	CPU	内存	系统盘	数据盘	用途
zdeops-master	192.168.9.9	2	4	40	200	Ansible 运维操控节点
ks-k8s-master-0	192.168.9.91	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-1	192.168.9.92	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-2	192.168.9.93	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
storage-node-0	192.168.9.95	2	8	40	200+200	ElasticSearch/GlusterFS
storage-node-0	192.168.9.96	2	8	40	200+200	ElasticSearch/GlusterFS
storage-node-0	192.168.9.97	2	8	40	200+200	ElasticSearch/GlusterFS
harbor	192.168.9.89	2	8	40	200	Harbor
合计	8	22	84	320	2800

演示环境触及软件版本信息

操作系统：CentOS-7.9-x86_64
Ansible：2.8.20
KubeSphere：3.3.0
Kubernetes：v1.24.1
Rook：v1.9.7
Ceph： v16.2.9
GlusterFS：9.5.1
ElasticSearch：7.17.5
Harbor：2.5.1
RabbitMQ：3.9.22
RabbitMQ Cluster Operator：1.14.0

简介

RabbitMQ 单节点如安在 K8s 集群上布置？RabbitMQ 集群如安在 K8s 集群上布置？60分钟带你实战入门。

单节点 RabbitMQ 布置

思路整理

StatefulSet
Headless Service：内部服务用
External Service：外部办理用
Secrets：办理用户名和暗码
Image：DockerHub 官方提供的 rabbitmq:3.9.22-management(带 management 插件)

预备离线镜像

此过程为可选项，离线内网环境可用，假如不装备内网镜像，后续的资源装备清单中留意更改容器的 image 为默认值。

在一台能一起拜访互联网和内网 Harbor 库房的服务器上进行下面的操作。

下载镜像

docker pull rabbitmq:3.9.22-management

从头打 tag

docker tag rabbitmq:3.9.22-management registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

推送到私有镜像库房

docker push registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

整理暂时镜像

docker rmi rabbitmq:3.9.22-management
docker rmi registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

资源装备清单

rabbitmq-secret.yaml

---
kind: Secret
apiVersion: v1
metadata:
  name: rabbitmq-secret
  namespace: zdevops
data:
  pass: UEA4OHcwcmQ=
  user: YWRtaW4=
type: Opaque

rabbitmq-sts.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: zdevops
  labels:
    app: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rabbitmq
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rabbitmq
          image: 'registry.zdevops.com.cn/library/rabbitmq:3.9.22-management'
          ports:
            - name: tcp-5672
              containerPort: 5672
              protocol: TCP
            - name: http-15672
              containerPort: 15672
              protocol: TCP
          env:
            - name: RABBITMQ_DEFAULT_USER
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: user
            - name: RABBITMQ_DEFAULT_PASS
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: pass
          resources:
            limits:
              cpu: '2'
              memory: 4000Mi
            requests:
              cpu: 100m
              memory: 500Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
  serviceName: rabbitmq-headless
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-headless
  namespace: zdevops
  labels:
    app: rabbitmq
spec:
  ports:
    - name: tcp-rabbitmq-5672
      protocol: TCP
      port: 5672
      targetPort: 5672
  selector:
    app: rabbitmq
  clusterIP: None
  type: ClusterIP

rabbitmq-external.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-external
  namespace: zdevops
  labels:
    app: rabbitmq-external
spec:
  ports:
    - name: http-rabbitmq-external
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app: rabbitmq
  type: NodePort

GitOps

在运维开发服务器上操作

# 在已有代码库房创立 rabbitmq/single 目录
[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/single

# 编辑资源装备清单
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-secret.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-sts.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-external.yaml

# 提交 Git
[root@zdevops-master k8s-yaml]# git add rabbitmq
[root@zdevops-master k8s-yaml]# git commit -am '增加rabbitmq 单节点资源装备清单'
[root@zdevops-master k8s-yaml]# git push

布置资源

在运维办理服务器上操作

更新镜像库房代码

[root@zdevops-master k8s-yaml]# git pull

布置资源

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/single/

验证

Secret

[root@zdevops-master k8s-yaml]# kubectl get secret -n zdevops
NAME              TYPE     DATA   AGE
rabbitmq-secret   Opaque   2      8s

StatefulSet

[root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevops
NAME       READY   AGE   CONTAINERS   IMAGES
rabbitmq   1/1     25s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

Pods

[root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevops
NAME         READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
rabbitmq-0   1/1     Running   0          26s   10.233.87.13   ks-k8s-master-1   <none>           <none>

Service

[root@zdevops-master k8s-yaml]# kubectl get svc -o wide -n zdevops
NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE   SELECTOR
rabbitmq-external   NodePort    10.233.4.224   <none>        15672:31672/TCP   36s   app=rabbitmq
rabbitmq-headless   ClusterIP   None           <none>        5672/TCP          36s   app=rabbitmq

图形化办理界面

整理资源

整理 StatefulSet

[root@zdevops-master k8s-yaml]# kubectl delete sts rabbitmq -n zdevops

整理服务

[root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-external rabbitmq-headless -n zdevops

集群形式 RabbitMQ 布置

思路整理

运用官方提供的 RabbitMQ Cluster Operator for Kubernetes。

Open source RabbitMQ Cluster Kubernetes Operator by VMware。

官方文档快速布置

官方提供了快速布置的比如，只需要两步，这儿咱们仅作参阅，后面咱们参阅内网离线镜像布置的方案。

布置 RabbitMQ Cluster Operator

kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

布置 RabbitMQ Cluster

kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

预备离线镜像

此过程为可选项，离线内网环境可用，假如不装备内网镜像，后续的资源装备清单中留意更改容器的 image 为默认值。

在一台能一起拜访互联网和内网 Harbor 库房的服务器上进行下面的操作。

在 Harbor 中创立项目

curl -u "admin:Harbor12345" -X POST -H "Content-Type: application/json" https://registry.zdevops.com.cn/api/v2.0/projects -d '{ "project_name": "rabbitmqoperator", "public": true}'

下载镜像

docker pull rabbitmqoperator/cluster-operator:1.14.0

从头打 tag

docker tag rabbitmqoperator/cluster-operator:1.14.0 registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

推送到私有镜像库房

docker push registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

整理暂时镜像

docker rmi rabbitmqoperator/cluster-operator:1.14.0
docker rmi registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

资源装备清单

经过官网获取 RabbitMQ Cluster Operator 布置资源装备清单「cluster-operator.yml」

wget https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

修改 RabbitMQ Cluster Operator image 为内网镜像

sed -i 's#rabbitmqoperator#registry.zdevops.com.cn/rabbitmqoperator#g' cluster-operator.yml

RabbitMQ Cluster 布置资源清单「rabbitmq-cluster.yaml」

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  namespace: zdevops
  name: rabbitmq-cluster
  labels:
    app: rabbitmq-cluster
spec:
  replicas: 3
  image: registry.zdevops.com.cn/library/rabbitmq:3.9.22-management
  resources:
    limits:
      cpu: 2
      memory: 4Gi
    requests:
      cpu: 100m
      memory: 500Mi
  rabbitmq:
    additionalConfig: |
      default_user=admin
      default_pass=P@88w0rd

更多装备参数和装备示例，请参阅官方文档

办理页面的外部拜访服务 rabbitmq-cluster-external.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-cluster-external
  namespace: zdevops
  labels:
    app: rabbitmq-cluster-external
spec:
  ports:
    - name: management
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app.kubernetes.io/name: rabbitmq-cluster
  type: NodePort

GitOps

在运维开发服务器上操作

# 在已有代码库房创立 rabbitmq/cluster 目录
[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/cluster

# 编辑资源装备清单
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/cluster-operator.yml
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster-external.yaml

# 提交 Git
[root@zdevops-master k8s-yaml]# git add rabbitmq/cluster
[root@zdevops-master k8s-yaml]# git commit -am '增加 rabbitmq 集群形式布置资源装备清单'
[root@zdevops-master k8s-yaml]# git push

布置资源

在运维办理服务器上操作

`更新镜像库房代码

[root@zdevops-master k8s-yaml]# git pull

布置 RabbitMQ Cluster Operator

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/cluster-operator.yml

布置 RabbitMQ Cluster

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster.yaml

布置办理页面外部拜访服务

[root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster-external.yaml

验证

RabbitMQ Cluster Operator Deployment

[root@zdevops-master k8s-yaml]# kubectl get deployments -n rabbitmq-system -o wide
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                             SELECTOR
rabbitmq-cluster-operator   1/1     1            1           107m   operator     registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0   app.kubernetes.io/name=rabbitmq-cluster-operator

RabbitmqClusters

[root@zdevops-master k8s-yaml]# kubectl get rabbitmqclusters -n zdevops
NAME               ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmq-cluster   False              Unknown            23s

StatefulSet

[root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevops
NAME                      READY   AGE   CONTAINERS   IMAGES
rabbitmq-cluster-server   3/3     74s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

Pods

[root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevops
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE              NOMINATED NODE   READINESS GATES
rabbitmq-cluster-server-0   1/1     Running   0          84s   10.233.116.26   ks-k8s-master-2   <none>           <none>
rabbitmq-cluster-server-1   1/1     Running   0          83s   10.233.117.28   ks-k8s-master-0   <none>           <none>
rabbitmq-cluster-server-2   1/1     Running   0          82s   10.233.87.31    ks-k8s-master-1   <none>           <none>

Services

[root@zdevops-master k8s-yaml]# kubectl get svc -n zdevops -o wide
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE    SELECTOR
rabbitmq-cluster            ClusterIP   10.233.56.153   <none>        15692/TCP,5672/TCP,15672/TCP   107s   app.kubernetes.io/name=rabbitmq-cluster
rabbitmq-cluster-external   NodePort    10.233.63.84    <none>        15672:31672/TCP                74m    app.kubernetes.io/name=rabbitmq-cluster
rabbitmq-cluster-nodes      ClusterIP   None            <none>        4369/TCP,25672/TCP             107s   app.kubernetes.io/name=rabbitmq-cluster

图形化办理界面

从办理界面中可以看到一个三节点的集群

整理资源

整理 RabbitmqClusters

[root@zdevops-master k8s-yaml]# kubectl delete rabbitmqclusters rabbitmq-cluster -n zdevops

整理办理页面外部服务

[root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-cluster-external -n zdevops

结束语

本系列文档是我在云原生技能领域的学习和运维实践的手记，用输出倒逼输入是一种高效的学习方法，可以快速堆集经验和提高技能，只要把学到的常识写出来并可以让其他人了解，才能说明真实掌握了这项常识。

本系列文档内容涵盖 (但不限于) 以下技能领域：

KubeSphere
Kubernetes
Ansible
自动化运维
CNCF 技能栈

假如你喜爱本文，请分享给你的小伙伴！

Get 文档

Github：github.com/devops/z-no…
Gitee：gitee.com/zdevops/z-n…
知乎：www.zhihu.com/people/zdev…

Get 代码

Github：github.com/devops/ansi…
Gitee：gitee.com/zdevops/ans…

Get 视频 B 站

老 Z 手记 space.bilibili.com/1039301316

本文由博客一文多发平台 OpenWrite 发布！

在 K8s 集群上部署 RabbitMQ 实战

前言

简介

单节点 RabbitMQ 布置

思路整理

预备离线镜像

资源装备清单

GitOps

布置资源

验证

整理资源

集群形式 RabbitMQ 布置

思路整理

官方文档快速布置

预备离线镜像

资源装备清单

GitOps

布置资源

验证

整理资源

结束语

相关文章

第一次录制B站视频，讲解自己的开源项目

产品经理：你能不能在网页里实现裸眼3D

年度大促将至，企业如何进行性能压测

iOS项目接入Flutter Module探索

作者信息