高可用k8s集群搭建:搭建etcd集群

  众所周知 kubernetes 使用 etcd 来保存集群数据,官方的 kubeadm 安装方式会创建一个单节点的 etcd 容器,为了保证集群高可用,我们需要搭建一个 etcd 集群,这里使用镜像gcr.io/google_containers/etcd-amd64:3.0.17搭建主要是为了避免使用其他版本带来一些兼容性问题,也是因为 kubernetes 一直使用这个镜像。

系统要求

单节点etcd服务

  在控制台执行以下命令

1
2
3
4
5
6
7
8
9
10
11
12
13
REGISTRY=gcr.io/google_containers/etcd-amd64
ETCD_VERSION=3.0.17
NODE1=127.0.0.1
sudo docker run -d \
-p 2379:2379 \
-p 2380:2380 \
--volume=/var/lib/etcd-data:/etcd-data \
--name etcd ${REGISTRY}:${ETCD_VERSION} \
/usr/local/bin/etcd \
--data-dir=/etcd-data --name node1 \
--initial-advertise-peer-urls http://${NODE1}:2380 --listen-peer-urls http://0.0.0.0:2380 \
--advertise-client-urls http://${NODE1}:2379 --listen-client-urls http://0.0.0.0:2379 \
--initial-cluster node1=http://${NODE1}:2380

验证安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "/usr/local/bin/etcd --version"
etcd Version: 3.0.17
Git SHA: cc198e2
Go Version: go1.6.4
Go OS/Arch: linux/amd64
[niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl version"
etcdctl version: 3.0.17
API version: 3.0
[niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl endpoint health"
127.0.0.1:2379 is healthy: successfully committed proposal: took = 999.973µs
[niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl put foo bar"
OK
[niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl get foo"
foo
bar

三节点etcd集群

机器准备

  • node1 192.168.17.101
  • node2 192.168.17.102
  • node3 192.168.17.103

    分别在每个node上执行以下命令

  • 在 node1 上执行以下命令
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    REGISTRY=gcr.io/google_containers/etcd-amd64
    ETCD_VERSION=3.0.17
    TOKEN=token-etcd-cluster
    CLUSTER_STATE=new
    NAME_1=etcd1
    NAME_2=etcd2
    NAME_3=etcd3
    HOST_1=192.168.17.101
    HOST_2=192.168.17.102
    HOST_3=192.168.17.103
    CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
    DATA_DIR=/home/niuhp/etcd/data

    THIS_NAME=${NAME_1}
    THIS_IP=${HOST_1}

    sudo docker stop etcd || true
    sudo docker rm etcd || true
    sudo docker run -d \
    -p 2379:2379 \
    -p 2380:2380 \
    --volume=${DATA_DIR}:/etcd-data \
    --name etcd ${REGISTRY}:${ETCD_VERSION} \
    /usr/local/bin/etcd \
    --data-dir=/etcd-data --name ${THIS_NAME} \
    --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
    --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
    --initial-cluster ${CLUSTER} \
    --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  • 分别将上述命令中 THIS_NAME=${NAME_1}THIS_IP=${HOST_1} 改为 THIS_NAME=${NAME_2}THIS_IP=${HOST_2}THIS_NAME=${NAME_3}THIS_IP=${HOST_3}并依次在 node2、node3 上执行即可
  • 验证安装,和单节点服务验证类似,分别在三个节点上执行
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    [niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "/usr/local/bin/etcd --version"
    etcd Version: 3.0.17
    Git SHA: cc198e2
    Go Version: go1.6.4
    Go OS/Arch: linux/amd64
    [niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl version"
    etcdctl version: 3.0.17
    API version: 3.0
    [niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl endpoint health"
    127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.54972ms
    另外可以通过在其中一个节点 put 值,在所有节点 get 检查结果是否一致,如在 node1 上执行 put 操作
    1
    2
    [niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl put foo value_from_node1"
    OK
    然后在三个节点上执行 get 操作均可以得到相同的结果value_from_node1,确定集群安装正确
    1
    2
    3
    [niuhp@localhost etcd]$ sudo docker exec etcd /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl get foo"
    foo
    value_from_node1

    参考

  • https://coreos.com/etcd/docs/3.0.17/op-guide/clustering.html
  • https://github.com/coreos/etcd/releases/
  • https://github.com/coreos/etcd/blob/master/Documentation/op-guide/container.md#docker