标签：clust ash sts member 磁盘分区 keygen physical === health
之前已详细介绍了Ceph分布式存储基础知识，下面简单记录下Centos7使用Ceph-deploy快速部署Ceph环境：
1）基本环境
192.168.10.220 ceph-admin ceph-deploy、mon1（也可以将monit节点另放一台机器） 192.168.10.239 ceph-node1 osd1 192.168.10.212 ceph-node2 osd2 192.168.10.213 ceph-node3 osd3 ------------------------------------------------- 每个节点修改修改主机名 # hostnamectl set-hostname ceph-admin # hostnamectl set-hostname ceph-node1 # hostnamectl set-hostname ceph-node2 # hostnamectl set-hostname ceph-node3 ------------------------------------------------- 每个节点修改配置文件 # cat /etc/hosts 192.168.10.220 ceph-admin 192.168.10.239 ceph-node1 192.168.10.212 ceph-node2 192.168.10.213 ceph-node3 ------------------------------------------------- 每个节点确认连通性 # ping -c 3 ceph-admin # ping -c 3 ceph-node1 # ping -c 3 ceph-node2 # ping -c 3 ceph-node3 ------------------------------------------------- 每个节点关闭防火墙和selinux # systemctl stop firewalld # systemctl disable firewalld # sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g‘ /etc/selinux/config # setenforce 0 ------------------------------------------------- 每个节点安装和配置NTP（官方推荐的是集群的所有节点全部安装并配置 NTP，需要保证各节点的系统时间一致。没有自己部署ntp服务器，就在线同步NTP） # yum install ntp ntpdate ntp-doc -y # systemctl restart ntpd # systemctl status ntpd ------------------------------------------------- 每个节点准备yum源 删除默认的源，国外的比较慢 # yum clean all # mkdir /mnt/bak # mv /etc/yum.repos.d/* /mnt/bak/ 下载阿里云的base源和epel源 # wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo # wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo 添加ceph源 # vim /etc/yum.repos.d/ceph.repo [ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ gpgcheck=0 priority =1 [ceph-noarch] name=cephnoarch baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/ gpgcheck=0 priority =1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS gpgcheck=0 priority=1 ------------------------------------------------------------ 每个节点创建cephuser用户，设置sudo权限 # useradd -d /home/cephuser -m cephuser # echo "cephuser"|passwd --stdin cephuser # echo "cephuser ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephuser # chmod 0440 /etc/sudoers.d/cephuser # sed -i s‘/Defaults requiretty/#Defaults requiretty‘/g /etc/sudoers 测试cephuser的sudo权限 # su - cephuser $ sudo su - # ------------------------------------------------------------ 配置相互间的ssh信任关系 现在ceph-admin节点上产生公私钥文件，然后将ceph-admin节点的.ssh目录拷贝给其他节点 [root@ceph-admin ~]# su - cephuser [cephuser@ceph-admin ~]$ ssh-keygen -t rsa #一路回车 [cephuser@ceph-admin ~]$ cd .ssh/ [cephuser@ceph-admin .ssh]$ ls id_rsa id_rsa.pub [cephuser@ceph-admin .ssh]$ cp id_rsa.pub authorized_keys [cephuser@ceph-admin .ssh]$ scp -r /home/cephuser/.ssh ceph-node1:/home/cephuser/ [cephuser@ceph-admin .ssh]$ scp -r /home/cephuser/.ssh ceph-node2:/home/cephuser/ [cephuser@ceph-admin .ssh]$ scp -r /home/cephuser/.ssh ceph-node3:/home/cephuser/ 然后在各节点直接验证cephuser用户下的ssh相互信任关系 $ ssh -p22 cephuser@ceph-admin $ ssh -p22 cephuser@ceph-node1 $ ssh -p22 cephuser@ceph-node2 $ ssh -p22 cephuser@ceph-node3
2）准备磁盘（ceph-node1、ceph-node2、ceph-node3三个节点）
测试时使用的磁盘不要太小，否则后面添加磁盘时会报错，建议磁盘大小为20G及以上。 三个节点均是在WebvirtMgr上创建的虚拟机，参考：https://www.cnblogs.com/kevingrace/p/8387999.html 文档中创建并挂载虚拟机磁盘。 如下分别在三个节点挂载了一块20G的裸盘 检查磁盘 $ sudo fdisk -l /dev/vdb Disk /dev/vdb: 21.5 GB, 21474836480 bytes, 41943040 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes 格式化磁盘 $ sudo parted -s /dev/vdb mklabel gpt mkpart primary xfs 0% 100% $ sudo mkfs.xfs /dev/vdb -f 查看磁盘格式（xfs格式） $ sudo blkid -o value -s TYPE /dev/vdb
3）部署阶段（ceph-admin节点上使用ceph-deploy快速部署）
[root@ceph-admin ~]# su - cephuser 安装ceph-deploy [cephuser@ceph-admin ~]$ sudo yum update -y && sudo yum install ceph-deploy -y 创建cluster目录 [cephuser@ceph-admin ~]$ mkdir cluster [cephuser@ceph-admin ~]$ cd cluster/ 创建集群（后面填写monit节点的主机名，这里monit节点和管理节点是同一台机器，即ceph-admin） [cephuser@ceph-admin cluster]$ ceph-deploy new ceph-admin ......... [ceph-admin][DEBUG ] IP addresses found: [u‘192.168.10.220‘] [ceph_deploy.new][DEBUG ] Resolving host ceph-admin [ceph_deploy.new][DEBUG ] Monitor ceph-admin at 192.168.10.220 [ceph_deploy.new][DEBUG ] Monitor initial members are [‘ceph-admin‘] [ceph_deploy.new][DEBUG ] Monitor addrs are [‘192.168.10.220‘] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf... 修改ceph.conf文件（注意：mon_host必须和public network 网络是同网段内！） [cephuser@ceph-admin cluster]$ vim ceph.conf #添加下面两行配置内容 ...... public network = 192.168.10.220/24 osd pool default size = 3 安装ceph（过程有点长，需要等待一段时间....） [cephuser@ceph-admin cluster]$ ceph-deploy install ceph-admin ceph-node1 ceph-node2 ceph-node3 初始化monit监控节点，并收集所有密钥 [cephuser@ceph-admin cluster]$ ceph-deploy mon create-initial [cephuser@ceph-admin cluster]$ ceph-deploy gatherkeys ceph-admin 添加OSD到集群 检查OSD节点上所有可用的磁盘 [cephuser@ceph-admin cluster]$ ceph-deploy disk list ceph-node1 ceph-node2 ceph-node3 使用zap选项删除所有osd节点上的分区 [cephuser@ceph-admin cluster]$ ceph-deploy disk zap ceph-node1:/dev/vdb ceph-node2:/dev/vdb ceph-node3:/dev/vdb 准备OSD（使用prepare命令） [cephuser@ceph-admin cluster]$ ceph-deploy osd prepare ceph-node1:/dev/vdb ceph-node2:/dev/vdb ceph-node3:/dev/vdb 激活OSD（注意由于ceph对磁盘进行了分区，/dev/vdb磁盘分区为/dev/vdb1） [cephuser@ceph-admin cluster]$ ceph-deploy osd activate ceph-node1:/dev/vdb1 ceph-node2:/dev/vdb1 ceph-node3:/dev/vdb1 --------------------------------------------------------------------------------------------- 可能出现下面的报错： [ceph-node1][WARNIN] ceph_disk.main.Error: Error: /dev/vdb1 is not a directory or block device [ceph-node1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/vdb1 但是这个报错没有影响ceph的部署，在三个osd节点上通过命令已显示磁盘已成功mount，比如： [cephuser@ceph-node1 ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 4.2G 0 rom vda 252:0 0 60G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 59G 0 part ├─centos-root 253:0 0 37G 0 lvm / ├─centos-swap 253:1 0 3.9G 0 lvm [SWAP] └─centos-home 253:2 0 18.1G 0 lvm /home vdb 252:32 0 20G 0 disk ├─vdb1 252:33 0 15G 0 part /var/lib/ceph/osd/ceph-1 └─vdb2 252:34 0 5G 0 part --------------------------------------------------------------------------------------------- 查看OSD [cephuser@ceph-admin cluster]$ ceph-deploy disk list ceph-node1 ceph-node2 ceph-node3 ....... [ceph-node3][DEBUG ] /dev/vdb2 ceph journal, for /dev/vdb1 #如下显示这两个分区，则表示成功了。 [ceph-node3][DEBUG ] /dev/vdb1 ceph data, active, cluster ceph, osd.1, journal /dev/vdb2 用ceph-deploy把配置文件和admin密钥拷贝到管理节点和Ceph节点，这样你每次执行Ceph命令行时就无需指定monit节点地址 和ceph.client.admin.keyring了 [cephuser@ceph-admin cluster]$ ceph-deploy admin ceph-admin ceph-node1 ceph-node2 ceph-node3 修改密钥权限 [cephuser@ceph-admin cluster]$ sudo chmod 644 /etc/ceph/ceph.client.admin.keyring 检查ceph状态 [cephuser@ceph-admin cluster]$ sudo ceph health [cephuser@ceph-admin cluster]$ sudo ceph -s 自此，Ceph分部署存储环境就搭建完成了。
==========================清除ceph存储======================
清除安装包 [cephuser@ceph-admin ~]$ ceph-deploy purge ceph1 ceph2 ceph3 清除配置信息 [cephuser@ceph-admin ~]$ ceph-deploy purgedata ceph1 ceph2 ceph3 [cephuser@ceph-admin ~]$ ceph-deploy forgetkeys 每个节点删除残留的配置文件 [cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/osd/* [cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/mon/* [cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/mds/* [cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/bootstrap-mds/* [cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/bootstrap-osd/* [cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/bootstrap-mon/* [cephuser@ceph-admin ~]$ sudo rm -rf /var/lib/ceph/tmp/* [cephuser@ceph-admin ~]$ sudo rm -rf /etc/ceph/* [cephuser@ceph-admin ~]$ sudo rm -rf /var/run/ceph/*
Centos7下使用Ceph-deploy快速部署Ceph分布式存储-操作记录
标签：clust ash sts member 磁盘分区 keygen physical === health
原文地址：https://www.cnblogs.com/kevingrace/p/9141432.html
