Installing Ceph/RDMA on CentOS 7

Submitted by peter on Mon, 04/06/2018 - 16:56

Start from a fresh install with networking configured. IPv6 is required because rxe_cfg expects it and gives an error if IPv6 isn't present. Can be link local.

Source machine has latest kernel and Ceph installed in /usr/local/ceph[-12.2.5] (soft link)

Set GRUB_DEFAULT in /etc/default/grub to 0

On source machine:
rsync -va /boot/*<KERNERVERSION>* root@<DESTINATIONHOST>:/boot/
rsync -va /lib/modules/<KERNERVERSION> root@<DESTINATIONHOST>:/lib/modules/

Run:
grub2-mkconfig -o /boot/grub2/grub.cfg

On source machine:
rsync -va /usr/local/ceph* root@<DESTINATIONHOST>:/usr/local/
rsync -va /etc/profile.d/ceph.sh root@<DESTINATIONHOST>:/etc/profile.d/
rsync -va /etc/ld.so.conf.d/ceph.conf root@<DESTINATIONHOST>:/etc/ld.so.conf.d/
rsync -va /etc/ceph root@<DESTINATIONHOST>:/etc/

Run:
ldconfig

Run:
groupadd -g 167 ceph
useradd -g ceph -u 167 -M -c "Ceph daemons" -d /var/lib/ceph -s /bin/nologin ceph

Run:
yum -y install epel-release
yum -y update

Run:
yum -y install rdma-core libibverbs libaio fuse-libs leveldb gperftools-libs python-prettytable.noarch libibverbs-utils

Reboot and login again (amongst other things, this sets up the paths via /etc/profile.d/ceph.sh

Run:
rxe_cfg start
rxe_cfg add eth0

Run (so Ceph daemons can set up UNIX socket for communication with ceph CLI program):
mkdir -p /var/run/ceph
chown ceph:ceph /var/run/ceph

Ready to configure Ceph daemons on host (e.g., OSD)
Examine /root/start_ceph_{mon|mgr|osd} on source machine for how to start Ceph daemons

Tags: