Installing Ceph/RDMA on CentOS 7

Start from a fresh install with networking configured. IPv6 is required because rxe_cfg expects it and gives an error if IPv6 isn't present. Can be link local.

Source machine has latest kernel and Ceph installed in /usr/local/ceph[-12.2.5] (soft link)

Set GRUB_DEFAULT in /etc/default/grub to 0

On source machine:
rsync -va /boot/*<KERNERVERSION>* root@<DESTINATIONHOST>:/boot/
rsync -va /lib/modules/<KERNERVERSION> root@<DESTINATIONHOST>:/lib/modules/

grub2-mkconfig -o /boot/grub2/grub.cfg

On source machine:
rsync -va /usr/local/ceph* root@<DESTINATIONHOST>:/usr/local/
rsync -va /etc/profile.d/ root@<DESTINATIONHOST>:/etc/profile.d/
rsync -va /etc/ root@<DESTINATIONHOST>:/etc/
rsync -va /etc/ceph root@<DESTINATIONHOST>:/etc/


groupadd -g 167 ceph
useradd -g ceph -u 167 -M -c "Ceph daemons" -d /var/lib/ceph -s /bin/nologin ceph

yum -y install epel-release
yum -y update

yum -y install rdma-core libibverbs libaio fuse-libs leveldb gperftools-libs python-prettytable.noarch libibverbs-utils

Reboot and login again (amongst other things, this sets up the paths via /etc/profile.d/

rxe_cfg start
rxe_cfg add eth0

Run (so Ceph daemons can set up UNIX socket for communication with ceph CLI program):
mkdir -p /var/run/ceph
chown ceph:ceph /var/run/ceph

Ready to configure Ceph daemons on host (e.g., OSD)
Examine /root/start_ceph_{mon|mgr|osd} on source machine for how to start Ceph daemons