Adding a new OSD (Jewel + CentOS 7.2)
This procedure is suitable for adding either a whole disk device or a single partition of a disk.
Prepare the device
Disks or partitions which have previously been a component of a RAID set or which have contained a filesystem should be cleared. The following commands will do the trick depending on what the disk or partition was used for:
sgdisk -Z /dev/{disk-or-partition}; partprobe
mdadm --zero-superblock /dev/{disk-or-partition}
dd if=/dev/zero of=/dev/{disk-or-partition} bs=4096 count=10240
Procedure
Perform as root on the host where the OSD will live.
Create a new logical OSD:
ceph osd create
The above will return the number of the new OSD.
Create the mount point for the new OSD:
mkdir /var/lib/ceph/osd/ceph-{osd-number}
Format the disk or partition and mount it:
mkfs -t xfs /dev/{disk-or-partition}
mount -t xfs -o noatime,inode64 /dev/{disk-or-partition} /var/lib/ceph/osd/ceph-{osd-number}
Initialize the top-level directory of the OSD:
ceph-osd -i {osd-number} --mkfs --mkkey
Register the OSD's key:
ceph auth add osd.{osd-number} osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-{osd-number}/keyring
Add the OSD to the CRUSH map:
ceph osd crush add {osd-number} {weight} host={host}
It can be a good idea to start with the {weight} set to 0.0 tp prevent premature migration starting.
{host} is the name of the host on which the OSD lives. This determines where the OSD appears in the OSD tree.
Set ownership of OSD:
chown -R ceph:ceph /var/lib/ceph/osd/ceph-{osd-number}
Get the OSD filesystem UUID:
blkid /dev/{disk-or-partition}
Add the following to /etc/fstab to ensure the OSD mounts at boot time:
UUID={uuid} /var/lib/ceph/osd/ceph-{osd-number} xfs noatime,inode64 1 2
Create the file nice.conf in /usr/lib/systemd/system/ceph-osd\@.service.d/:
[Service]
Nice=-5
This ensures that the OSD daemon processes get timely access to CPU and I/O resources.
Start the OSD daemon:
systemctl start ceph-osd\@{osd-number}
systemctl enable ceph-osd\@{osd-number}