Distributed ceph @ CSE notes


To run a ceph distributed storage cluster primarily on CSE lab computers using unused local disk storage to create one OSD per host.

In addition to running ceph-osd, the cluster will also run ceph-mon and ceph-mgr on one host per lab.

conform product

The product is Local.plceph.

Gaol environment

The ceph software (currently ceph-12.1.0) is not compatible with the current CSE conform software set though it is compatible with the kernel. To run, the necessary binaries and their required libraries are located in a chroot gaol at /usr/local/ceph/gaol. Scripts to support the gaol are located in /usr/local/ceph/scripts.

As much as possible, only the files necessary to support the ceph daemons are in the gaol. Also as much as possible, binaries are located (relatively) in /bin and libraries are located in /lib.

Three primary binaries are supported:

  1. ceph-osd
  2. ceph-mon
  3. ceph-mgr

The binaries and libraries are copied from a dedicated, i386, cleanly-installed Debian host.

The ceph cluster operates on CSE conformed hosts with, at most, one OSD daemon per host. The OSD daemon has its datastore on an XFS-formatted filesystem on a loop(back) device whose associated file is /export/<hostname>/1/SCRATCH/plinich/plceph_osd.dat.

See also /etc/fstab.

ceph user

In the gaol environment, programs run as user games. This was chosen because it already existed in /etc/passwd and was not used by anything else. See also the product spec file and /usr/local/ceph/scripts/rungaol.

Create a new MON

The first MON is created using the bootstrap instructions on www.ceph.com.

Subsequent MONs are done like this:

  1. On the management host:
  2. Use monmaptool to create a new monitor map, with actual fsid, containing the current MON hosts/IP addresses plus the new host
  3. Copy the new monitor map to the new MON host as /usr/local/ceph/gaol/monmap
  4. On the new MON host:
  5. Copy /usr/local/ceph/gaol/var/lib/ceph/mon/ceph-<hostname>/keyring from a running MON host to /usr/local/ceph/gaol/
  6. chown games /usr/local/ceph/gaol/monmap
  7. /usr/local/ceph/scripts/rungaol /bin/ceph-mon -i <hostname> --mkfs --keyring /keyring --monmap /monmap -f -d
  8. mv /usr/local/ceph/gaol/keyring /usr/local/ceph/gaol/var/lib/ceph/mon/ceph-viola09/keyring
  9. /usr/local/ceph/scripts/rungaol /bin/ceph-mon -i <hostname>
  10. rm /usr/local/ceph/gaol/monmap


The /usr/local/ceph/gaol/var/lib/ceph/mon/ceph-<hostname>/keyring file is the same on all MON hosts.

Create a new OSD

  1. Add host to plceph_cluster_osd class, add xfsprogs package and run conform on host,
  2. On OSD host:
  3. Create sparse disk file for filesystem using dd if=/dev/zero of=/export/<hostname>/1/SCRATCH/plinich/plceph_osd.dat count=1 bs=1 seek=<SIZE>G
  4. losetup /dev/loop0 /export/<hostname>/1/SCRATCH/plinich/plceph_osd.dat
  5. mkfs.xfs /dev/loop0
  6. losetup -d /dev/loop0
  7. mount /usr/local/ceph/gaol/osd
  8. chown games:games /usr/local/ceph/gaol/osd
  9. On management host:
  10. ceph -c /etc/plceph/ceph.conf osd create (outpts new OSD number)
  11. On OSD host:
  12. /usr/local/ceph/scripts/rungaol /bin/ceph-osd --mkfs --mkkey -i <osdnum> --osd-data /osd --osd-journal /osd/journal
  13. Copy /usr/local/ceph/gaol/osd/keyring to management host
  14. On management host:
  15. ceph -c /etc/plceph/ceph.conf auth add osd.<osdnum> osd 'allow *' mon 'allow profile osd' -i <path-to-keyring-file>
  16. On OSD host:
  17. Start OSD daemon = /usr/local/ceph/scripts/rungaol /bin/ceph-osd -i <osdnum> --osd-data /osd --osd-journal /osd/journal

See also /etc/plceph/create_new_osd_on_host on management host for script implementation.

Creating a new MGR

ceph-mgr should run on the same hosts as ceph-mon.

  1. On MGR host:
  2. Create the directory /usr/local/ceph/gaol/var/lib/ceph/mgr/ceph-<hostname>. Make it root-owned with it and its parent directories 755.
  3. On management host:
  4. ceph -c /etc/plceph/ceph.conf auth get-or-create mgr.<hostname> mon 'allow profile mgr' osd 'allow *' mds 'allow *' > /tmp/keyring.
  5. Copy /tmp/keyring to MGR host directory created above. chown root:games plus chmod 640.
  6. On MGR host:
  7. /usr/local/ceph/scripts/rungaol /bin/ceph-mgr -i <hostname>