Goal

To run a ceph distributed storage cluster primarily on CSE lab computers using unused local disk storage to create one OSD per host.

In addition to running ceph-osd, the cluster will also run ceph-mon and ceph-mgr on one host per lab.

conform product

The product is Local.plceph.

Gaol environment

The ceph software (currently ceph-12.1.0) is not compatible with the current CSE conform software set though it is compatible with the kernel. To run, the necessary binaries and their required libraries are located in a chroot gaol at /usr/local/ceph/gaol. Scripts to support the gaol are located in /usr/local/ceph/scripts.

As much as possible, only the files necessary to support the ceph daemons are in the gaol. Also as much as possible, binaries are located (relatively) in /bin and libraries are located in /lib.

Three primary binaries are supported:

1. ceph-osd
2. ceph-mon
3. ceph-mgr

The binaries and libraries are copied from a dedicated, i386, cleanly-installed Debian host.

The ceph cluster operates on CSE conformed hosts with, at most, one OSD daemon per host. The OSD daemon has its datastore on an XFS-formatted filesystem on a loop(back) device whose associated file is /export/<hostname>/1/SCRATCH/plinich/plceph_osd.dat.

See also /etc/fstab.

ceph user

In the gaol environment, programs run as user games. This was chosen because it already existed in /etc/passwd and was not used by anything else. See also the product spec file and /usr/local/ceph/scripts/rungaol.

Create a new MON

The first MON is created using the bootstrap instructions on www.ceph.com.

Subsequent MONs are done like this:

1. On the management host:
2. Use monmaptool to create a new monitor map, with actual fsid, containing the current MON hosts/IP addresses plus the new host
3. Copy the new monitor map to the new MON host as /usr/local/ceph/gaol/monmap
4. On the new MON host:
5. Copy /usr/local/ceph/gaol/var/lib/ceph/mon/ceph-<hostname>/keyring from a running MON host to /usr/local/ceph/gaol/
6. chown games /usr/local/ceph/gaol/monmap
7. /usr/local/ceph/scripts/rungaol /bin/ceph-mon -i <hostname> --mkfs --keyring /keyring --monmap /monmap -f -d
8. mv /usr/local/ceph/gaol/keyring /usr/local/ceph/gaol/var/lib/ceph/mon/ceph-viola09/keyring
9. /usr/local/ceph/scripts/rungaol /bin/ceph-mon -i <hostname>
10. rm /usr/local/ceph/gaol/monmap

Notes

The /usr/local/ceph/gaol/var/lib/ceph/mon/ceph-<hostname>/keyring file is the same on all MON hosts.

Create a new OSD

1. Add host to plceph_cluster_osd class, add xfsprogs package and run conform on host,
2. On OSD host:
3. Create sparse disk file for filesystem using dd if=/dev/zero of=/export/<hostname>/1/SCRATCH/plinich/plceph_osd.dat count=1 bs=1 seek=<SIZE>G
4. losetup /dev/loop0 /export/<hostname>/1/SCRATCH/plinich/plceph_osd.dat
5. mkfs.xfs /dev/loop0
6. losetup -d /dev/loop0
7. mount /usr/local/ceph/gaol/osd
8. chown games:games /usr/local/ceph/gaol/osd
9. On management host:
10. ceph -c /etc/plceph/ceph.conf osd create (outpts new OSD number)
11. On OSD host:
12. /usr/local/ceph/scripts/rungaol /bin/ceph-osd --mkfs --mkkey -i <osdnum> --osd-data /osd --osd-journal /osd/journal
13. Copy /usr/local/ceph/gaol/osd/keyring to management host
14. On management host:
15. ceph -c /etc/plceph/ceph.conf auth add osd.<osdnum> osd 'allow *' mon 'allow profile osd' -i <path-to-keyring-file>
16. On OSD host:
17. Start OSD daemon = /usr/local/ceph/scripts/rungaol /bin/ceph-osd -i <osdnum> --osd-data /osd --osd-journal /osd/journal