Building Ceph mimic (13.2.0) on ARM (notes and optimism)

Hardware: Odroid-XU4
OS: Arch Linux ARM (rolling release)
gcc: 8.1
cmake: 3.11.4

Current running make ceph-osd. We'll see what happens...

Not good. It seems there are problems with min() and max() templates. See two errors below. Initially the problem showed up in the Bluestore code, but I tried again with -DWITH_BLUESTORE=OFF and the problem appeared elsewhere as well.

/mnt/build/ceph-13.2.0/src/osd/PrimaryLogPG.cc: In member function 'virtual void PrimaryLogPG::calc_trim_to()':
/mnt/build/ceph-13.2.0/src/osd/PrimaryLogPG.cc:1632:38: error: no matching function for call to 'min(size_t, uint64_t&)'
       cct->_conf->osd_pg_log_trim_max);
                                      ^
In file included from /usr/include/c++/8.1.0/algorithm:61,
                 from /mnt/build/ceph-13.2.0/build/boost/include/boost/core/swap.hpp:25,
                 from /mnt/build/ceph-13.2.0/build/boost/include/boost/utility/swap.hpp:15,
                 from /mnt/build/ceph-13.2.0/build/boost/include/boost/tuple/detail/tuple_basic.hpp:40,
                 from /mnt/build/ceph-13.2.0/build/boost/include/boost/tuple/tuple.hpp:28,
                 from /mnt/build/ceph-13.2.0/src/osd/PrimaryLogPG.cc:18:
/usr/include/c++/8.1.0/bits/stl_algobase.h:195:5: note: candidate: 'template<class _Tp> constexpr const _Tp& std::min(const _Tp&, const _Tp&)'
     min(const _Tp& __a, const _Tp& __b)
     ^~~
/usr/include/c++/8.1.0/bits/stl_algobase.h:195:5: note:   template argument deduction/substitution failed:
/mnt/build/ceph-13.2.0/src/osd/PrimaryLogPG.cc:1632:38: note:   deduced conflicting types for parameter 'const _Tp' ('unsigned int' and 'uint64_t' {aka 'long long unsigned int'})
       cct->_conf->osd_pg_log_trim_max);
                                      ^

Here are the packages which I installed:

[2018-06-21 09:05] [ALPM] installed binutils (2.30-4)
[2018-06-21 09:05] [ALPM] installed libmpc (1.1.0-1)
[2018-06-21 09:05] [ALPM] installed gcc (8.1.0-1)
[2018-06-21 09:17] [ALPM] installed shared-mime-info (1.9-1)
[2018-06-21 09:17] [ALPM] installed jsoncpp (1.8.4-2)
[2018-06-21 09:17] [ALPM] installed libuv (1.20.3-1)
[2018-06-21 09:17] [ALPM] installed rhash (1.3.6-1)
[2018-06-21 09:18] [ALPM] installed cmake (3.11.4-1)
[2018-06-21 09:18] [ALPM] installed libtool (2.4.6+40+g6ca5e224-6)
[2018-06-21 09:18] [ALPM] installed libatomic_ops (7.6.4-1)
[2018-06-21 09:18] [ALPM] installed gc (7.6.6-1)
[2018-06-21 09:18] [ALPM] installed guile (2.2.3-1)
[2018-06-21 09:18] [ALPM] installed make (4.2.1-2)
[2018-06-21 09:58] [ALPM] installed python-markupsafe (1.0-1)
[2018-06-21 09:58] [ALPM] installed python-jinja (2.10-1)
[2018-06-21 09:58] [ALPM] installed python-pygments (2.2.0-1)
[2018-06-21 09:58] [ALPM] installed python-docutils (0.14-1)
[2018-06-21 09:58] [ALPM] installed python-sphinx_rtd_theme (0.4.0-1)
[2018-06-21 09:58] [ALPM] installed python-sphinx-alabaster-theme (0.7.10-1)
[2018-06-21 09:58] [ALPM] installed python-babel (2.6.0-1)
[2018-06-21 09:58] [ALPM] installed python-snowballstemmer (1.2.1-2)
[2018-06-21 09:58] [ALPM] installed python-imagesize (1.0.0-1)
[2018-06-21 09:58] [ALPM] installed python-sphinxcontrib-websupport (1.1.0-1)
[2018-06-21 09:58] [ALPM] installed python-sqlalchemy (1.2.8-1)
[2018-06-21 09:58] [ALPM] installed python-whoosh (2.7.4-2)
[2018-06-21 09:58] [ALPM] installed python-sphinx (1.7.5-1)
[2018-06-21 10:00] [ALPM] installed gperftools (2.7-1)
[2018-06-21 10:00] [ALPM] installed snappy (1.1.7-1)
[2018-06-21 10:00] [ALPM] installed leveldb (1.20-1)
[2018-06-21 10:03] [ALPM] installed nspr (4.19-1)
[2018-06-21 10:03] [ALPM] installed nss (3.37.3-1)
[2018-06-21 10:05] [ALPM] installed liburcu (0.10.1-1)
[2018-06-21 10:05] [ALPM] installed lttng-ust (2.10.1-1)
[2018-06-21 10:05] [ALPM] installed libxslt (1.1.32+3+g32c88216-1)
[2018-06-21 10:05] [ALPM] installed xmlsec (1.2.26-1)
[2018-06-21 10:05] [ALPM] installed oath-toolkit (2.6.2-3)
[2018-06-21 10:06] [ALPM] installed elfutils (0.171-1)
[2018-06-21 10:06] [ALPM] installed babeltrace (1.5.3-1)
[2018-06-21 10:07] [ALPM] installed cython (0.28.3-2)
[2018-06-21 10:07] [ALPM] installed yasm (1.3.0-2)
[2018-06-21 10:10] [ALPM] installed pkg-config (0.29.2-1)
[2018-06-21 10:19] [ALPM] installed gperf (3.1-1)

Here's the script I used to do the configuration:

#!/bin/sh

clear && printf '\e[3J'

rm -fr build || exit 1

export PATH="/mnt/build/tempbin:$PATH"
./do_cmake.sh -DCMAKE_INSTALL_PREFIX=/usr/local/ceph-13.2.0 -DWITH_TESTS=OFF -DWITH_RDMA=OFF -DWITH_PYTHON3=ON -DWITH_FUSE=OFF

/proc/cpuinfo:

[root@odroid1 ~]# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 90.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xc07
CPU revision	: 3

processor	: 1
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 90.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xc07
CPU revision	: 3

processor	: 2
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 90.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xc07
CPU revision	: 3

processor	: 3
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 90.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xc07
CPU revision	: 3

processor	: 4
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 96.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 3

processor	: 5
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 96.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 3

processor	: 6
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 96.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 3

processor	: 7
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 96.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 3

Hardware	: ODROID-XU4
Revision	: 0100
Serial		: 0000000000000000

uname -a:

[root@odroid1 ~]# uname -a
Linux odroid1 4.14.50-1-ARCH #1 SMP PREEMPT Tue Jun 19 00:29:56 UTC 2018 armv7l GNU/Linux
[root@odroid1 ~]#

Notes:

1. The script used above, supplied with the Ceph source, uses a program called nproc to get the number of processors on the host and this number is then used to parallelize the build. This is unfortunate on a small host like the Odroid-XU4 because it only has 2GB of RAM and even just a single-processor build will swap extensively at times (especially around ceph-dencoder).
   
   So, I wrote my own nproc which always returns ONE, placed it in /mnt/build/tempbin, and that forces the build to be serial.
2. To get extra disk space and necessary swap, I added an external USB drive, set up a 10GB swap partition and a 100GB ext4 partition (which is mounted at /mnt/build) (see above), and this is where I do the build
3. The configuration turns off RDMA because the ibverbs library doesn't seem to be available for ArchLinux (ARM)
4. Also the building of test programs is turned off
5. Even though yasm is present on the system, cmake gives a defective error message regarding an empty string and says there isn't a usable yasm present