How to deploy a Ceph storage cluster
What is Ceph
Ceph is free open-source clustering software that ties together multiple storage servers, each containing large amounts of hard drives. Essentially, Ceph provides object, block and file storage in a single, horizontally scalable cluster, with no single points of failure. A Ceph storage cluster can be easily scaled over time. It may be configured for high-availability by removing single points of failure. It also has many enterprise features including snapshots, thin provisioning, tiering and self-healing capabilities.
How Ceph Works
A Ceph Storage Cluster consists of multiple types of daemons:
- Ceph Monitor
- Ceph OSD Daemon
- Ceph Manager
- Ceph Metadata Server
Ceph Monitor (ceph-mon)
A Ceph Monitor maintains a master copy of the cluster map. A cluster of Ceph monitors ensures high availability should a monitor daemon fail. Storage cluster clients retrieve a copy of the cluster map from the Ceph Monitor.
Ceph OSD (ceph-osd)
A Ceph OSD Daemon checks its own state and the state of other OSDs and reports back to monitors.
Ceph Manager (ceph-mgr)
A Ceph Manager acts as an endpoint for monitoring, orchestration, and plug-in modules.
Ceph Metadata Server (ceph-mds)
A Ceph Metadata Server (MDS) manages file metadata when CephFS is used to provide file services.
Storage cluster clients and each Ceph OSD Daemon use the CRUSH algorithm to efficiently compute information about data location, instead of having to depend on a central lookup table.
When one or more monitors and two or more object storage are deployed, it is known as a Ceph Storage Cluster. The file system, object storage, and block devices read and write data to and from the storage cluster.
Ceph Pacific Cluster Creation
To start and test how it all works, we basically need 3 nodes for the LAB. In our case, we will use virtual servers in a VMware environment.
Each server will have a minimum of virtual resources:
- 2 vCPU
- 4GB RAM
- 2x vDisk (30GB OS disk and 20GB disk as OSD for Ceph)
- Ubuntu 20.04
|ceph-node-01||10.99.107.81||Ceph MON, MGR, OSD, MDS|
|ceph-node-02||10.99.107.82||Ceph MON, MGR, OSD, MDS|
|ceph-node-03||10.99.107.83||Ceph MON, MGR, OSD, MDS|
Cephadm creates a new Ceph cluster by “bootstrapping” on a single host, expanding the cluster to encompass any additional hosts, and then deploying the needed services.
- Python 3
- Podman or Docker for running containers
- Time synchronization (such as chrony or NTP)
- LVM2 for provisioning storage devices
Any modern Linux distribution should be sufficient. Dependencies are installed automatically by the bootstrap process.
Since we live in modern times, we have the ansible tool at our disposal, which simplifies the whole process and can provide us with everything we need.
Log in to ceph-node-01, which will be our ADMIN node using SSH:
We update the
/etc/hosts file with entries for all IP addresses and hostnames.
Install Ansible and other basic utilities:
/usr/local/bin path is added to PATH.
Generate SSH keys:
Upload the SSH key to all nodes using a simple BASH script:
cephadm is available for local use:
With the first node
ceph-node-01 configured, we will create the corresponding Ansible playbook for updating all nodes and upload the SSH public key and the update file
/etc/hosts to all nodes.
We’ll edit the content below to have the correct time zone and add it to the file:
Create inventory file.
Execute Ansible playbook which will install on all nodes
Chrony (TZ Europe / Bratislava) and
/etc/hosts on all nodes using Ansible:
Create a folder for the CEPH configuration:
Prepare the YAML configuration
cluster.yaml for the CEPH bootstrap:
The first step in creating a new Ceph cluster is running the cephadm bootstrap command on the Ceph cluster’s first host.
This command will:
- Create a monitor and manager daemon for the new cluster on the all hosts.
- Generate a new SSH key for the Ceph cluster and add it to the root user’s
- Write a minimal configuration file to
/etc/ceph/ceph.conf. This file is needed to communicate with the new cluster.
- Write a copy of the client.admin administrative (privileged!) secret key to
- Write a copy of the public key to
If everything is done and Bootstrap has been successful we can install the tools needed for CEPH:
You can install the ceph-common package, which contains all of the ceph commands, including ceph, rbd, mount.ceph (for mounting CephFS file systems), etc.:
Confirm that the ceph command is accessible with:
Confirm that the ceph command can connect to the cluster and also its status with:
And that’s how we can deploy a fully functional Ceph cluster. There is important flexibility when running Ceph with containers, mainly in terms of upgrades, failures, etc. Cephadm installs and manages a Ceph cluster using containers and systemd, with tight integration with the CLI and dashboard GUI. Cephadm is fully integrated with the new orchestration API and fully supports the new CLI and dashboard features to manage cluster deployment.