Introduction

On the world of storage, ceph storage cluster is used to centralize all the storage of our K8s / Openshift cluster to one place to manage. Knowing Ceph basic and useful commands can help in managing a Ceph cluster and especially in debugging issues. In this article we will see the Top Useful Ceph Commands We Can’t Survive Without.
For more information about ceph refer to the following post: Intro To Ceph.

ODF and Ceph

In our case, our Ceph cluster runs on an openshift cluster using ODF.
ODF (OpenShift Data Foundation) is an Openshift operator that integrates with Openshift clusters and enable us to establish a Ceph cluster.
For more information make sure you checkout: OpenShift Data Foundation
If you are using ODF, in order to enter the Ceph cli, use the following:

$ oc rsh deploy/rook-ceph-tools

Our Top Useful Commands

There are some basic commands that help us see what is the status on our Ceph storage cluster.
Understanding Ceph and knowing how to interact with it will grant a big benefit for resolving problems.
The following commands help us see the status of the disks, pools, pgs etc…

Ceph’s status

$ ceph -s / ceph status

Display all the disks and their status

$ ceph osd df

Show all the pools and their status

$ ceph df detail

See the connections between disks, symlinks, and osds

$ ceph device ls

There are some more advanced commands that can help us in investigation of problems in a ceph cluster. The first one related to rebalancing.

“In distributed storage systems like Ceph, it is important to balance write and read requests for optimal performance. Write balancing ensures fast storage and replication of data in a cluster, while read balancing ensures quick access and retrieval of data in a cluster. Both types of balancing are important in distributed systems for different reasons.“
Taken from: Ceph Balancing.

Force rebalancing

$ ceph osd reweight-by-utilization

The following commands are used for managing crush rules.
“CRUSH rules define how a Ceph client selects buckets and the primary OSD within them to store objects, and how the primary OSD selects buckets and the secondary OSDs to store replicas or coding chunks. ”
Taken from: Crush Rules IBM.

Archive previous alerts

$ ceph crush archive-all

Show all crush classes

$ ceph osd crush class ls

Show all the disks under ssd crush class

$ ceph osd crush class ls-osd ssd

Create new crush class

$ ceph osd crush class create pool
$ ceph osd crush class create pool2

Remove device class from disks

$ ceph osd crush rm-device-class osd.0 osd.1 osd.2

Set new device class on disks

$ ceph osd crush set-device-class pool2 osd.0 osd.1 osd.2

The following commands help us to deal with pools.
“The Ceph storage cluster stores data objects in logical partitions called pools. Ceph administrators can create pools for particular types of data, such as for Ceph Block Devices, Ceph Object Gateways, or simply just to separate one group of users from another. “
Taken from: Ceph Pools IBM.

Create a pool

$ ceph osd pool create testpool 100 100

Delete a pool

$ ceph osd pool delete pool1 pool1 –yes-i-really-really-mean-it

Delete a pool with force

$ ceph osd pool delete pool1 pool1 –yes-i-really-really-mean-it-not-faking

Note: Deleting a ceph pool is a dangerous act and make sure you do it on test pools you created and not on those that contain data! Be aware of the really great flags that help you notice!

Summary

In conclusion, Ceph is a powerful tool for storage management, and managing the skill of useful commands might save you from debugging long hours.
If you feel like diving even more into the world of Ceph checkout the following post on How to Deploy Highly Available Ceph on Proxmox.