Mastering Ceph Object Storage: From Concepts to RADOS Gateway Deployment

What is Object Storage

Object storage manages data as independent objects rather than using traditional hierarchical file systems or block storage. Each object contains the data, metadata (such as creation date, type, etc.), and a unique identifier.

It is mainly used for unstructured data like multimedia, backups, and analytics, and suits applications that require massive, easily accessible, cost‑effective storage. Public cloud services such as Amazon S3, Google Cloud Storage, and OpenStack Swift provide object storage, while Ceph and MinIO are common in private clouds or on‑premises.

Unlike file storage, object storage does not use a directory tree. All data are treated as objects identified by a globally unique ID, and each object includes its data, metadata, and the ID.

The advantages of object storage are scalability and accessibility; it is designed for large‑scale unstructured data, can distribute objects across many servers or regions, offers high redundancy and availability, and is accessed via RESTful APIs, making integration into applications straightforward.

Bucket Concept in Object Storage

A bucket is a logical container used to organize and manage stored objects. Each bucket has a unique name and serves as a flat namespace where objects can be stored, listed, and deleted.

Buckets cannot be nested like folders; they are independent and can have individual configurations such as access permissions and lifecycle rules.

What is RGW

RGW (RADOS Gateway) is a FastCGI service built on the LIBRADOS interface that provides HTTP‑based RESTful APIs compatible with Amazon S3 and OpenStack Swift, allowing a Ceph cluster to function as an object storage system.

Provides S3/Swift compatible APIs.

Supports ACLs and access control mechanisms.

Offers data redundancy and replication.

Handles massive amounts of data.

Supports multi‑tenant environments.

Installing Ceph RADOS Gateway

Online Installation

<code>sudo apt install ceph-radosgw</code>

Offline Installation

First install

apt-rdepends

in a connected environment:

<code>sudo apt-get update
sudo apt-get install apt-rdepends</code>

List all recursive dependencies of the

radosgw

package:

<code>apt-rdepends radosgw | grep -v "^ " > packages.txt</code>

Download the packages:

<code>mkdir packages
cd packages
xargs -a ../packages.txt apt-get download</code>

Copy the

packages

directory to the offline machine and install:

<code>tar -zcvf ceph_radosgw.tar.gz packages
# On the offline machine
tar -zxvf ceph_radosgw.tar.gz
cd packages
dpkg -i *.deb</code>

Configuring Ceph RADOS Gateway

Add a

[client.rgw]

section for each RGW host in

/etc/ceph/ceph.conf

so each instance can be started independently.

<code>[client.rgw.node1]
host = node1
rgw_frontends = "civetweb port=80"
[client.rgw.node2]
host = node2
rgw_frontends = "civetweb port=80"
</code>

Creating a Keyring

Create a keyring for the RGW daemon and set proper ownership:

<code>sudo mkdir -p /var/lib/ceph/radosgw/ceph-rgw.`hostname`
sudo ceph-authtool /var/lib/ceph/radosgw/ceph-rgw.`hostname`/keyring --create-keyring --gen-key -n client.rgw.`hostname`
sudo chown ceph:ceph /var/lib/ceph/radosgw/ceph-rgw.`hostname`/keyring
</code>

Add the keyring to the Ceph cluster:

<code>sudo ceph auth add client.rgw.`hostname` osd 'allow rwx' mon 'allow rwx' -i /var/lib/ceph/radosgw/ceph-rgw.`hostname`/keyring
</code>

Restarting Ceph RADOS Gateway

<code>sudo systemctl start ceph-radosgw@rgw.`hostname`
</code>

Creating RGW Data Pools

Create the pools required by RGW:

<code>ceph osd pool create .rgw.root 64
ceph osd pool create default.rgw.control 64
ceph osd pool create default.rgw.data.root 64
ceph osd pool create default.rgw.gc 64
ceph osd pool create default.rgw.log 64
ceph osd pool create default.rgw.users.uid 64
ceph osd pool create default.rgw.users.email 64
ceph osd pool create default.rgw.users.swift 64
ceph osd pool create default.rgw.buckets.index 64
ceph osd pool create default.rgw.buckets.data 64
</code>

Pool purposes:

.rgw.root – stores RGW configuration and metadata.

default.rgw.control – control data for RGW.

default.rgw.data.root – metadata of newly created buckets.

default.rgw.gc – objects pending garbage collection.

default.rgw.log – access logs.

default.rgw.users.uid / email / swift – user information.

default.rgw.buckets.index – bucket index for fast lookup.

default.rgw.buckets.data – actual object data.

Handling PG Limit Errors

If creating pools exceeds the cluster's PG limit, you may see an error like:

<code>pg_num 64 size 3 would mean 771 total pgs, which exceeds max 750 (mon_max_pg_per_osd 250 * num_in_osds 3)
</code>

Solutions:

Increase

mon_max_pg_per_osd

(e.g., to 300):

<code>ceph config set global mon_max_pg_per_osd 300
ceph config get mon mon_max_pg_per_osd
</code>

Add more OSDs to raise the overall PG capacity.

Reduce

pg_num

values, balancing distribution and performance.

Creating Users with radosgw-admin

<code>radosgw-admin user create --uid "wanger" --display-name "wanger"
</code>

Save the generated

access_key

and

secret_key

for client configuration.

Using AWS CLI with RGW

Install and configure AWS CLI to interact with the RGW S3‑compatible endpoint:

<code>apt-get install awscli
aws configure   # provide access_key, secret_key, region (e.g., us-east-1), output format (json)
</code>

Create a bucket:

<code>aws s3api create-bucket --bucket mybucket --endpoint-url http://node1
</code>

Upload a file:

<code>aws s3 cp myfile.txt s3://mybucket/myfile.txt --endpoint-url http://node1
</code>

List objects:

<code>aws s3 ls s3://mybucket --endpoint-url http://node1
</code>

Delete an object:

<code>aws s3 rm s3://mybucket/myfile.txt --endpoint-url http://node1
</code>