Collocating Ceph volumes and instances in a multi-datacenter setup

loic

OpenStack Havana is installed on machines rented from OVH and Hetzner. An aggregate is created for machines hosted at OVH and another for machines hosted at Hetzner. A Ceph cluster is created with a pool using disks from OVH and another pool using disks from Hetzner. A cinder backend is created for each Ceph pool. From the dashboard, an instance can be created in the OVH availability zone using a Ceph volume provided by the matching OVH pool.

Creating availability zones

The availability zones are created as a side effect of creating an aggregate.

# nova aggregate-create ovh ovh +----+------+-------------------+-------+--------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+------+-------------------+-------+--------------------------------+ | 2 | ovh | ovh | [] | {u'availability_zone': u'ovh'} | +----+------+-------------------+-------+--------------------------------+

nova aggregate-create hetzner hetzner

+----+---------+-------------------+-------+------------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+---------+-------------------+-------+------------------------------------+ | 3 | hetzner | hetzner | [] | {u'availability_zone': u'hetzner'} | +----+---------+-------------------+-------+------------------------------------+

The hosts are assigned to their availability zone:

# nova aggregate-add-host ovh bm0015.the.re Aggregate 2 has been successfully updated. +----+------+-------------------+--------------------+--------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+------+-------------------+--------------------+--------------------------------+ | 2 | ovh | ovh | [u'bm0015.the.re'] | {u'availability_zone': u'ovh'} | +----+------+-------------------+--------------------+--------------------------------+

The result can be checked with

# nova availability-zone-list +-----------------------+----------------------------------------+ | Name | Status | +-----------------------+----------------------------------------+ | internal | available | | |- bm0015.the.re | | | | |- nova-conductor | enabled :-) 2013-11-11T14:26:43.000000 | | | |- nova-consoleauth | enabled :-) 2013-11-11T14:26:43.000000 | | | |- nova-scheduler | enabled :-) 2013-11-11T14:26:43.000000 | | | |- nova-cert | enabled :-) 2013-11-11T14:26:43.000000 | | ovh | available | | |- bm0015.the.re | | | | |- nova-compute | enabled :-) 2013-11-11T14:26:48.000000 | | hetzner | available | | |- bm0016.the.re | | | | |- nova-compute | enabled :-) 2013-11-11T14:26:49.000000 | | nova | available | +-----------------------+----------------------------------------+

Creating the Ceph pools

The crush map is extracted with

ceph osd getcrushmap -o crush.bin crushtool -d crush.bin -o crush.txt

It is edited to add

datacenter ovh { id -5 alg straw hash 0 item bm0014 weight 1.820 item bm0015 weight 1.820 }

rule ovh { ruleset 3 type replicated min_size 1 max_size 10 step take ovh step chooseleaf firstn 0 type host step emit }

and sent back to the Ceph monitors with

crushtool -c crush.txt -o crush.bin ceph osd setcrushmap crush.bin

An ovh pool is created and set to use the ovh ruleset:

ceph osd pool create ovh 128 ceph osd pool set ovh crush_ruleset 3

The crush.txt file also contains the ruleset for the hetzner pool.

Creating cinder backends

In the /etc/cinder/cinder.conf file of the host running cinder-volume, one cinder backend is defined for each Ceph pool:

enabled_backends=rbd-hetzner,rbd-ovh [rbd-hetzner] volume_driver=cinder.volume.driver.RBDDriver rbd_pool=hetzner volume_backend_name=RBD_HETZNER [rbd-ovh] volume_driver=cinder.volume.driver.RBDDriver rbd_pool=ovh volume_backend_name=RBD_OVH

In order to enable the –volume-type ovh option of cinder create, the corresponding type keys must be created:

# cinder type-create ovh +--------------------------------------+------+ | ID | Name | +--------------------------------------+------+ | 48645332-4835-4a9b-9078-cd735f47dae5 | ovh | +--------------------------------------+------+

cinder type-key ovh set volume_backend_name=RBD_OVH

cinder extra-specs-list

+--------------------------------------+---------+------------------------------------------+ | ID | Name | extra_specs | +--------------------------------------+---------+------------------------------------------+ | 48645332-4835-4a9b-9078-cd735f47dae5 | ovh | {u'volume_backend_name': u'RBD_OVH'} | +--------------------------------------+---------+------------------------------------------+

Check that the cinder scheduler is set as follows in /etc/cinder/cinder.conf

scheduler_driver=cinder.scheduler.filter_scheduler.FilterScheduler

Assembling instance and volume

After creating a volume using the OVH cinder backend:

cinder create --volume-type ovh --display-name test 1

An instance is created in the OVH availability zone:

nova boot --availability-zone ovh \ --image 'cirros image' \ --key-name key_loic \ --nic net-id=e1d72366-1f25-42c1-a953-a944c9f932e3 \ --flavor m1.tiny --poll try

The volume is attached to the instance

nova volume-attach try 045d1cae-cd9b-4d64-b0b8-544f5b6d0c5a /dev/vdb