Archives:

RGW Metadata Search

I have recently been working on adding metadata search to rgw. It’s not in yet, nor is it completely ready. I do think that it’s at a point where it would be great to get some feedback. This feature is built on top of another feature that I talked about a few months ago on CDM, which is the “sync modules” (formerly known as “sync plugins”) feature. The current code can be found in the following PR:

https://github.com/ceph/ceph/pull/10731

read more…

v10.0.0 released

This is the first development release for the Jewel cycle.  We are off to a good start, with lots of performance improvements flowing into the tree.  We are targeting sometime in Q1 2016 for the final Jewel.

NOTABLE CHANGES

read more…

v0.80.11 Firefly released

This is a bugfix release for Firefly.  As the Firefly 0.80.x series is nearing its planned end of life in January 2016 it may also be the last.

We recommend that all Firefly users upgrade.

For more detailed information, see the complete changelog.

NOTABLE CHANGES

read more…

monash-clayton

In case you didn’t know, Ceph Days are a series of regular events in support of the Ceph community that happen at locations around the world from New York to Berlin, and more recently in APAC to Shanghai and Melbourne.

The Melbourne edition on November 5th, 2015 was Australia’s first ever Ceph Day. It was hosted at the Clayton campus of sponsor Monash University, about a $60 Uber ride from the city centre. Monash is also home to a part of Australia’s national research cloud: NeCTAR, including Ceph-backed storage capacity provided as part of the Research Data Services initiative.

Watch this video to get a feel for the day in about 2 minutes.

read more…

v9.2.0 Infernalis released

InfernalisThis major release will be the foundation for the next stable series. There have been some major changes since v0.94.x Hammer, and the upgrade process is non-trivial. Please read these release notes carefully.

MAJOR CHANGES FROM HAMMER

read more…

v0.94.5 Hammer released

This Hammer point release fixes a critical regression in librbd that can cause Qemu/KVM to crash when caching is enabled on images that have been cloned.

All v0.94.4 Hammer users are strongly encouraged to upgrade.

NOTABLE CHANGES

  • librbd: potential assertion failure during cache read (issue#13559, pr#6348, Jason Dillaman)
  • osd: osd/ReplicatedPG: remove stray debug line (issue#13455, pr#6362, Sage Weil)
  • tests: qemu workunit refers to apt-mirror.front.sepia.ceph.com (issue#13420, pr#6330, Yuan Zhou)

For more detailed information, see the complete changelog.

GETTING CEPH

Ceph read-only mirror on gitlab

The gitlab-mirrors scripts are installed to setup a a read-only Ceph mirror, updated hourly. It is used for permalinks such as src/osd/ClassHandler.cc#L170.

The gitlab-mirrors config.sh is as follows:

#Environment file

#
# gitlab-mirrors settings
#

#The user git-mirrors will run as.
system_user="gitmirror"
#The home directory path of the $system_user
user_home="/home/${system_user}"
#The repository directory where gitlab-mirrors will contain copies of mirrored
#repositories before pushing them to gitlab.
repo_dir="${user_home}/repositories"
#colorize output of add_mirror.sh, update_mirror.sh, and git-mirrors.sh
#commands.
enable_colors=true
#These are additional options which should be passed to git-svn.  On the command
#line type "git help svn"
git_svn_additional_options="-s"
#Force gitlab-mirrors to not create the gitlab remote so a remote URL must be
#provided. (superceded by no_remote_set)
no_create_set=false
#Force gitlab-mirrors to only allow local remotes only.
no_remote_set=false
#Enable force fetching and pushing.  Will overwrite references if upstream
#forced pushed.  Applies to git projects only.
force_update=false
#This option is for pruning mirrors.  If a branch is deleted upstream then that
#change will propagate into your GitLab mirror.  Aplies to git projects only.
prune_mirrors=false

#
# Gitlab settings
#

#This is the Gitlab group where all project mirrors will be grouped.
gitlab_namespace="Ceph"
#This is the base web url of your Gitlab server.
gitlab_url="http://workbench.dachary.org"
#Special user you created in Gitlab whose only purpose is to update mirror sites
#and admin the $gitlab_namespace group.
gitlab_user="gitmirror"
#Generate a token for your $gitlab_user and set it here.
gitlab_user_token_secret="$(head -n1 "${user_home}/private_token" 2> /dev/null || echo "")"
#Verify signed SSL certificates?
ssl_verify=false
#Push to GitLab over http?  Otherwise will push projects via SSH.
http_remote=false

#
# Gitlab new project default settings.  If a project needs to be created by
# gitlab-mirrors then it will assign the following values as defaults.
#

#values must be true or false
issues_enabled=false
wall_enabled=false
wiki_enabled=false
snippets_enabled=false
merge_requests_enabled=true
public=true

It is configured for mirrors to be accessible to unauthenticated users by default public=true and disables SSL verification because the gitlab does not have https anyway : ssl_verify=false.
Although the three easy steps contain many sub steps, they can be followed to completion without problems.
For fine control over what is mirrored (release branches, tags and pull requests from github), the following can be used instead:

#!/bin/bash
git fetch --force origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* +refs/pull/*:refs/pull/*
git for-each-ref 'refs/pull/*/head' | cut -f2 | xargs --no-run-if-empty --max-args 1 git update-ref -d
git push --prune --force gitlab $(for branch in dumpling emperor firefly giant next master ; do echo +refs/heads/$branch:refs/heads/$branch ; done) +refs/pull/*:refs/heads/pull/*
git prune

The merge ref exists if the head of the pull request refs/pull/XXX/head can successfully be merged. They are removed to keep only one ref per pull request. All pull request refs are mapped under the refs/head so that they are noticed by GitLab CI. If they were kept under refs/pull, they would not be.
It is run hourly with:

$ cat ~/crontab
@hourly ( date ; /home/gitmirror/mirror.sh ) > /home/gitmirror/cron.log 2>&1

Placement_pools on Rados-GW

The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive…

1
2
   standard_bucket datas  --> .rgw.buckets        (default pool)
   specific_bucket datas  --> .rgw.buckets.custom

First, we create a pool .rgw.buckets.custom, with, for example, some specific parameters (different size and different ruleset in crushmap) :

1
2
3
4
5
6
7
8
$ ceph osd pool create .rgw.buckets.custom 64 64
pool '.rgw.buckets.custom' created

$ ceph osd pool set .rgw.buckets.custom size 2
set pool 59 size to 2

$ ceph osd pool set .rgw.buckets.custom crush_ruleset 6
set pool 59 crush_ruleset to 6

Then, we need to configure a specific placement_targets in region map and zone.
For next step, you need to have a running config of rados-gw…

1
2
$ radosgw-admin region get > region.conf.json
$ vim region.conf.json            # Add an entry in placement_targets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{ "name": "default",
  "api_name": "",
  "is_master": "true",
  "endpoints": [],
  "master_zone": "",
  "zones": [
        { "name": "default",
          "endpoints": [],
          "log_meta": "false",
          "log_data": "false"}],
  "placement_targets": [
        { "name": "default-placement",
          "tags": []},
        { "name": "custom-placement",
          "tags": []}],
  "default_placement": "default-placement"}
1
2
$ radosgw-admin region set < region.conf.json
....
1
2
$ radosgw-admin zone get > zone.conf.json
$ vim zone.conf.json            # Add an entry in placement_pools with key "custom-placement"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{ "domain_root": ".rgw",
  "control_pool": ".rgw.control",
  "gc_pool": ".rgw.gc",
  "log_pool": ".log",
  "intent_log_pool": ".intent-log",
  "usage_log_pool": ".usage",
  "user_keys_pool": ".users",
  "user_email_pool": ".users.email",
  "user_swift_pool": ".users.swift",
  "user_uid_pool": ".users.uid",
  "system_key": { "access_key": "",
      "secret_key": ""},
  "placement_pools": [
        { "key": "default-placement",
          "val": { "index_pool": ".rgw.buckets.index",
              "data_pool": ".rgw.buckets",
              "data_extra_pool": ".rgw.buckets.extra"}},
        { "key": "custom-placement",
          "val": { "index_pool": ".rgw.buckets.index",
              "data_pool": ".rgw.buckets.custom",
              "data_extra_pool": ".rgw.buckets.extra"}}]}
1
2
3
$ radosgw-admin zone set <zone.conf.json
2014-11-25 18:03:23.894153 7f728c0f2780  0 couldn't find old data placement pools config, setting up new ones for the zone
.....
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ radosgw-admin regionmap update
{ "regions": [
        { "key": "default",
          "val": { "name": "default",
              "api_name": "",
              "is_master": "true",
              "endpoints": [],
              "master_zone": "",
              "zones": [
                    { "name": "default",
                      "endpoints": [],
                      "log_meta": "false",
                      "log_data": "false"}],
              "placement_targets": [
                    { "name": "custom-placement",
                      "tags": []},
                    { "name": "default-placement",
                      "tags": []}],
              "default_placement": "default-placement"}}],
  "master_region": "default",
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}

$ /etc/init.d/radosgw reload
Reloading ...

To configure s3cmd for RadosGW you can have a look here :
http://lollyrock.com/articles/s3cmd-with-radosgw/

Now we can test bucket creation :

1
2
3
4
5
6
7
8
9
$ s3cmd mb s3://custombucket --bucket-location=custom-placement
Bucket 'custombucket' created

$ touch "file_on_custom_pool"

$ s3cmd put file_on_custom_pool s3://custombucket
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
file_on_custom_pool -> s3://custombucket/file_on_custom_pool  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

Verify :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ radosgw-admin bucket stats --bucket=custombucket
{ "bucket": "custombucket",
  "pool": ".rgw.buckets.custom",
  "index_pool": ".rgw.buckets.index",
  "id": "default.240909.1",
  "marker": "default.240909.1",
  "owner": "testuser",
  "ver": 1,
  "master_ver": 0,
  "mtime": 1417016078,
  "max_marker": "",
  "usage": {},
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}

Pool var is set on “.rgw.buckets.custom”.

1
2
$ rados -p .rgw.buckets.custom ls
default.241071.1_file_on_custom_pool

It’s here !

Data placement pool is define in this order :

  1. from the request (“bucket location”)
  2. from user (“default_placement” : see with radosgw-admin metadata get user:<uid>)
  3. from region map (“default_placement”)

A common recommendation is to store OSD journal on a SSD drive which implies loosing your OSD if this journal fails.
This article assumes that your OSDs have been originally deployed with ceph-disk.
You will also realise that it’s really simple to bring your OSDs back to life after replacing your faulty SSD with a new one.

read more…

HOWTO debug a teuthology task

To debug a modification to a ceph-qa-suite task ( for instance repair_test.py), a teuthology target is locked with:

$ ./virtualenv/bin/teuthology-lock --lock-many 1 --owner loic@dachary.org
$ ./virtualenv/bin/teuthology-lock --list-targets --owner loic@dachary.org > targets.yaml

and used to run the test with:

./virtualenv/bin/teuthology \
  --suite-path $HOME/software/ceph/ceph-qa-suite \
  --owner loic@dachary.org \
  $HOME/software/ceph/ceph-qa-suite/suites/rados/basic/tasks/repair_test.yaml \
  roles.yaml

where roles.yaml sets all roles to one target:

roles:
- [mon.0, osd.0, osd.1, osd.2, osd.3, osd.4, client.0]

Each run requires the installation and deinstallation of all Ceph packages and takes minutes. The installation part of repair_test.yaml can be commented out and the packages installed manually.

$ cat repair.yaml
...
tasks:
#- install:
- ceph:
- repair_test:


The existing packages are removed:

sudo apt-get remove --purge python-ceph ceph-common librados2 librbd1 ceph-fuse libcephfs1-dbg libcephfs-java libcephfs1 libcephfs-jni ceph-fs-common

The repository is set to the desired branch (for instance wip-10018-primary-erasure-code-hinfo), as compiled by gitbuilder.cgi with:

$ echo deb http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ref/wip-10018-primary-erasure-code-hinfo trusty main | sudo tee /etc/apt/sources.list.d/ceph.list

The packages are installed:

$ sudo apt-get update
$ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph ceph-dbg ceph-mds ceph-mds-dbg ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-test ceph-test-dbg radosgw radosgw-dbg python-ceph libcephfs1 libcephfs1-dbg libcephfs-java librados2 librados2-dbg librbd1 librbd1-dbg

and the repositories cleaned (this must be done again after each test completes):

$ rm -fr cephtest/ && sudo rm -fr /var/lib/ceph && sudo rm -fr /var/log/ceph/*

Utilities must be copied from the teuthology sources with (assuming vpm178 is the target):

$ scp teuthology/task/daemon-helper teuthology/task/adjust-ulimits ubuntu@vpm178:/tmp
$ ssh ubuntu@vpm178 sudo cp -a /tmp/{daemon-helper,adjust-ulimits} /usr/bin
© 2016, Red Hat, Inc. All rights reserved.