Planet Ceph

Aggregated news from external sources

  • December 17, 2014
    Ceph: collect Kernel RBD logs

    {% img center http://sebastien-han.fr/images/ceph-collect-krbd-logs.jpg Ceph: collect Kernel RBD logs %}

    Quick tip to collect Kernel RBD logs.

    Make sure your kernel is compiled with CONFIG_DYNAMIC_DEBUG (and CONFIG_DEBUG_FS) enabled:

    bash
    $ sudo cat /boot/config-`uname -r` | grep DYNAMIC_DEBUG
    CONFIG_DYNAMIC_DEBUG=y

    Then mount debugfs:

    bash
    $ sudo mount -t debugfs none /sys/kernel/debug

    Set the console log level to 9:

    bash
    $ sudo echo 9 > /proc/sysrq-trigger

    Then chose the module that you want to log:

    bash
    $ sudo echo 'module rbd +p' | sudo tee -a /sys/kernel/debug/dynamic_debug/control

    Looking at dmesg will show the corresponding logs.
    You can use this script from the Ceph repo as well to enable all of them:

    “`bash

    !/bin/sh -x

    p() {
    echo “$*” > /sys/kernel/debug/dynamic_debug/control
    }

    echo 9 > /proc/sysrq-trigger
    p ‘module ceph +p’
    p ‘module libceph +p’
    p ‘module rbd +p’
    “`

  • December 15, 2014
    Ceph: Cache Tier Statistics

    {% img center http://sebastien-han.fr/images/ceph-cache-statistics.jpg Ceph: Cache Tier Statistics %}

    Quick tip on how to retrieve cache statistics from the a cache pool.

    Simply use the admin socket:

    “`bash
    $ sudo ceph daemon osd.{id} perf …

  • December 15, 2014
    Teuthology docker targets hack (3/4)

    The teuthology container hack is improved so each Ceph command is run via docker exec -i which can read from stdin as of docker 1.4 released in December 2014. It can run the following job machine_type: container os_type: ubuntu os_version: … Continue reading

  • December 13, 2014
    Why are by-partuuid symlinks missing or outdated ?

    The ceph-disk script manages Ceph devices and rely on the content of the /dev/disk/by-partuuid directory which is updated by udev rules. For instance: a new partition is created with /sbin/sgdisk –largest-new=1 –change-name=1:ceph data –partition-guid=1:83c14a9b-0493-4ccf-83ff-e3e07adae202 –typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be — /dev/loop4 the kernel is … Continue reading

  • December 11, 2014
    DevStack and remote Ceph cluster

    {% img center http://sebastien-han.fr/images/devstack-ceph-remote-cluster.jpg DevStack and remote Ceph cluster %}

    Introducing the ability to connect DevStack to a remote Ceph cluster.
    So DevStack won’t bootstrap any Ceph cluster, it will simply connec…

  • December 9, 2014
    How many PGs in each OSD of a Ceph cluster ?

    To display how many PGs in each OSD of a Ceph cluster: $ ceph –format xml pg dump | \ xmlstarlet sel -t -m “//pg_stats/pg_stat/acting” -v osd -n | \ sort -n | uniq -c 332 0 312 1 299 … Continue reading

  • December 9, 2014
    OpenStack: import existing Ceph volumes in Cinder

    {% img center http://sebastien-han.fr/images/openstack-import-existing-vol-ceph-cinder.jpg OpenStack: import existing Ceph volumes in Cinder %}

    This method can be useful while migrating from one OpenStack to another.

    Imagine you have operatin…

  • November 29, 2014
    Ceph read-only mirror on gitlab

    The gitlab-mirrors scripts are installed to setup a a read-only Ceph mirror, updated hourly. It is used for permalinks such as src/osd/ClassHandler.cc#L170. The gitlab-mirrors config.sh is as follows: #Environment file # # gitlab-mirrors settings # #The user git-mirrors will run … Continue reading

  • November 28, 2014
    Placement_pools on Rados-GW

    The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive…

    1
    2
    
       standard_bucket datas  --> .rgw.buckets        (default pool)
       specific_bucket datas  --> .rgw.buckets.custom

    First, we create a pool .rgw.buckets.custom, with, for example, some specific parameters (different size and different ruleset in crushmap) :

    1
    2
    3
    4
    5
    6
    7
    8
    
    $ ceph osd pool create .rgw.buckets.custom 64 64
    pool '.rgw.buckets.custom' created
    
    $ ceph osd pool set .rgw.buckets.custom size 2
    set pool 59 size to 2
    
    $ ceph osd pool set .rgw.buckets.custom crush_ruleset 6
    set pool 59 crush_ruleset to 6
    

    Then, we need to configure a specific placement_targets in region map and zone.
    For next step, you need to have a running config of rados-gw…

    1
    2
    
    $ radosgw-admin region get > region.conf.json
    $ vim region.conf.json            # Add an entry in placement_targets
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
    { "name": "default",
      "api_name": "",
      "is_master": "true",
      "endpoints": [],
      "master_zone": "",
      "zones": [
            { "name": "default",
              "endpoints": [],
              "log_meta": "false",
              "log_data": "false"}],
      "placement_targets": [
            { "name": "default-placement",
              "tags": []},
            { "name": "custom-placement",
              "tags": []}],
      "default_placement": "default-placement"}
    
    1
    2
    
    $ radosgw-admin region set < region.conf.json
    ....
    
    1
    2
    
    $ radosgw-admin zone get > zone.conf.json
    $ vim zone.conf.json            # Add an entry in placement_pools with key "custom-placement"
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    { "domain_root": ".rgw",
      "control_pool": ".rgw.control",
      "gc_pool": ".rgw.gc",
      "log_pool": ".log",
      "intent_log_pool": ".intent-log",
      "usage_log_pool": ".usage",
      "user_keys_pool": ".users",
      "user_email_pool": ".users.email",
      "user_swift_pool": ".users.swift",
      "user_uid_pool": ".users.uid",
      "system_key": { "access_key": "",
          "secret_key": ""},
      "placement_pools": [
            { "key": "default-placement",
              "val": { "index_pool": ".rgw.buckets.index",
                  "data_pool": ".rgw.buckets",
                  "data_extra_pool": ".rgw.buckets.extra"}},
            { "key": "custom-placement",
              "val": { "index_pool": ".rgw.buckets.index",
                  "data_pool": ".rgw.buckets.custom",
                  "data_extra_pool": ".rgw.buckets.extra"}}]}
    
    1
    2
    3
    
    $ radosgw-admin zone set <zone.conf.json
    2014-11-25 18:03:23.894153 7f728c0f2780  0 couldn't find old data placement pools config, setting up new ones for the zone
    .....
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    
    $ radosgw-admin regionmap update
    { "regions": [
            { "key": "default",
              "val": { "name": "default",
                  "api_name": "",
                  "is_master": "true",
                  "endpoints": [],
                  "master_zone": "",
                  "zones": [
                        { "name": "default",
                          "endpoints": [],
                          "log_meta": "false",
                          "log_data": "false"}],
                  "placement_targets": [
                        { "name": "custom-placement",
                          "tags": []},
                        { "name": "default-placement",
                          "tags": []}],
                  "default_placement": "default-placement"}}],
      "master_region": "default",
      "bucket_quota": { "enabled": false,
          "max_size_kb": -1,
          "max_objects": -1},
      "user_quota": { "enabled": false,
          "max_size_kb": -1,
          "max_objects": -1}}
    
    $ /etc/init.d/radosgw reload
    Reloading ...
    

    To configure s3cmd for RadosGW you can have a look here :
    http://lollyrock.com/articles/s3cmd-with-radosgw/

    Now we can test bucket creation :

    1
    2
    3
    4
    5
    6
    7
    8
    9
    
    $ s3cmd mb s3://custombucket --bucket-location=custom-placement
    Bucket 'custombucket' created
    
    $ touch "file_on_custom_pool"
    
    $ s3cmd put file_on_custom_pool s3://custombucket
    WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
    file_on_custom_pool -> s3://custombucket/file_on_custom_pool  [1 of 1]
     0 of 0     0% in    0s     0.00 B/s  done
    

    Verify :

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    
    $ radosgw-admin bucket stats --bucket=custombucket
    { "bucket": "custombucket",
      "pool": ".rgw.buckets.custom",
      "index_pool": ".rgw.buckets.index",
      "id": "default.240909.1",
      "marker": "default.240909.1",
      "owner": "testuser",
      "ver": 1,
      "master_ver": 0,
      "mtime": 1417016078,
      "max_marker": "",
      "usage": {},
      "bucket_quota": { "enabled": false,
          "max_size_kb": -1,
          "max_objects": -1}}
    

    Pool var is set on “.rgw.buckets.custom”.

    1
    2
    
    $ rados -p .rgw.buckets.custom ls
    default.241071.1_file_on_custom_pool
    

    It’s here !

    Data placement pool is define in this order :

    1. from the request (“bucket location”)
    2. from user (“default_placement” : see with radosgw-admin metadata get user:<uid>)
    3. from region map (“default_placement”)
  • November 27, 2014
    Ceph: recover OSDs after SSD journal failure

    A common recommendation is to store OSD journal on a SSD drive which implies loosing your OSD if this journal fails.
    This article assumes that your OSDs have been originally deployed with ceph-disk.
    You will also realise that it’s really simple to br…

  • November 25, 2014
    HOWTO debug a teuthology task

    To debug a modification to a ceph-qa-suite task ( for instance repair_test.py), a teuthology target is locked with: $ ./virtualenv/bin/teuthology-lock –lock-many 1 –owner loic@dachary.org $ ./virtualenv/bin/teuthology-lock –list-targets –owner loic@dachary.org > targets.yaml and used to run the test with: ./virtualenv/bin/teuthology \ … Continue reading

  • November 17, 2014
    Ceph RBD With LXC Containers

    A simple way to secure your data with containers is to use a distributed storage such as Ceph for LXC root storage.

    For exemple :

    1
    
    # lxc-create -n mycontainer -t debian -B rbd --pool rbd --rbd mycontainer --fstype ext4 --fssize 500
    

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    
    mke2fs 1.42.5 (29-Jul-2012)
    Filesystem label=
    OS type: Linux
    Block size=1024 (log=0)
    Fragment size=1024 (log=0)
    Stride=4096 blocks, Stripe width=4096 blocks
    128016 inodes, 512000 blocks
    25600 blocks (5.00%) reserved for the super user
    First data block=1
    Maximum filesystem blocks=67633152
    63 block groups
    8192 blocks per group, 8192 fragments per group
    2032 inodes per group
    Superblock backups stored on blocks:
      8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
    
    Allocating group tables: done                            
    Writing inode tables: done                            
    Creating journal (8192 blocks): done
    Writing superblocks and filesystem accounting information: done 
    
    Note: Usually the template option is called with a configuration
    file option too, mostly to configure the network.
    For more information look at lxc.conf (5)
    
    debootstrap is /usr/sbin/debootstrap
    Checking cache download in /var/cache/lxc/debian/rootfs-wheezy-amd64 ...
    Copying rootfs to /var/lib/lxc/mycontainer/rootfs...Generating locales (this might take a while)...
      en_US.UTF-8... done
    Generation complete.
    update-rc.d: using dependency based boot sequencing
    update-rc.d: using dependency based boot sequencing
    update-rc.d: using dependency based boot sequencing
    update-rc.d: using dependency based boot sequencing
    
    Current default time zone: 'America/New_York'
    Local time is now:      Tue Nov 18 09:34:16 EST 2014.
    Universal Time is now:  Tue Nov 18 14:34:16 UTC 2014.
    
    Root password is 'root', please change !
    'debian' template installed
    'mycontainer' created
    
    1
    2
    
    # mount | grep mycontainer
    /dev/rbd1 on /var/lib/lxc/mycontainer/rootfs type ext4 (rw,relatime,stripe=4096,data=ordered)
    

    Diff file for lxc-create :

    1
    
    # diff -u /usr/bin/lxc-create.orig /usr/bin/lxc-create
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    
    --- /usr/bin/lxc-create.orig    2014-11-17 04:16:41.181942000 -0500
    +++ /usr/bin/lxc-create  2014-11-17 04:35:27.225942000 -0500
    @@ -24,6 +24,7 @@
         echo "usage: lxc-create -n <name> [-f configuration] [-t template] [-h] [fsopts] -- [template_options]"
         echo "   fsopts: -B none"
         echo "   fsopts: -B lvm [--lvname lvname] [--vgname vgname] [--fstype fstype] [--fssize fssize]"
    +    echo "   fsopts: -B rbd [--pool poolname] [--rbd rbd] [--fstype fstype] [--fssize fssize]"
         echo "   fsopts: -B btrfs"
         echo "           flag is not necessary, if possible btrfs support will be used"
     #    echo "   fsopts: -B union [--uniontype overlayfs]"
    @@ -64,7 +65,7 @@
     }
    
     shortoptions='hn:f:t:B:'
    -longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,fssize:'
    +longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,pool:,rbd:,fssize:'
     localstatedir=/var
     lxc_path=${localstatedir}/lib/lxc
     bindir=/usr/bin
    @@ -119,6 +120,16 @@
          vgname=$1
          shift
          ;;
    +        --pool)
    +        shift
    +        pool=$1
    +        shift
    +        ;;
    +        --rbd)
    +        shift
    +        rbd=$1
    +        shift
    +        ;;
          --fstype)
          shift
          fstype=$1
    @@ -161,7 +172,7 @@
     fi
    
     case "$backingstore" in
    -    lvm|none|btrfs|_unset) :;;
    +    lvm|rbd|none|btrfs|_unset) :;;
         *) echo "'$backingstore' is not known ('none', 'lvm', 'btrfs')"
             usage
             exit 1
    @@ -216,6 +227,13 @@
             echo "please delete it (using \"lvremove $rootdev\") and try again"
             exit 1
         fi
    +elif [ "$backingstore" = "rbd" ]; then
    +    which rbd > /dev/null
    +    if [ $? -ne 0 ]; then
    +        echo "rbd command not found. Please install ceph-common package"
    +        exit 1
    +    fi
    +    rootdev=/dev/rbd/$pool/$rbd
     elif [ "$backingstore" = "btrfs" ]; then
         mkdir "$lxc_path/$lxc_name"
         if ! out=$(btrfs subvolume create "$rootfs" 2>&1); then
    @@ -257,6 +275,14 @@
         mkfs -t $fstype $rootdev || exit 1
         mount -t $fstype $rootdev $rootfs
     fi
    +
    +if [ $backingstore = "rbd" ]; then
    +    [ -d "$rootfs" ] || mkdir $rootfs
    +    rbd create $pool/$rbd --size=$fssize || exit 1
    +    rbd map $pool/$rbd || exit 1
    +    mkfs -t $fstype $rootdev || exit 1
    +    mount -t $fstype $rootdev $rootfs
    +fi
    
     if [ ! -z $lxc_template ]; then
    

    If you want to make persistent after reboot, you must add rbd in /etc/ceph/rbdmap and add line in fstab.

Careers