Planet Ceph

Aggregated news from external sources

  • November 29, 2014
    Ceph read-only mirror on gitlab

    The gitlab-mirrors scripts are installed to setup a a read-only Ceph mirror, updated hourly. It is used for permalinks such as src/osd/ClassHandler.cc#L170. The gitlab-mirrors config.sh is as follows: #Environment file # # gitlab-mirrors settings # #The user git-mirrors will run … Continue reading

  • November 28, 2014
    Placement_pools on Rados-GW

    The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive…

    1
    2
    
       standard_bucket datas  --> .rgw.buckets        (default pool)
       specific_bucket datas  --> .rgw.buckets.custom

    First, we create a pool .rgw.buckets.custom, with, for example, some specific parameters (different size and different ruleset in crushmap) :

    1
    2
    3
    4
    5
    6
    7
    8
    
    $ ceph osd pool create .rgw.buckets.custom 64 64
    pool '.rgw.buckets.custom' created
    
    $ ceph osd pool set .rgw.buckets.custom size 2
    set pool 59 size to 2
    
    $ ceph osd pool set .rgw.buckets.custom crush_ruleset 6
    set pool 59 crush_ruleset to 6
    

    Then, we need to configure a specific placement_targets in region map and zone.
    For next step, you need to have a running config of rados-gw…

    1
    2
    
    $ radosgw-admin region get > region.conf.json
    $ vim region.conf.json            # Add an entry in placement_targets
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
    { "name": "default",
      "api_name": "",
      "is_master": "true",
      "endpoints": [],
      "master_zone": "",
      "zones": [
            { "name": "default",
              "endpoints": [],
              "log_meta": "false",
              "log_data": "false"}],
      "placement_targets": [
            { "name": "default-placement",
              "tags": []},
            { "name": "custom-placement",
              "tags": []}],
      "default_placement": "default-placement"}
    
    1
    2
    
    $ radosgw-admin region set < region.conf.json
    ....
    
    1
    2
    
    $ radosgw-admin zone get > zone.conf.json
    $ vim zone.conf.json            # Add an entry in placement_pools with key "custom-placement"
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    { "domain_root": ".rgw",
      "control_pool": ".rgw.control",
      "gc_pool": ".rgw.gc",
      "log_pool": ".log",
      "intent_log_pool": ".intent-log",
      "usage_log_pool": ".usage",
      "user_keys_pool": ".users",
      "user_email_pool": ".users.email",
      "user_swift_pool": ".users.swift",
      "user_uid_pool": ".users.uid",
      "system_key": { "access_key": "",
          "secret_key": ""},
      "placement_pools": [
            { "key": "default-placement",
              "val": { "index_pool": ".rgw.buckets.index",
                  "data_pool": ".rgw.buckets",
                  "data_extra_pool": ".rgw.buckets.extra"}},
            { "key": "custom-placement",
              "val": { "index_pool": ".rgw.buckets.index",
                  "data_pool": ".rgw.buckets.custom",
                  "data_extra_pool": ".rgw.buckets.extra"}}]}
    
    1
    2
    3
    
    $ radosgw-admin zone set <zone.conf.json
    2014-11-25 18:03:23.894153 7f728c0f2780  0 couldn't find old data placement pools config, setting up new ones for the zone
    .....
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    
    $ radosgw-admin regionmap update
    { "regions": [
            { "key": "default",
              "val": { "name": "default",
                  "api_name": "",
                  "is_master": "true",
                  "endpoints": [],
                  "master_zone": "",
                  "zones": [
                        { "name": "default",
                          "endpoints": [],
                          "log_meta": "false",
                          "log_data": "false"}],
                  "placement_targets": [
                        { "name": "custom-placement",
                          "tags": []},
                        { "name": "default-placement",
                          "tags": []}],
                  "default_placement": "default-placement"}}],
      "master_region": "default",
      "bucket_quota": { "enabled": false,
          "max_size_kb": -1,
          "max_objects": -1},
      "user_quota": { "enabled": false,
          "max_size_kb": -1,
          "max_objects": -1}}
    
    $ /etc/init.d/radosgw reload
    Reloading ...
    

    To configure s3cmd for RadosGW you can have a look here :
    http://lollyrock.com/articles/s3cmd-with-radosgw/

    Now we can test bucket creation :

    1
    2
    3
    4
    5
    6
    7
    8
    9
    
    $ s3cmd mb s3://custombucket --bucket-location=custom-placement
    Bucket 'custombucket' created
    
    $ touch "file_on_custom_pool"
    
    $ s3cmd put file_on_custom_pool s3://custombucket
    WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
    file_on_custom_pool -> s3://custombucket/file_on_custom_pool  [1 of 1]
     0 of 0     0% in    0s     0.00 B/s  done
    

    Verify :

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    
    $ radosgw-admin bucket stats --bucket=custombucket
    { "bucket": "custombucket",
      "pool": ".rgw.buckets.custom",
      "index_pool": ".rgw.buckets.index",
      "id": "default.240909.1",
      "marker": "default.240909.1",
      "owner": "testuser",
      "ver": 1,
      "master_ver": 0,
      "mtime": 1417016078,
      "max_marker": "",
      "usage": {},
      "bucket_quota": { "enabled": false,
          "max_size_kb": -1,
          "max_objects": -1}}
    

    Pool var is set on “.rgw.buckets.custom”.

    1
    2
    
    $ rados -p .rgw.buckets.custom ls
    default.241071.1_file_on_custom_pool
    

    It’s here !

    Data placement pool is define in this order :

    1. from the request (“bucket location”)
    2. from user (“default_placement” : see with radosgw-admin metadata get user:<uid>)
    3. from region map (“default_placement”)
  • November 27, 2014
    Ceph: recover OSDs after SSD journal failure

    A common recommendation is to store OSD journal on a SSD drive which implies loosing your OSD if this journal fails.
    This article assumes that your OSDs have been originally deployed with ceph-disk.
    You will also realise that it’s really simple to br…

  • November 25, 2014
    HOWTO debug a teuthology task

    To debug a modification to a ceph-qa-suite task ( for instance repair_test.py), a teuthology target is locked with: $ ./virtualenv/bin/teuthology-lock –lock-many 1 –owner loic@dachary.org $ ./virtualenv/bin/teuthology-lock –list-targets –owner loic@dachary.org > targets.yaml and used to run the test with: ./virtualenv/bin/teuthology \ … Continue reading

  • November 17, 2014
    Ceph RBD With LXC Containers

    A simple way to secure your data with containers is to use a distributed storage such as Ceph for LXC root storage.

    For exemple :

    1
    
    # lxc-create -n mycontainer -t debian -B rbd --pool rbd --rbd mycontainer --fstype ext4 --fssize 500
    

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    
    mke2fs 1.42.5 (29-Jul-2012)
    Filesystem label=
    OS type: Linux
    Block size=1024 (log=0)
    Fragment size=1024 (log=0)
    Stride=4096 blocks, Stripe width=4096 blocks
    128016 inodes, 512000 blocks
    25600 blocks (5.00%) reserved for the super user
    First data block=1
    Maximum filesystem blocks=67633152
    63 block groups
    8192 blocks per group, 8192 fragments per group
    2032 inodes per group
    Superblock backups stored on blocks:
      8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
    
    Allocating group tables: done                            
    Writing inode tables: done                            
    Creating journal (8192 blocks): done
    Writing superblocks and filesystem accounting information: done 
    
    Note: Usually the template option is called with a configuration
    file option too, mostly to configure the network.
    For more information look at lxc.conf (5)
    
    debootstrap is /usr/sbin/debootstrap
    Checking cache download in /var/cache/lxc/debian/rootfs-wheezy-amd64 ...
    Copying rootfs to /var/lib/lxc/mycontainer/rootfs...Generating locales (this might take a while)...
      en_US.UTF-8... done
    Generation complete.
    update-rc.d: using dependency based boot sequencing
    update-rc.d: using dependency based boot sequencing
    update-rc.d: using dependency based boot sequencing
    update-rc.d: using dependency based boot sequencing
    
    Current default time zone: 'America/New_York'
    Local time is now:      Tue Nov 18 09:34:16 EST 2014.
    Universal Time is now:  Tue Nov 18 14:34:16 UTC 2014.
    
    Root password is 'root', please change !
    'debian' template installed
    'mycontainer' created
    
    1
    2
    
    # mount | grep mycontainer
    /dev/rbd1 on /var/lib/lxc/mycontainer/rootfs type ext4 (rw,relatime,stripe=4096,data=ordered)
    

    Diff file for lxc-create :

    1
    
    # diff -u /usr/bin/lxc-create.orig /usr/bin/lxc-create
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    
    --- /usr/bin/lxc-create.orig    2014-11-17 04:16:41.181942000 -0500
    +++ /usr/bin/lxc-create  2014-11-17 04:35:27.225942000 -0500
    @@ -24,6 +24,7 @@
         echo "usage: lxc-create -n <name> [-f configuration] [-t template] [-h] [fsopts] -- [template_options]"
         echo "   fsopts: -B none"
         echo "   fsopts: -B lvm [--lvname lvname] [--vgname vgname] [--fstype fstype] [--fssize fssize]"
    +    echo "   fsopts: -B rbd [--pool poolname] [--rbd rbd] [--fstype fstype] [--fssize fssize]"
         echo "   fsopts: -B btrfs"
         echo "           flag is not necessary, if possible btrfs support will be used"
     #    echo "   fsopts: -B union [--uniontype overlayfs]"
    @@ -64,7 +65,7 @@
     }
    
     shortoptions='hn:f:t:B:'
    -longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,fssize:'
    +longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,pool:,rbd:,fssize:'
     localstatedir=/var
     lxc_path=${localstatedir}/lib/lxc
     bindir=/usr/bin
    @@ -119,6 +120,16 @@
          vgname=$1
          shift
          ;;
    +        --pool)
    +        shift
    +        pool=$1
    +        shift
    +        ;;
    +        --rbd)
    +        shift
    +        rbd=$1
    +        shift
    +        ;;
          --fstype)
          shift
          fstype=$1
    @@ -161,7 +172,7 @@
     fi
    
     case "$backingstore" in
    -    lvm|none|btrfs|_unset) :;;
    +    lvm|rbd|none|btrfs|_unset) :;;
         *) echo "'$backingstore' is not known ('none', 'lvm', 'btrfs')"
             usage
             exit 1
    @@ -216,6 +227,13 @@
             echo "please delete it (using \"lvremove $rootdev\") and try again"
             exit 1
         fi
    +elif [ "$backingstore" = "rbd" ]; then
    +    which rbd > /dev/null
    +    if [ $? -ne 0 ]; then
    +        echo "rbd command not found. Please install ceph-common package"
    +        exit 1
    +    fi
    +    rootdev=/dev/rbd/$pool/$rbd
     elif [ "$backingstore" = "btrfs" ]; then
         mkdir "$lxc_path/$lxc_name"
         if ! out=$(btrfs subvolume create "$rootfs" 2>&1); then
    @@ -257,6 +275,14 @@
         mkfs -t $fstype $rootdev || exit 1
         mount -t $fstype $rootdev $rootfs
     fi
    +
    +if [ $backingstore = "rbd" ]; then
    +    [ -d "$rootfs" ] || mkdir $rootfs
    +    rbd create $pool/$rbd --size=$fssize || exit 1
    +    rbd map $pool/$rbd || exit 1
    +    mkfs -t $fstype $rootdev || exit 1
    +    mount -t $fstype $rootdev $rootfs
    +fi
    
     if [ ! -z $lxc_template ]; then
    

    If you want to make persistent after reboot, you must add rbd in /etc/ceph/rbdmap and add line in fstab.

  • November 17, 2014
    Teuthology docker targets hack (2/4)

    The teuthology container hack is improved to snapshot the container after Ceph and its dependencies have been installed. It helps quickly testing ceph-qa-suite tasks. A job doing nothing but install the Firefly version of Ceph takes 14 seconds after the … Continue reading

  • November 14, 2014
    OpenNebula 4.8 With Ceph Support on Debian Wheezy

    A quick howto to install OpenNebula 4.8 with support for Ceph on Debian Wheezy.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    
    $ onedatastore show cephds
    DATASTORE 101 INFORMATION
    ID             : 101
    NAME           : cephds
    USER           : oneadmin
    GROUP          : oneadmin
    CLUSTER        : -
    TYPE           : IMAGE
    DS_MAD         : ceph
    TM_MAD         : ceph
    BASE PATH      : /var/lib/one//datastores/101
    DISK_TYPE      : RBD
    

    OpenNebula Installation

    OpenNebula Frontend

    Install OpenNebula repo

    wget -q -O- http://downloads.opennebula.org/repo/Ubuntu/repo.key | apt-key add -
    echo "deb http://downloads.opennebula.org/repo/4.8/Debian/7/ stable opennebula" > /etc/apt/sources.list.d/opennebula.list
    

    Download packages

    apt-get update
    apt-get install opennebula opennebula-sunstone nfs-kernel-server
    

    Configure and start the service

    sed -i -e 's/:host: 127.0.0.1/:host: 0.0.0.0/g' /etc/one/sunstone-server.conf
    /etc/init.d/opennebula-sunstone restart
    

    Export NFS

    echo "/var/lib/one/ *(rw,sync,no_subtree_check,root_squash)" >> /etc/exports
    service nfs-kernel-server restart
    

    Configure SSH Public Key

    su - oneadmin
    $ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
    
    
    $ cat << EOT > ~/.ssh/config
    Host *
        StrictHostKeyChecking no
        UserKnownHostsFile /dev/null
    EOT
    $ chmod 600 ~/.ssh/config
    

    OpenNebula Compute Nodes

    Install OpenNebula repo

    wget -q -O- http://downloads.opennebula.org/repo/Ubuntu/repo.key | apt-key add -
    echo "deb http://downloads.opennebula.org/repo/4.8/Debian/7/ stable opennebula" > /etc/apt/sources.list.d/opennebula.list
    

    Download packages

    apt-get update
    apt-get install opennebula-node nfs-common bridge-utils
    

    Network config

    vim /etc/network/interfaces
        ...
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off
    

    Fstab

    vim /etc/fstab
    10.2.0.130:/var/lib/one/  /var/lib/one/  nfs   soft,intr,rsize=8192,wsize=8192,noauto
    mount /var/lib/one/
    

    Qemu config

    cat << EOT > /etc/libvirt/qemu.conf
    user  = "oneadmin"
    group = "oneadmin"
    dynamic_ownership = 0
    EOT
    

    Ceph configuration

    I suppose that you have already a Ceph cluster running.

    Now create a pool “one” for Opennebula and create auth key.

    ceph osd pool create one 128 128
    ceph osd pool set one crush_ruleset 2
    ceph auth get-or-create client.one mon 'allow r' osd 'allow rwx pool=one'
    [client.one]
        key = AQCfTjVUeOPqIhAAiCAiBIgYd85fuMFT0dXVpA==
    

    Add Ceph firefly for support rbd on libvirt + qemu

    Before, you need compile rbd support for libvirt and qemu. You can have a look to this post : http://cephnotes.ksperis.com/blog/2013/09/12/using-ceph-rbd-with-libvirt-on-debian-wheezy (If you do not want to compile qemu you can directluy download those packages here : http://ksperis.com/files/qemu-kvm_1.1.2+dfsg-6_amd64.deb, http://ksperis.com/files/qemu-kvm_1.1.2+dfsg-6_amd64.deb)

    On each node, add Ceph firefly repo, and reinstall libvirt and qemu packages

    apt-get install lsb-release
    wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
    echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
    apt-get update
    apt-get install librados2 librbd1
    
    dpkg -i qemu-kvm_1.1.2+dfsg-6+deb7u4_amd64.deb qemu-utils_1.1.2+dfsg-6a+deb7u4_amd64.deb
    

    Create the secret for libvirt :

    echo "
    <secret ephemeral='no' private='no'>
       <usage type='ceph'>
         <name>client.one secret</name>
       </usage>
    </secret>" > secret.xml
    
    
    service libvirt reload
    

    Add ceph datastore on OpenNebula

    $ su - oneadmin
    $ vim ceph.one
    NAME         = cephds
    DS_MAD       = ceph
    TM_MAD       = ceph
    DISK_TYPE    = RBD
    POOL_NAME    = one
    BRIDGE_LIST  ="onenode1 onenode2 onenode3"
    CEPH_HOST    ="192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789"
    CEPH_SECRET  ="26a8b4d7-eb24-bf85-396d-fbf0x252e402"
    CEPH_USER    ="one"
    
    $ onedatastore create ceph.one
    ID: 101
    

    Example config :

    $ onedatastore show cephds
    DATASTORE 101 INFORMATION                                                       
    ID             : 101                 
    NAME           : cephds              
    USER           : oneadmin            
    GROUP          : oneadmin            
    CLUSTER        : -                   
    TYPE           : IMAGE               
    DS_MAD         : ceph                
    TM_MAD         : ceph                
    BASE PATH      : /var/lib/one//datastores/101
    DISK_TYPE      : RBD                 
    
    DATASTORE CAPACITY                                                              
    TOTAL:         : 15.9T               
    FREE:          : 13.4T               
    USED:          : 2.5T                
    LIMIT:         : -                   
    
    PERMISSIONS                                                                     
    OWNER          : um-                 
    GROUP          : u--                 
    OTHER          : ---                 
    
    DATASTORE TEMPLATE                                                              
    BASE_PATH="/var/lib/one//datastores/"
    BRIDGE_LIST="onenode1 onenode2 onenode3"
    CEPH_HOST="192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789"
    CEPH_SECRET="26a8b4d7-eb24-bf85-396d-fbf0x252e402"
    CEPH_USER="one"
    CLONE_TARGET="SELF"
    DISK_TYPE="RBD"
    DS_MAD="ceph"
    LN_TARGET="NONE"
    POOL_NAME="one"
    TM_MAD="ceph"
    TYPE="IMAGE_DS"
    
    IMAGES         
    9              
    10             
    11             
    12             
    16             
    21             
    24             
    27             
    28             
    29             
    31             
    32             
    

    Try import image in new datastore :

    $ oneimage create
        --name "CentOS-6.5_x86_64"
        --path "http://appliances.c12g.com/CentOS-6.5/centos6.5.qcow2.gz"
        --driver qcow2
        --datastore cephds
    

    Datastore view in Sunstone :

    More details here :

    http://ceph.com/community/ceph-support-in-opennebula-4-0/

    http://archives.opennebula.org/documentation:archives:rel4.0:ceph_ds

  • November 12, 2014
    v0.88 released

    This is the first development release after Giant. The two main features merged this round are the new AsyncMessenger (an alternative implementation of the network layer) from Haomai Wang at UnitedStack, and support for POSIX file locks in ceph-fuse and libcephfs from Yan, Zheng. There is also a big pile of smaller items that re …Read more

  • November 11, 2014
    OpenStack Glance: import images and convert them directly in Ceph

    Ceph, to work in optimal circumstances requires the usage of RAW images.
    However, it is painful to upload RAW images in Glance because it takes a while.
    Let see how we can make our life easier.

    First let’s upload our image, for the purpose o…

  • November 10, 2014
    Running make check on Ceph pull requests

    Each Ceph contribution is expected to successfully run make check and pass all the unit tests it contains. The developer runs make check locally before submitting his changes but the result may be influenced by the development environment. A draft … Continue reading

  • November 6, 2014
    make -j150 ceph

    A power8 machine was recently donated to the GCC compile farm and /proc/cpuinfo shows 160 processors. Compiling Ceph from sources with make -j150 makes for a nice htop display. The result of the compilation passes most of the unit tests, … Continue reading

  • November 3, 2014
    OpenStack Glance: disable cache management while using Ceph RBD

    The OpenStack documentation often recommends to enable the Glance cache while using the default store file, with the Ceph RBD backend things are slightly different.

    Depending on how you are consuming your Cloud platform, using the keystone+cachem…

Careers