Planet Ceph

Aggregated news from external sources

  • June 18, 2015
    Add Support of Curl_multi_wait for RadosGW on Debian Wheezy
    WARNING: libcurl doesn't support curl_multi_wait()
    WARNING: cross zone / region transfer performance may be affected
    

    If you have already been confronted to this error at startup of RadosGW, the problem is the version of libcurl used.
    To enable support of curl_multi_wait, you will need to compile radosgw with libcurl >= 7.28.0 :
    http://curl.haxx.se/libcurl/c/curl_multi_wait.html

    On debian wheezy, you can use ceph-extras repo which contains libcurl 7.29.0 to recompile ceph packages :

    # apt-cache policy libcurl4-gnutls-dev
    libcurl4-gnutls-dev:
      Installed: (none)
      Candidate: 7.26.0-1+wheezy13
    
    # echo  deb http://ceph.com/packages/ceph-extras/debian wheezy main | tee /etc/apt/sources.list.d/ceph-extras.list
    # apt-get update
    
    # apt-cache policy libcurl4-gnutls-dev
    libcurl4-gnutls-dev:
      Installed: (none)
      Candidate: 7.29.0-1~bpo70+1.ceph
    

    Retrieve Ceph repo on Github (in this example, I use hammer version) :

    # apt-get install git build-essential automake
    # git clone --recursive https://github.com/ceph/ceph.git -b hammer
    # cd ceph
    

    Install dependencies and build packages (without libbabeltrace-ctf-dev libbabeltrace-dev, here we not need…)

    # apt-get install autoconf automake autotools-dev libbz2-dev cryptsetup-bin debhelper default-jdk gdisk javahelper junit4 libaio-dev libatomic-ops-dev libblkid-dev libboost-dev libboost-program-options-dev libboost-system-dev libboost-thread-dev libcurl4-gnutls-dev libedit-dev libexpat1-dev libfcgi-dev libfuse-dev libgoogle-perftools-dev libkeyutils-dev libleveldb-dev libnss3-dev libsnappy-dev liblttng-ust-dev libtool libudev-dev libxml2-dev parted pkg-config python-nose python-virtualenv sdparm uuid-dev uuid-runtime xfslibs-dev xfsprogs xmlstarlet yasm zlib1g-dev
    
    # dpkg-buildpackage -d
    

    On RadosGW host, you will need to add “ceph-extras” repo (for libcurl) and install radosgw packages and dependencies :

    # echo  deb http://ceph.com/packages/ceph-extras/debian wheezy main | tee /etc/apt/sources.list.d/ceph-extras.list
    # apt-get update
    
    # dpkg -i ceph-common_*.deb librbd1_*.deb python-cephfs_*.deb python-rbd_*.deb librados2_*.deb python-ceph_*.deb python-rados_*.deb radosgw_*.deb
    
  • June 18, 2015
    Deploy Ceph monitors on Kubernetes

    As part of my work on ceph-docker, I have been playing a bit with Kubernetes.
    As a first exercise, I thought it would be interesting to run my Ceph monitors with Kubernetes.
    Let’s see how we can achieve that.

    The following assumes that you have a Kubernetes cluster up and running.

    “`bash
    $ wget https://raw.githubusercontent.com/ceph/ceph-docker/master/examples/kubernetes/ceph-mon.json

    $ kubectl create -f mon.json
    replicationControllers/ceph-mon

    $ kubectl get pod
    POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE
    ceph-mon-qo4yc 127.0.0.1/ name=frontend Pending 3 seconds

                           ceph-mon       ceph/daemon
    

    ceph-mon-sv56v 127.0.0.1/ name=frontend Pending 3 seconds

                           ceph-mon       ceph/daemon
    

    ceph-mon-zc0lb 127.0.0.1/ name=frontend Pending 3 seconds

                           ceph-mon       ceph/daemon
    

    $ kubectl log ceph-mon-qo4yc
    creating /etc/ceph/ceph.client.admin.keyring
    creating /etc/ceph/ceph.mon.keyring
    creating /var/lib/ceph/bootstrap-osd/ceph.keyring
    creating /var/lib/ceph/bootstrap-mds/ceph.keyring
    creating /var/lib/ceph/bootstrap-rgw/ceph.keyring
    monmaptool: monmap file /etc/ceph/monmap
    monmaptool: set fsid to ef67b6f1-8d50-49fe-a8dd-c81bd7dd09c2
    monmaptool: writing epoch 0 to /etc/ceph/monmap (1 monitors)
    creating /tmp/ceph.mon.keyring
    importing contents of /etc/ceph/ceph.client.admin.keyring into /tmp/ceph.mon.keyring
    importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into /tmp/ceph.mon.keyring
    importing contents of /var/lib/ceph/bootstrap-mds/ceph.keyring into /tmp/ceph.mon.keyring
    importing contents of /var/lib/ceph/bootstrap-rgw/ceph.keyring into /tmp/ceph.mon.keyring
    importing contents of /etc/ceph/ceph.mon.keyring into /tmp/ceph.mon.keyring
    ceph-mon: set fsid to b2beb954-6791-408c-8212-9559fd5daf36
    ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-mon-qo4yc for mon.ceph-mon-qo4yc
    2015-06-18 09:29:11.347368 7f4924a168c0 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-mon, pid 1
    2015-06-18 09:29:11.717765 7f4924a168c0 0 mon.ceph-mon-qo4yc does not exist in monmap, will attempt to join an existing cluster
    2015-06-18 09:29:11.718113 7f4924a168c0 0 using public_addr 172.17.0.11:6789/0 -> 172.17.0.11:6789/0
    2015-06-18 09:29:11.718225 7f4924a168c0 0 starting mon.ceph-mon-qo4yc rank -1 at 172.17.0.11:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-mon-qo4yc fsid b2beb954-6791-408c-8212-9
    559fd5daf36




    “`

    It seems to run fine, now let’s have a look at the Ceph status:

    “`bash
    $ sudo docker ps
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    1f86ffbd3897 ceph/daemon:latest “/entrypoint.sh” 18 seconds ago Up 17 seconds k8s_ceph-mon.b9e89a17_ceph-mon-zc0lb_de
    fault_76b6b559-159c-11e5-9d52-fa163ebdff5b_895951f7
    a0f966c5cd54 ceph/daemon:latest “/entrypoint.sh” 26 seconds ago Up 23 seconds k8s_ceph-mon.b9e89a17_ceph-mon-qo4yc_de
    fault_76b6eefc-159c-11e5-9d52-fa163ebdff5b_cea0ff5e
    a9fa1663d712 ceph/daemon:latest “/entrypoint.sh” 26 seconds ago Up 24 seconds k8s_ceph-mon.b9e89a17_ceph-mon-sv56v_de
    fault_76b71555-159c-11e5-9d52-fa163ebdff5b_8d5b1f56
    fceabf9c1b77 gcr.io/google_containers/pause:0.8.0 “/pause” 28 seconds ago Up 26 seconds k8s_POD.e4cc795_ceph-mon-sv56v_default
    76b71555-159c-11e5-9d52-fa163ebdff5b_998441ff
    5f9eecd36fb5 gcr.io/google_containers/pause:0.8.0 “/pause” 28 seconds ago Up 26 seconds k8s_POD.e4cc795_ceph-mon-qo4yc_default

    76b6eefc-159c-11e5-9d52-fa163ebdff5b_3af31e82
    c99221438e93 gcr.io/google_containers/pause:0.8.0 “/pause” 28 seconds ago Up 26 seconds k8s_POD.e4cc795_ceph-mon-zc0lb_default_
    76b6b559-159c-11e5-9d52-fa163ebdff5b_dfc6b697

    $ sudo docker exec 1f86ffbd3897 ceph -s

    cluster b2beb954-6791-408c-8212-9559fd5daf36
     health HEALTH_ERR
            64 pgs stuck inactive
            64 pgs stuck unclean
            no osds
     monmap e3: 3 mons at {ceph-mon-qo4yc=172.17.0.11:6789/0,ceph-mon-sv56v=172.17.0.12:6789/0,ceph-mon-zc0lb=172.17.0.10:6789/0}
            election epoch 8, quorum 0,1,2 ceph-mon-zc0lb,ceph-mon-qo4yc,ceph-mon-sv56v
     osdmap e1: 0 osds: 0 up, 0 in
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating
    

    “`

    As we can see, all the three monitors successfully formed a quorum.

  • June 16, 2015
    Ceph is complaining: too many PGs

    Quick tip.

    Sometimes by running ceph -s, you can get a WARNING state saying:

    health HEALTH_WARN
    too many PGs per OSD (438 > max 300)

    To suppress this warning, append the following configuration options into your ceph.conf:

    mon_pg_warn_max_…

  • June 15, 2015
    Deutsche OpenStack Tage 2015

    Next week, from 23.-24.06.2015, the “German OpenStack Days” (Deutsche OpenStack Tage) take place in Frankfurt. I will give a talk about “Ceph in a security critical OpenStack Cloud” on Wednesday. The presentations are conducted in German, so it’s …

  • June 11, 2015
    v0.94.2 Hammer released

    This Hammer point release fixes a few critical bugs in RGW that can prevent objects starting with underscore from behaving properly and that prevent garbage collection of deleted objects when using the Civetweb standalone mode. All v0.94.x Hammer users are strongly encouraged to upgrade, and to make note of the repair procedure below if RGW …Read more

  • June 11, 2015
    v9.0.1 released

    This development release is delayed a bit due to tooling changes in the build environment. As a result the next one (v9.0.2) will have a bit more work than is usual. Highlights here include lots of RGW Swift fixes, RBD feature work surrounding the new object map feature, more CephFS snapshot fixes, and a few …Read more

  • June 6, 2015
    Teuthology docker targets hack (5/5)

    The teuthology container hack is improved to run teuthology-suite. For instance: ./virtualenv/bin/teuthology-suite \ –distro ubuntu \ –suite-dir $HOME/software/ceph/ceph-qa-suite \ –config-file docker-integration/teuthology.yaml \ –machine-type container \ –owner loic@dachary.org \ –filter ‘rados:basic/{clusters/fixed-2.yaml fs/btrfs.yaml \ msgr-failures/few.yaml tasks/rados_cls_all.yaml}’ \ –suite rados/basic –ceph ANY \ … Continue reading

  • June 3, 2015
    Ceph: activate RBD readahead

    {% img center http://sebastien-han.fr/images/ceph-rbd-readhead.jpg Ceph: activate RBD readhead %}

    RBD readahead was introduced with Giant.

    During the boot sequence of a virtual machine if the librbd detects contiguous reads, it will attempt to readahead on the OSDs and fill up the RBD cache with the content.
    When the OS reads it will get those reads from the librbd cache.
    Parameters that control the readahead:

    rbd readahead trigger requests = 10 # number of sequential requests necessary to trigger readahead.
    rbd readahead max bytes = 524288 # maximum size of a readahead request, in bytes.
    rbd readahead disable after bytes = 52428800
    

    Testing procedure

    The way I tested this is rather simple, I simply calculated the time it took to SSH into the virtual machine.
    I ran this test 10 times with and without the readahead in order to get an average value

    Execution script:

    bash
    for i in $(seq 1 10)
    do
    nova delete leseb > /dev/null 2>&1
    sleep 5
    nova boot --flavor m1.small --image 19dab28e-5d13-4d13-9fd4-dbc597fdccb7 leseb > /dev/null 2>&1
    time ./checkup.sh 10.0.0.2 22
    done

    Checkup script:

    “`bash

    !/bin/bash

    host=$1
    port=$2
    max=1000000
    counter=1

    while true
    do
    python -c “import socket;s = socket.socket(socket.AF_INET, socket.SOCK_STREAM);s.connect((‘$host’, $port))” > /dev/null 2>&1 && break || \
    echo -n “.”

    if [[ ${counter} == ${max} ]];then

      echo "Could not connect"
      exit 1
    

    fi
    (( counter++ ))
    done
    “`

    Boot time comparison

    At some point, I tried to look at the virtual machine logs and analysed the block size.
    I was hoping that using a more accurate value for rbd_readahead_max_bytes would bring me some befenit.
    So I queried the admin socket to hopefully get something useful about the reads that happen during the boot sequence:

    “`bash
    $ sudo ceph –admin-daemon /var/run/ceph/guests/ceph-client.cinder.463407.139639582721120.asok perf dump

        "flush": 0,
        "aio_rd": 5477,
        "aio_rd_bytes": 117972992,
        "aio_rd_latency": {
            "avgcount": 5477,
            "sum": 16.090880101
    


    “`

    Unfortunately I don’t see to get anything interesting, ideally I’d have gotten average reads.
    My last resort is to log every single read entries of the librbd.
    I used one of my previous article as a reference.
    Over 9903 reads during the boot sequence, it resulted the average read block size was 98304.
    I eventually decided to give it a try.

    Here are the results:

    {% img center http://sebastien-han.fr/images/ceph-rbd-readahead-boot-time-comp.jpg Ceph RBD readahead boot time comparison %}

    My second optimisation attempt was clearly the most successful since we are almost below 23 seconds to boot a virtual machine.
    In the meantime the default values are not that bad and sound pretty reasonnable.
    Thus sticking with the default should not be an issue.

  • June 1, 2015
    Ceph OSD daemon config diff

    {% img center http://sebastien-han.fr/images/ceph-osd-config-diff.jpg Ceph OSD daemon config diff %}

    Quick tip.
    Simply check the diff between the applied configuration in your ceph.conf and the default values on an OSD.

    “`bash
    $ sudo ceph daemon …

  • May 31, 2015
    Docker Containers in Just 10 Commands

    Docker container in 10 commands

    If you are on this page, then you definately know what is Docker , i will not take your time with the introduction part.

    Lets do Docker !!!

    • Install docker packages on your Linux host , in my case its CentOS.
    1
    
    # yum install -y docker-io
    

    • Start Docker service and enable it as a startup process.
    1
    
    # service docker start ; chkconfig docker on
    
    • Docker pull CentOS image
    1
    
    # docker pull centos:latest
    
    • Check docker images
    1
    2
    3
    4
    
    [root@karan-ws ~]# docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    docker.io/centos    latest              fd44297e2ddb        5 weeks ago         215.7 MB
    [root@karan-ws ~]#
    
    • Create Docker container
    1
    2
    3
    
    [root@karan-ws ~]# docker create -ti --name="mona" centos bash
    c7f9eb6b32eba38242b9d9ced309314f8eee720dbf29c656885aa0cbfff15aa6
    [root@karan-ws ~]#
    
    • Start your docker container
    1
    
    # docker start mona
    
    • Get IP address of your newly created docker container
    1
    2
    3
    
    [root@karan-ws ~]# docker inspect mona | grep -i ipaddress
            "IPAddress": "172.17.0.1",
    [root@karan-ws ~]#
    
    • Attach (login) to your docker container
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
    [root@karan-ws ~]# docker attach mona
    
    [root@c7f9eb6b32eb /]#
    [root@c7f9eb6b32eb /]# cat /etc/redhat-release
    CentOS Linux release 7.1.1503 (Core)
    [root@c7f9eb6b32eb /]# df -h
    Filesystem                                                                                          Size  Used Avail Use% Mounted on
    /dev/mapper/docker-253:1-16852579-c7f9eb6b32eba38242b9d9ced309314f8eee720dbf29c656885aa0cbfff15aa6  9.8G  268M  9.0G   3% /
    tmpfs                                                                                               1.6G     0  1.6G   0% /dev
    shm                                                                                                  64M     0   64M   0% /dev/shm
    tmpfs                                                                                               1.6G     0  1.6G   0% /run
    tmpfs                                                                                               1.6G     0  1.6G   0% /tmp
    /dev/vda1                                                                                            10G  1.6G  8.5G  16% /etc/hosts
    tmpfs                                                                                               1.6G     0  1.6G   0% /run/secrets
    tmpfs                                                                                               1.6G     0  1.6G   0% /proc/kcore
    [root@c7f9eb6b32eb /]#
    

    To detach from docker container use ctrl+p+q , avoid using exit command as it will stop container and exit.

    • List container
    1
    2
    3
    4
    
    [root@karan-ws ~]# docker ps
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    c7f9eb6b32eb        centos:latest       "bash"              9 minutes ago       Up 28 seconds                           mona
    [root@karan-ws ~]#
    
    • Stop and destroy container
    1
    2
    3
    4
    5
    6
    7
    
    [root@karan-ws ~]# docker stop mona ; docker kill mona
    mona
    mona
    [root@karan-ws ~]#
    [root@karan-ws ~]# docker ps
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    [root@karan-ws ~]#
    

    These are elementary basic docker operations that you can perform to take a feel of Docker Container technology. In future posts i will cover more advance docker topics. Stay Tuned !!!

  • May 30, 2015
    Ceph: Monitor Troubleshooting

    Ceph monitor ip change

    While playing with your Ceph cluster , you might have seen HEALTH_WARN cluster status.

    Cluster warning can occur due to several reasons of component malfunctioning such as MON,OSD,PG & MDS.

    In my case i saw warning due to Ceph monitors, which was like :

    1
    
    health HEALTH_WARN 1 mons down, quorum 0,1 ceph-mon1,ceph-mon2
    

    At first i tried restarting MON service , but no luck.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
    [root@ceph-mon3 ~]# service ceph status mon
    === mon.ceph-mon3 ===
    mon.ceph-mon3: not running.
    [root@ceph-mon3 ~]# service ceph start mon
    === mon.ceph-mon3 ===
    Starting Ceph mon.ceph-mon3 on ceph-mon3...
    Invalid argument: /var/lib/ceph/mon/ceph-ceph-mon3/store.db: does not exist (create_if_missing is false)
    IO error: /var/lib/ceph/mon/ceph-ceph-mon3/store.db/000001.dbtmp: Input/output error
    2015-05-22 11:44:38.065906 7fad6c6967a0 -1 failed to create new leveldb store
    failed: 'ulimit -n 131072;  /usr/bin/ceph-mon -i ceph-mon3 --pid-file /var/run/ceph/mon.ceph-mon3.pid -c /etc/ceph/ceph.conf --cluster ceph '
    Starting ceph-create-keys on ceph-mon3...
    [root@ceph-mon3 ~]#
    [root@ceph-mon3 ~]# service ceph status mon
    === mon.ceph-mon3 ===
    mon.ceph-mon3: not running.
    [root@ceph-mon3 ~]#
    

    The error message that i received was not something normal, so i started playing rough with my cluster , by moving monitor store.db files. !!! Be Cautious

    1
    
    mv /var/lib/ceph/mon/ceph-ceph-mon3/store.db /var/lib/ceph/mon/ceph-ceph-mon3/store.db.orig
    

    And this broke MON really badly, so now i know another way that causes a new error YAY 🙂

    1
    2
    3
    4
    5
    6
    7
    
    [root@ceph-mon3 ceph-ceph-mon3]# service ceph start mon
    === mon.ceph-mon3 ===
    Starting Ceph mon.ceph-mon3 on ceph-mon3...
    2015-05-22 11:59:45.385826 7faa43dfb7a0 -1 unable to read magic from mon data.. did you run mkcephfs?
    failed: 'ulimit -n 131072;  /usr/bin/ceph-mon -i ceph-mon3 --pid-file /var/run/ceph/mon.ceph-mon3.pid -c /etc/ceph/ceph.conf --cluster ceph '
    Starting ceph-create-keys on ceph-mon3...
    [root@ceph-mon3 ceph-ceph-mon3]#
    

    Show Time begins 🙂

    Then i started doing real work by reading monitor logs and what i found was monitor IP address were incorrect , they need to have a different address range.

    To fix this first we need to change monitor IP address to the correct rage.

    Changing Ceph Monitor IP Address

    • Get monitor maps , you could see the current IP range is 80.50.X.X , we need to change this to the correct range.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    
    [root@ceph-mon1 ~]# ceph mon getmap -o /tmp/monmap
    got monmap epoch 3
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]# monmaptool --print /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    epoch 3
    fsid 98d89661-f616-49eb-9ccf-84d720e179c0
    last_changed 2015-05-18 14:42:01.287460
    created 2015-05-18 14:41:00.514879
    0: 80.50.50.35:6789/0 mon.ceph-mon1
    1: 80.50.50.37:6789/0 mon.ceph-mon2
    2: 80.50.50.39:6789/0 mon.ceph-mon3
    [root@ceph-mon1 ~]#
    
    • Remove monitor nodes from monitor map
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    [root@ceph-mon1 ~]# monmaptool --rm ceph-mon1 /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    monmaptool: removing ceph-mon1
    monmaptool: writing epoch 3 to /tmp/monmap (2 monitors)
    [root@ceph-mon1 ~]# monmaptool --rm ceph-mon2 /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    monmaptool: removing ceph-mon2
    monmaptool: writing epoch 3 to /tmp/monmap (1 monitors)
    [root@ceph-mon1 ~]# monmaptool --rm ceph-mon3 /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    monmaptool: removing ceph-mon3
    monmaptool: writing epoch 3 to /tmp/monmap (0 monitors)
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]# monmaptool --print /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    epoch 3
    fsid 98d89661-f616-49eb-9ccf-84d720e179c0
    last_changed 2015-05-18 14:42:01.287460
    created 2015-05-18 14:41:00.514879
    [root@ceph-mon1 ~]#
    
    • Add the correct hostname and IP address for monitor nodes
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    
    [root@ceph-mon1 ~]# monmaptool --add ceph-mon1-ib 10.100.1.101:6789 /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    monmaptool: writing epoch 3 to /tmp/monmap (1 monitors)
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]# monmaptool --add ceph-mon2-ib 10.100.1.102:6789 /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    monmaptool: writing epoch 3 to /tmp/monmap (2 monitors)
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]# monmaptool --add ceph-mon3-ib 10.100.1.103:6789 /tmp/monmap
    monmaptool: monmap file /tmp/monmap
    monmaptool: writing epoch 3 to /tmp/monmap (3 monitors)
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ~]# monmaptool --print /tmp/monmap monmaptool: monmap file /tmp/monmap
    epoch 3
    fsid 98d89661-f616-49eb-9ccf-84d720e179c0
    last_changed 2015-05-18 14:42:01.287460
    created 2015-05-18 14:41:00.514879
    0: 10.100.1.101:6789/0 mon.ceph-mon1-ib
    1: 10.100.1.102:6789/0 mon.ceph-mon2-ib
    2: 10.100.1.103:6789/0 mon.ceph-mon3-ib
    [root@ceph-mon1 ~]#
    
    • Before injecting the new monmap , bring down monitor services and then inject the monmap.
    1
    2
    3
    4
    5
    6
    
    [root@ceph-mon1 ~]# service ceph stop mon
    === mon.ceph-mon1 ===
    Stopping Ceph mon.ceph-mon1 on ceph-mon1...kill 441540...done
    [root@ceph-mon1 ~]#
    [root@ceph-mon1 ceph]# ceph-mon -i ceph-mon1 --inject-monmap /tmp/monmap
    [root@ceph-mon1 ceph]#
    
    • Repeat these steps for the other monitors in your cluster , to save some time you can copy the new monmap file from first monitor node (ceph-mon1) to other monitor nodes and simply inject this new monmap into their ceph monitor instance.
    • Finally bring up the monitor services on all the monitor nodes.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    
    [root@ceph-mon1 ceph]# service ceph start mon
    === mon.ceph-mon1 ===
    Starting Ceph mon.ceph-mon1 on ceph-mon1...
    Starting ceph-create-keys on ceph-mon1...
    [root@ceph-mon1 ceph]#
    [root@ceph-mon1 ceph]# service ceph status mon
    === mon.ceph-mon1 ===
    mon.ceph-mon1: running {"version":"0.80.9"}
    [root@ceph-mon1 ceph]#
    

    If you still see monitor problems, you can redeploy the monitor node

    1
    
    [root@ceph-mon1 ceph]# ceph-deploy --overwrite-conf  mon create ceph-mon3
    
    • Finally your cluster should attain Health_OK status
    1
    2
    3
    4
    5
    6
    7
    8
    9
    
    [root@ceph-mon3 ceph]# ceph -s
        cluster 98d89661-f616-49eb-9ccf-84d720e179c0
         health HEALTH_OK
         monmap e4: 3 mons at {ceph-mon1=10.100.1.101:6789/0,ceph-mon2=10.100.1.102:6789/0,ceph-mon3=10.100.1.103:6789/0}, election epoch 18, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
         osdmap e244: 55 osds: 54 up, 54 in
          pgmap v693: 192 pgs, 3 pools, 0 bytes data, 0 objects
                5327 MB used, 146 TB / 146 TB avail
                     192 active+clean
    [root@ceph-mon3 ceph]#
    

    This might give you some idea of Ceph monitor troubleshooting. You could also follow more detailed steps mentioned in Ceph documentation.

  • May 29, 2015
    OpenStack Summit Vancouver

    Finally back from Vancouver, back from an interesting week at the OpenStack Summit including a quite packed schedule with many interesting presentations, work sessions and meetings. I presented together with Sage Weil about “Storage security in a …

Careers