Archives:

v10.0.0 released

This is the first development release for the Jewel cycle.  We are off to a good start, with lots of performance improvements flowing into the tree.  We are targeting sometime in Q1 2016 for the final Jewel.

NOTABLE CHANGES

  • build: cmake tweaks (pr#6254, John Spray)
  • build: more CMake package check fixes (pr#6108, Daniel Gryniewicz)
  • ceph-disk: get Nonetype when ceph-disk list with –format plain on single device. (pr#6410, Vicente Cheng)
  • ceph: fix tell behavior (pr#6329, David Zafman)
  • ceph-fuse: While starting ceph-fuse, start the log thread first (issue#13443, pr#6224, Wenjun Huang)
  • client: don’t mark_down on command reply (pr#6204, John Spray)
  • client: drop prefix from ints (pr#6275, John Coyle)
  • client: sys/file.h includes for flock operations (pr#6282, John Coyle)
  • cls_rbd: change object_map_update to return 0 on success, add logging (pr#6467, Douglas Fuller)
  • cmake: Use uname instead of arch. (pr#6358, John Coyle)
  • common: assert: __STRING macro is not defined by musl libc. (pr#6210, John Coyle)
  • common: fix OpTracker age histogram calculation (pr#5065, Zhiqiang Wang)
  • common/MemoryModel: Added explicit feature check for mallinfo(). (pr#6252, John Coyle)
  • common/obj_bencher.cc: fix verification crashing when there’s no objects (pr#5853, Piotr Dałek)
  • common: optimize debug logging (pr#6307, Adam Kupczyk)
  • common: Thread: move copy constructor and assignment op (pr#5133, Michal Jarzabek)
  • common: WorkQueue: new PointerWQ base class for ContextWQ (issue#13636, pr#6525, Jason Dillaman)
  • compat: use prefixed typeof extension (pr#6216, John Coyle)
  • crush: validate bucket id before indexing buckets array (issue#13477, pr#6246, Sage Weil)
  • doc: download GPG key from download.ceph.com (issue#13603, pr#6384, Ken Dreyer)
  • doc: fix outdated content in cache tier (pr#6272, Yuan Zhou)
  • doc/release-notes: v9.1.0 (pr#6281, Loic Dachary)
  • doc/releases-notes: fix build error (pr#6483, Kefu Chai)
  • doc: remove toctree items under Create CephFS (pr#6241, Jevon Qiao)
  • doc: rename the “Create a Ceph User” section and add verbage about… (issue#13502, pr#6297, ritz303)
  • docs: Fix styling of newly added mirror docs (pr#6127, Wido den Hollander)
  • doc, tests: update all http://ceph.com/ to download.ceph.com (pr#6435, Alfredo Deza)
  • doc: update doc for with new pool settings (pr#5951, Guang Yang)
  • doc: update radosgw-admin example (pr#6256, YankunLi)
  • doc: update the OS recommendations for newer Ceph releases (pr#6355, ritz303)
  • drop envz.h includes (pr#6285, John Coyle)
  • libcephfs: Improve portability by replacing loff_t type usage with off_t (pr#6301, John Coyle)
  • libcephfs: only check file offset on glibc platforms (pr#6288, John Coyle)
  • librados: fix examples/librados/Makefile error. (pr#6320, You Ji)
  • librados: init crush_location from config file. (issue#13473, pr#6243, Wei Luo)
  • librados: wrongly passed in argument for stat command (issue#13703, pr#6476, xie xingguo)
  • librbd: deadlock while attempting to flush AIO requests (issue#13726, pr#6508, Jason Dillaman)
  • librbd: fix enable objectmap feature issue (issue#13558, pr#6339, xinxin shu)
  • librbd: remove duplicate read_only test in librbd::async_flatten (pr#5856, runsisi)
  • mailmap: modify member info (pr#6468, Xiaowei Chen)
  • mailmap: updates (pr#6258, M Ranga Swami Reddy)
  • mailmap: Xie Xingguo affiliation (pr#6409, Loic Dachary)
  • mds: implement snapshot rename (pr#5645, xinxin shu)
  • mds: messages/MOSDOp: cast in assert to eliminate warnings (issue#13625, pr#6414, David Zafman)
  • mds: new filtered MDS tell commands for sessions (pr#6180, John Spray)
  • mds/Session: use projected parent for auth path check (issue#13364, pr#6200, Sage Weil)
  • mon: should not set isvalid = true when cephx_verify_authorizer return false (issue#13525, pr#6306, Ruifeng Yang)
  • osd: Add config option osd_read_ec_check_for_errors for testing (pr#5865, David Zafman)
  • osd: add pin/unpin support to cache tier (11066) (pr#6326, Zhiqiang Wang)
  • osd: auto repair EC pool (issue#12754, pr#6196, Guang Yang)
  • osd: drop the interim set from load_pgs() (pr#6277, Piotr Dałek)
  • osd: FileJournal: _fdump wrongly returns if journal is currently unreadable. (issue#13626, pr#6406, xie xingguo)
  • osd: FileStore: add a field indicate xattr only one chunk for set xattr. (pr#6244, Jianpeng Ma)
  • osd: FileStore: LFNIndex: remove redundant local variable ‘obj’. (issue#13552, pr#6333, xiexingguo)
  • osd: FileStore: potential memory leak if _fgetattrs fails (issue#13597, pr#6377, xie xingguo)
  • osd: FileStore: remove unused local variable ‘handle’ (pr#6381, xie xingguo)
  • osd: fix bogus scrub results when missing a clone (issue#12738, issue#12740, pr#5783, David Zafman)
  • osd: fix debug message in OSD::is_healthy (pr#6226, Xiaoxi Chen)
  • osd: fix MOSDOp encoding (pr#6174, Sage Weil)
  • osd: init started to 0 (issue#13206, pr#6107, Sage Weil)
  • osd: KeyValueStore: fix the name’s typo of keyvaluestore_default_strip_size (pr#6375, Zhi Zhang)
  • osd: new and delete ObjectStore::Transaction in a function is not necessary (pr#6299, Ruifeng Yang)
  • osd: optimize get_object_context (pr#6305, Jianpeng Ma)
  • osd: optimize MOSDOp/do_op/handle_op (pr#5211, Jacek J. Lakis)
  • osd: os/chain_xattr: On linux use linux/limits.h for XATTR_NAME_MAX. (pr#6343, John Coyle)
  • osd: reorder bool fields in PGLog struct (pr#6279, Piotr Dałek)
  • osd: ReplicatedPG: remove unused local variables (issue#13575, pr#6360, xiexingguo)
  • osd: reset primary and up_primary when building a new past_interval. (issue#13471, pr#6240, xiexingguo)
  • radosgw-admin: Checking the legality of the parameters (issue#13018, pr#5879, Qiankun Zheng)
  • radosgw-admin: Create –secret-key alias for –secret (issue#5821, pr#5335, Yuan Zhou)
  • radosgw-admin: metadata list user should return an empty list when user pool is empty (issue#13596, pr#6465, Orit Wasserman)
  • rados: new options for write benchmark (pr#6340, Joaquim Rocha)
  • rbd: fix clone isssue (issue#13553, pr#6334, xinxin shu)
  • rbd: fix init-rbdmap CMDPARAMS (issue#13214, pr#6109, Sage Weil)
  • rbdmap: systemd support (issue#13374, pr#6479, Boris Ranto)
  • rbd: rbdmap improvements (pr#6445, Boris Ranto)
  • release-notes: draft v0.94.4 release notes (pr#5907, Loic Dachary)
  • release-notes: draft v0.94.4 release notes (pr#6195, Loic Dachary)
  • release-notes: draft v0.94.4 release notes (pr#6238, Loic Dachary)
  • rgw: add compat header for TEMP_FAILURE_RETRY (pr#6294, John Coyle)
  • rgw: add default quota config (pr#6400, Daniel Gryniewicz)
  • rgw: add support for getting Swift’s DLO without manifest handling (pr#6206, Radoslaw Zarzynski)
  • rgw: clarify the error message when trying to create an existed user (pr#5938, Zeqiang Zhuang)
  • rgw: fix objects can not be displayed which object name does not cont… (issue#12963, pr#5738, Weijun Duan)
  • rgw: fix typo in RGWHTTPClient::process error message (pr#6424, Brad Hubbard)
  • rgw: fix wrong etag calculation during POST on S3 bucket. (issue#11241, pr#6030, Radoslaw Zarzynski)
  • rgw: mdlog trim add usage prompt (pr#6059, Weijun Duan)
  • rgw: modify the conditional statement in parse_metadata_key method. (pr#5875, Zengran Zhang)
  • rgw: refuse to calculate digest when the s3 secret key is empty (issue#13133, pr#6045, Sangdi Xu)
  • rgw: remove extra check in RGWGetObj::execute (issue#12352, pr#5262, Javier M. Mellid)
  • rgw: support core file limit for radosgw daemon (pr#6346, Guang Yang)
  • rgw: swift use Civetweb ssl can not get right url (issue#13628, pr#6408, Weijun Duan)
  • rocksdb: build with PORTABLE=1 (pr#6311, Sage Weil)
  • rocksdb: remove rdb source files from dist tarball (issue#13554, pr#6379, Kefu Chai)
  • rocksdb: use native rocksdb makefile (and our autotools) (pr#6290, Sage Weil)
  • rpm: ceph.spec.in: correctly declare systemd dependency for SLE/openSUSE (pr#6114, Nathan Cutler)
  • rpm: ceph.spec.in: fix libs-compat / devel-compat conditional (issue#12315, pr#5219, Ken Dreyer)
  • rpm: rhel 5.9 librados compile fix, moved blkid to RBD check/compilation (issue#13177, pr#5954, Rohan Mars)
  • scripts: release_notes can track original issue (pr#6009, Abhishek Lekshmanan)
  • test/libcephfs/flock: add sys/file.h include for flock operations (pr#6310, John Coyle)
  • test_rgw_admin: use freopen for output redirection. (pr#6303, John Coyle)
  • tests: allow docker-test.sh to run under root (issue#13355, pr#6173, Loic Dachary)
  • tests: ceph-disk workunit uses configobj (pr#6342, Loic Dachary)
  • tests: destroy testprofile before creating one (issue#13664, pr#6446, Loic Dachary)
  • tests: port uniqueness reminder (pr#6387, Loic Dachary)
  • tests: test/librados/test.cc must create profile (issue#13664, pr#6452, Loic Dachary)
  • tools/cephfs: fix overflow writing header to fixed size buffer (#13816) (pr#6617, John Spray)
  • tools: ceph-monstore-update-crush: add “–test” when testing crushmap (pr#6418, Kefu Chai)
  • tools:remove duplicate references (pr#5917, Bo Cai)
  • vstart: grant full access to Swift testing account (pr#6239, Yuan Zhou)
  • vstart: set cephfs root uid/gid to caller (pr#6255, John Spray)

GETTING CEPH

v0.80.11 Firefly released

This is a bugfix release for Firefly.  As the Firefly 0.80.x series is nearing its planned end of life in January 2016 it may also be the last.

We recommend that all Firefly users upgrade.

For more detailed information, see the complete changelog.

NOTABLE CHANGES

  • build/ops: /etc/init.d/radosgw restart does not work correctly (issue#11140, pr#5831, Dmitry Yatsushkevich)
  • build/ops: Fix -Wno-format and -Werror=format-security options clash (issue#13417, pr#6207, Boris Ranto)
  • build/ops: ceph-common needs python-argparse on older distros, but doesn’t require it (issue#12034, pr#5217, Nathan Cutler)
  • build/ops: ceph.spec.in running fdupes unnecessarily (issue#12301, pr#5224, Nathan Cutler)
  • build/ops: ceph.spec.in: 50-rbd.rules conditional is wrong (issue#12166, pr#5225, Nathan Cutler)
  • build/ops: ceph.spec.in: useless %py_requires breaks SLE11-SP3 build (issue#12351, pr#5394, Nathan Cutler)
  • build/ops: fedora21 has junit, not junit4 (issue#10728, pr#6203, Ken Dreyer, Loic Dachary)
  • build/ops: upstart: configuration is too generous on restarts (issue#11798, pr#5992, Sage Weil)
  • common: Client admin socket leaks file descriptors (issue#11535, pr#4633, Jon Bernard)
  • common: FileStore calls syncfs(2) even it is not supported (issue#12512, pr#5529, Danny Al-Gaaf, Kefu Chai, Jianpeng Ma)
  • common: HeartBeat: include types (issue#13088, pr#6038, Sage Weil)
  • common: Malformed JSON command output when non-ASCII strings are present (issue#7387, pr#4635, Kefu Chai, Tim Serong)
  • common: Memory leak in Mutex.cc, pthread_mutexattr_init without pthread_mutexattr_destroy (issue#11762, pr#5403, Ketor Meng)
  • common: Thread:pthread_attr_destroy(thread_attr) when done with it (issue#12570, pr#6325, Piotr Dałek, Zheng Qiankun)
  • common: ThreadPool add/remove work queue methods not thread safe (issue#12662, pr#5991, Jason Dillaman)
  • common: buffer: critical bufferlist::zero bug (issue#12252, pr#5388, Haomai Wang)
  • common: log: take mutex while opening fd (issue#12465, pr#5406, Samuel Just)
  • common: recursive lock of md_config_t (0) (issue#12614, pr#5814, Josh Durgin)
  • crush: take crashes due to invalid arg (issue#11602, pr#4769, Sage Weil)
  • doc: backport v0.80.10 release notes to firefly (issue#11090, pr#5307, Loic Dachary, Sage Weil)
  • doc: update docs to point to download.ceph.com (issue#13162, pr#5993, Alfredo Deza)
  • fs: MDSMonitor: handle MDSBeacon messages properly (issue#11590, pr#5199, Kefu Chai)
  • fs: client nonce collision due to unshared pid namespaces (issue#13032, pr#6087, Josh Durgin, Sage Weil)
  • librbd: Objectcacher setting max object counts too low (issue#7385, pr#4639, Jason Dillaman)
  • librbd: aio calls may block (issue#11056, pr#4854, Haomai Wang, Sage Weil, Jason Dillaman)
  • librbd: internal.cc: 1967: FAILED assert(watchers.size() == 1) (issue#12176, pr#5171, Jason Dillaman)
  • mon: Clock skew causes missing summary and confuses Calamari (issue#11877, pr#4867, Thorsten Behrens)
  • mon: EC pools are not allowed as cache pools, disallow in the mon (issue#11650, pr#5389, Samuel Just)
  • mon: Make it more difficult to delete pools in firefly (issue#11800, pr#4788, Sage Weil)
  • mon: MonitorDBStore: get_next_key() only if prefix matches (issue#11786, pr#5360, Joao Eduardo Luis)
  • mon: PaxosService: call post_refresh() instead of post_paxos_update() (issue#11470, pr#5358, Joao Eduardo Luis)
  • mon: add a cache layer over MonitorDBStore (issue#12638, pr#5698, Kefu Chai)
  • mon: adding exsting pool as tier with –force-nonempty clobbers removed_snaps (issue#11493, pr#5236, Sage Weil, Samuel Just)
  • mon: ceph fails to compile with boost 1.58 (issue#11576, pr#5129, Kefu Chai)
  • mon: does not check for IO errors on every transaction (issue#13089, pr#6091, Sage Weil)
  • mon: get pools health’info have error (issue#12402, pr#5410, renhwztetecs)
  • mon: increase globalid default for firefly (issue#13255, pr#6010, Sage Weil)
  • mon: pgmonitor: wrong at/near target max“ reporting (issue#12401, pr#5409, huangjun)
  • mon: register_new_pgs() should check ruleno instead of its index (issue#12210, pr#5404, Xinze Chi)
  • mon: scrub error (osdmap encoding mismatch?) upgrading from 0.80 to ~0.80.2 (issue#8815, issue#8674, issue#9064, pr#5200, Sage Weil, Zhiqiang Wang, Samuel Just)
  • mon: the output is wrong when runing ceph osd reweight (issue#12251, pr#5408, Joao Eduardo Luis)
  • objecter: can get stuck in redirect loop if osdmap epoch == last_force_op_resend (issue#11026, pr#4597, Jianpeng Ma, Sage Weil)
  • objecter: pg listing can deadlock when throttling is in use (issue#9008, pr#5043, Guang Yang)
  • objecter: resend linger ops on split (issue#9806, pr#5062, Josh Durgin, Samuel Just)
  • osd: Cleanup boost optionals for boost 1.56 (issue#9983, pr#5039, William A. Kennington III)
  • osd: LibRadosTwoPools[EC]PP.PromoteSnap failure (issue#10052, pr#5050, Sage Weil)
  • osd: Mutex Assert from PipeConnection::try_get_pipe (issue#12437, pr#5815, David Zafman)
  • osd: PG stuck with remapped (issue#9614, pr#5044, Guang Yang)
  • osd: PG::handle_advance_map: on_pool_change after handling the map change (issue#12809, pr#5988, Samuel Just)
  • osd: PGLog: split divergent priors as well (issue#11069, pr#4631, Samuel Just)
  • osd: PGLog::proc_replica_log: correctly handle case where entries between olog.head and log.tail were split out (issue#11358, pr#5287, Samuel Just)
  • osd: WBThrottle::clear_object: signal on cond when we reduce throttle values (issue#12223, pr#5822, Samuel Just)
  • osd: cache full mode still skips young objects (issue#10006, pr#5051, Xinze Chi, Zhiqiang Wang)
  • osd: crash creating/deleting pools (issue#12429, pr#5526, John Spray)
  • osd: explicitly specify OSD features in MOSDBoot (issue#10911, pr#4960, Sage Weil)
  • osd: is_new_interval() fixes (issue#11771, issue#10399, pr#5726, Samuel Just, Jason Dillaman)
  • osd: make the all osd/filestore thread pool suicide timeouts separately configurable (issue#11439, pr#5823, Samuel Just)
  • osd: object creation by write cannot use an offset on an erasure coded pool (issue#11507, pr#4632, Jianpeng Ma, Loic Dachary)
  • osd: os/FileJournal: Fix journal write fail, align for direct io (issue#12943, pr#5619, Xie Rui)
  • osd: osd/PGLog.cc: 732: FAILED assert(log.log.size() == log_keys_debug.size()) (issue#12652, pr#5820, Sage Weil)
  • osd: read on chunk-aligned xattr not handled (issue#12309, pr#5235, Sage Weil)
  • rgw: Change variable length array of std::strings (not legal in C++) to std::vector<std::string> (issue#12467, pr#4583, Daniel J. Hofmann)
  • rgw: Civetweb RGW appears to report full size of object as downloaded when only partially downloaded (issue#11851, pr#5234, Yehuda Sadeh)
  • rgw: Keystone PKI token expiration is not enforced (issue#11367, pr#4765, Anton Aksola)
  • rgw: Object copy bug (issue#11639, pr#4762, Javier M. Mellid)
  • rgw: RGW returns requested bucket name raw in “Bucket” response header (issue#11860, issue#12537, pr#5730, Yehuda Sadeh, Wido den Hollander)
  • rgw: Swift API: response for PUT on /container does not contain the mandatory Content-Length header when FCGI is used (issue#11036, pr#5170, Radoslaw Zarzynski)
  • rgw: content length parsing calls strtol() instead of strtoll() (issue#10701, pr#5997, Yehuda Sadeh)
  • rgw: delete bucket does not remove .bucket.meta file (issue#11149, pr#4641, Orit Wasserman)
  • rgw: doesn’t return ‘x-timestamp’ in header which is used by ‘View Details’ of OpenStack (issue#8911, pr#4584, Yehuda Sadeh)
  • rgw: init some manifest fields when handling explicit objs (issue#11455, pr#5729, Yehuda Sadeh)
  • rgw: logfile does not get chowned properly (issue#12073, pr#5233, Thorsten Behrens)
  • rgw: logrotate.conf calls service with wrong init script name (issue#12043, pr#5390, wuxingyi)
  • rgw: quota not respected in POST object (issue#11323, pr#4642, Sergey Arkhipov)
  • rgw: swift smoke test fails on TestAccountUTF8 (issue#11091, issue#11438, issue#12939, issue#12157, issue#12158, issue#12363, pr#5532, Radoslaw Zarzynski, Orit Wasserman, Robin H. Johnson)
  • rgw: use correct objv_tracker for bucket instance (issue#11416, pr#4535, Yehuda Sadeh)
  • tests: ceph-fuse crash in test_client_recovery (issue#12673, pr#5813, Loic Dachary)
  • tests: kernel_untar_build fails on EL7 (issue#11758, pr#6000, Greg Farnum)
  • tests: qemu workunit refers to apt-mirror.front.sepia.ceph.com (issue#13420, pr#6328, Yuan Zhou, Sage Weil)
  • tools: src/ceph-disk : disk zap sgdisk invocation (issue#11143, pr#4636, Thorsten Behrens, Owen Synge)
  • tools: ceph-disk: sometimes the journal symlink is not created (issue#10146, pr#5541, Dan van der Ster)
  • tools: ceph-disk: support NVMe device partitions (issue#11612, pr#4771, Ilja Slepnev)
  • tools: ceph-post-file fails on rhel7 (issue#11836, pr#5037, Joseph McDonald, Sage Weil)
  • tools: ceph_argparse_flag has no regular 3rd parameter (issue#11543, pr#4582, Thorsten Behrens)
  • tools: use a new disk as journal disk,ceph-disk prepare fail (issue#10983, pr#4630, Loic Dachary)

GETTING CEPH

monash-clayton

In case you didn’t know, Ceph Days are a series of regular events in support of the Ceph community that happen at locations around the world from New York to Berlin, and more recently in APAC to Shanghai and Melbourne.

The Melbourne edition on November 5th, 2015 was Australia’s first ever Ceph Day. It was hosted at the Clayton campus of sponsor Monash University, about a $60 Uber ride from the city centre. Monash is also home to a part of Australia’s national research cloud: NeCTAR, including Ceph-backed storage capacity provided as part of the Research Data Services initiative.

Watch this video to get a feel for the day in about 2 minutes.

read more…

v9.2.0 Infernalis released

InfernalisThis major release will be the foundation for the next stable series. There have been some major changes since v0.94.x Hammer, and the upgrade process is non-trivial. Please read these release notes carefully.

MAJOR CHANGES FROM HAMMER

read more…

v0.94.5 Hammer released

This Hammer point release fixes a critical regression in librbd that can cause Qemu/KVM to crash when caching is enabled on images that have been cloned.

All v0.94.4 Hammer users are strongly encouraged to upgrade.

NOTABLE CHANGES

  • librbd: potential assertion failure during cache read (issue#13559, pr#6348, Jason Dillaman)
  • osd: osd/ReplicatedPG: remove stray debug line (issue#13455, pr#6362, Sage Weil)
  • tests: qemu workunit refers to apt-mirror.front.sepia.ceph.com (issue#13420, pr#6330, Yuan Zhou)

For more detailed information, see the complete changelog.

GETTING CEPH

Ceph read-only mirror on gitlab

The gitlab-mirrors scripts are installed to setup a a read-only Ceph mirror, updated hourly. It is used for permalinks such as src/osd/ClassHandler.cc#L170.

The gitlab-mirrors config.sh is as follows:

#Environment file

#
# gitlab-mirrors settings
#

#The user git-mirrors will run as.
system_user="gitmirror"
#The home directory path of the $system_user
user_home="/home/${system_user}"
#The repository directory where gitlab-mirrors will contain copies of mirrored
#repositories before pushing them to gitlab.
repo_dir="${user_home}/repositories"
#colorize output of add_mirror.sh, update_mirror.sh, and git-mirrors.sh
#commands.
enable_colors=true
#These are additional options which should be passed to git-svn.  On the command
#line type "git help svn"
git_svn_additional_options="-s"
#Force gitlab-mirrors to not create the gitlab remote so a remote URL must be
#provided. (superceded by no_remote_set)
no_create_set=false
#Force gitlab-mirrors to only allow local remotes only.
no_remote_set=false
#Enable force fetching and pushing.  Will overwrite references if upstream
#forced pushed.  Applies to git projects only.
force_update=false
#This option is for pruning mirrors.  If a branch is deleted upstream then that
#change will propagate into your GitLab mirror.  Aplies to git projects only.
prune_mirrors=false

#
# Gitlab settings
#

#This is the Gitlab group where all project mirrors will be grouped.
gitlab_namespace="Ceph"
#This is the base web url of your Gitlab server.
gitlab_url="http://workbench.dachary.org"
#Special user you created in Gitlab whose only purpose is to update mirror sites
#and admin the $gitlab_namespace group.
gitlab_user="gitmirror"
#Generate a token for your $gitlab_user and set it here.
gitlab_user_token_secret="$(head -n1 "${user_home}/private_token" 2> /dev/null || echo "")"
#Verify signed SSL certificates?
ssl_verify=false
#Push to GitLab over http?  Otherwise will push projects via SSH.
http_remote=false

#
# Gitlab new project default settings.  If a project needs to be created by
# gitlab-mirrors then it will assign the following values as defaults.
#

#values must be true or false
issues_enabled=false
wall_enabled=false
wiki_enabled=false
snippets_enabled=false
merge_requests_enabled=true
public=true

It is configured for mirrors to be accessible to unauthenticated users by default public=true and disables SSL verification because the gitlab does not have https anyway : ssl_verify=false.
Although the three easy steps contain many sub steps, they can be followed to completion without problems.
For fine control over what is mirrored (release branches, tags and pull requests from github), the following can be used instead:

#!/bin/bash
git fetch --force origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* +refs/pull/*:refs/pull/*
git for-each-ref 'refs/pull/*/head' | cut -f2 | xargs --no-run-if-empty --max-args 1 git update-ref -d
git push --prune --force gitlab $(for branch in dumpling emperor firefly giant next master ; do echo +refs/heads/$branch:refs/heads/$branch ; done) +refs/pull/*:refs/heads/pull/*
git prune

The merge ref exists if the head of the pull request refs/pull/XXX/head can successfully be merged. They are removed to keep only one ref per pull request. All pull request refs are mapped under the refs/head so that they are noticed by GitLab CI. If they were kept under refs/pull, they would not be.
It is run hourly with:

$ cat ~/crontab
@hourly ( date ; /home/gitmirror/mirror.sh ) > /home/gitmirror/cron.log 2>&1

Placement_pools on Rados-GW

The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive…

1
2
   standard_bucket datas  --> .rgw.buckets        (default pool)
   specific_bucket datas  --> .rgw.buckets.custom

First, we create a pool .rgw.buckets.custom, with, for example, some specific parameters (different size and different ruleset in crushmap) :

1
2
3
4
5
6
7
8
$ ceph osd pool create .rgw.buckets.custom 64 64
pool '.rgw.buckets.custom' created

$ ceph osd pool set .rgw.buckets.custom size 2
set pool 59 size to 2

$ ceph osd pool set .rgw.buckets.custom crush_ruleset 6
set pool 59 crush_ruleset to 6

Then, we need to configure a specific placement_targets in region map and zone.
For next step, you need to have a running config of rados-gw…

1
2
$ radosgw-admin region get > region.conf.json
$ vim region.conf.json            # Add an entry in placement_targets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{ "name": "default",
  "api_name": "",
  "is_master": "true",
  "endpoints": [],
  "master_zone": "",
  "zones": [
        { "name": "default",
          "endpoints": [],
          "log_meta": "false",
          "log_data": "false"}],
  "placement_targets": [
        { "name": "default-placement",
          "tags": []},
        { "name": "custom-placement",
          "tags": []}],
  "default_placement": "default-placement"}
1
2
$ radosgw-admin region set < region.conf.json
....
1
2
$ radosgw-admin zone get > zone.conf.json
$ vim zone.conf.json            # Add an entry in placement_pools with key "custom-placement"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{ "domain_root": ".rgw",
  "control_pool": ".rgw.control",
  "gc_pool": ".rgw.gc",
  "log_pool": ".log",
  "intent_log_pool": ".intent-log",
  "usage_log_pool": ".usage",
  "user_keys_pool": ".users",
  "user_email_pool": ".users.email",
  "user_swift_pool": ".users.swift",
  "user_uid_pool": ".users.uid",
  "system_key": { "access_key": "",
      "secret_key": ""},
  "placement_pools": [
        { "key": "default-placement",
          "val": { "index_pool": ".rgw.buckets.index",
              "data_pool": ".rgw.buckets",
              "data_extra_pool": ".rgw.buckets.extra"}},
        { "key": "custom-placement",
          "val": { "index_pool": ".rgw.buckets.index",
              "data_pool": ".rgw.buckets.custom",
              "data_extra_pool": ".rgw.buckets.extra"}}]}
1
2
3
$ radosgw-admin zone set <zone.conf.json
2014-11-25 18:03:23.894153 7f728c0f2780  0 couldn't find old data placement pools config, setting up new ones for the zone
.....
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ radosgw-admin regionmap update
{ "regions": [
        { "key": "default",
          "val": { "name": "default",
              "api_name": "",
              "is_master": "true",
              "endpoints": [],
              "master_zone": "",
              "zones": [
                    { "name": "default",
                      "endpoints": [],
                      "log_meta": "false",
                      "log_data": "false"}],
              "placement_targets": [
                    { "name": "custom-placement",
                      "tags": []},
                    { "name": "default-placement",
                      "tags": []}],
              "default_placement": "default-placement"}}],
  "master_region": "default",
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}

$ /etc/init.d/radosgw reload
Reloading ...

To configure s3cmd for RadosGW you can have a look here :
http://lollyrock.com/articles/s3cmd-with-radosgw/

Now we can test bucket creation :

1
2
3
4
5
6
7
8
9
$ s3cmd mb s3://custombucket --bucket-location=custom-placement
Bucket 'custombucket' created

$ touch "file_on_custom_pool"

$ s3cmd put file_on_custom_pool s3://custombucket
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
file_on_custom_pool -> s3://custombucket/file_on_custom_pool  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

Verify :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ radosgw-admin bucket stats --bucket=custombucket
{ "bucket": "custombucket",
  "pool": ".rgw.buckets.custom",
  "index_pool": ".rgw.buckets.index",
  "id": "default.240909.1",
  "marker": "default.240909.1",
  "owner": "testuser",
  "ver": 1,
  "master_ver": 0,
  "mtime": 1417016078,
  "max_marker": "",
  "usage": {},
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}

Pool var is set on “.rgw.buckets.custom”.

1
2
$ rados -p .rgw.buckets.custom ls
default.241071.1_file_on_custom_pool

It’s here !

Data placement pool is define in this order :

  1. from the request (“bucket location”)
  2. from user (“default_placement” : see with radosgw-admin metadata get user:<uid>)
  3. from region map (“default_placement”)

A common recommendation is to store OSD journal on a SSD drive which implies loosing your OSD if this journal fails.
This article assumes that your OSDs have been originally deployed with ceph-disk.
You will also realise that it’s really simple to bring your OSDs back to life after replacing your faulty SSD with a new one.

read more…

HOWTO debug a teuthology task

To debug a modification to a ceph-qa-suite task ( for instance repair_test.py), a teuthology target is locked with:

$ ./virtualenv/bin/teuthology-lock --lock-many 1 --owner loic@dachary.org
$ ./virtualenv/bin/teuthology-lock --list-targets --owner loic@dachary.org > targets.yaml

and used to run the test with:

./virtualenv/bin/teuthology \
  --suite-path $HOME/software/ceph/ceph-qa-suite \
  --owner loic@dachary.org \
  $HOME/software/ceph/ceph-qa-suite/suites/rados/basic/tasks/repair_test.yaml \
  roles.yaml

where roles.yaml sets all roles to one target:

roles:
- [mon.0, osd.0, osd.1, osd.2, osd.3, osd.4, client.0]

Each run requires the installation and deinstallation of all Ceph packages and takes minutes. The installation part of repair_test.yaml can be commented out and the packages installed manually.

$ cat repair.yaml
...
tasks:
#- install:
- ceph:
- repair_test:


The existing packages are removed:

sudo apt-get remove --purge python-ceph ceph-common librados2 librbd1 ceph-fuse libcephfs1-dbg libcephfs-java libcephfs1 libcephfs-jni ceph-fs-common

The repository is set to the desired branch (for instance wip-10018-primary-erasure-code-hinfo), as compiled by gitbuilder.cgi with:

$ echo deb http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ref/wip-10018-primary-erasure-code-hinfo trusty main | sudo tee /etc/apt/sources.list.d/ceph.list

The packages are installed:

$ sudo apt-get update
$ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph ceph-dbg ceph-mds ceph-mds-dbg ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-test ceph-test-dbg radosgw radosgw-dbg python-ceph libcephfs1 libcephfs1-dbg libcephfs-java librados2 librados2-dbg librbd1 librbd1-dbg

and the repositories cleaned (this must be done again after each test completes):

$ rm -fr cephtest/ && sudo rm -fr /var/lib/ceph && sudo rm -fr /var/log/ceph/*

Utilities must be copied from the teuthology sources with (assuming vpm178 is the target):

$ scp teuthology/task/daemon-helper teuthology/task/adjust-ulimits ubuntu@vpm178:/tmp
$ ssh ubuntu@vpm178 sudo cp -a /tmp/{daemon-helper,adjust-ulimits} /usr/bin

Ceph RBD With LXC Containers

A simple way to secure your data with containers is to use a distributed storage such as Ceph for LXC root storage.

For exemple :

1
# lxc-create -n mycontainer -t debian -B rbd --pool rbd --rbd mycontainer --fstype ext4 --fssize 500

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=4096 blocks, Stripe width=4096 blocks
128016 inodes, 512000 blocks
25600 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
63 block groups
8192 blocks per group, 8192 fragments per group
2032 inodes per group
Superblock backups stored on blocks:
  8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 

Note: Usually the template option is called with a configuration
file option too, mostly to configure the network.
For more information look at lxc.conf (5)

debootstrap is /usr/sbin/debootstrap
Checking cache download in /var/cache/lxc/debian/rootfs-wheezy-amd64 ...
Copying rootfs to /var/lib/lxc/mycontainer/rootfs...Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing

Current default time zone: 'America/New_York'
Local time is now:      Tue Nov 18 09:34:16 EST 2014.
Universal Time is now:  Tue Nov 18 14:34:16 UTC 2014.

Root password is 'root', please change !
'debian' template installed
'mycontainer' created
1
2
# mount | grep mycontainer
/dev/rbd1 on /var/lib/lxc/mycontainer/rootfs type ext4 (rw,relatime,stripe=4096,data=ordered)

Diff file for lxc-create :

1
# diff -u /usr/bin/lxc-create.orig /usr/bin/lxc-create
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
--- /usr/bin/lxc-create.orig    2014-11-17 04:16:41.181942000 -0500
+++ /usr/bin/lxc-create  2014-11-17 04:35:27.225942000 -0500
@@ -24,6 +24,7 @@
     echo "usage: lxc-create -n <name> [-f configuration] [-t template] [-h] [fsopts] -- [template_options]"
     echo "   fsopts: -B none"
     echo "   fsopts: -B lvm [--lvname lvname] [--vgname vgname] [--fstype fstype] [--fssize fssize]"
+    echo "   fsopts: -B rbd [--pool poolname] [--rbd rbd] [--fstype fstype] [--fssize fssize]"
     echo "   fsopts: -B btrfs"
     echo "           flag is not necessary, if possible btrfs support will be used"
 #    echo "   fsopts: -B union [--uniontype overlayfs]"
@@ -64,7 +65,7 @@
 }

 shortoptions='hn:f:t:B:'
-longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,fssize:'
+longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,pool:,rbd:,fssize:'
 localstatedir=/var
 lxc_path=${localstatedir}/lib/lxc
 bindir=/usr/bin
@@ -119,6 +120,16 @@
      vgname=$1
      shift
      ;;
+        --pool)
+        shift
+        pool=$1
+        shift
+        ;;
+        --rbd)
+        shift
+        rbd=$1
+        shift
+        ;;
      --fstype)
      shift
      fstype=$1
@@ -161,7 +172,7 @@
 fi

 case "$backingstore" in
-    lvm|none|btrfs|_unset) :;;
+    lvm|rbd|none|btrfs|_unset) :;;
     *) echo "'$backingstore' is not known ('none', 'lvm', 'btrfs')"
         usage
         exit 1
@@ -216,6 +227,13 @@
         echo "please delete it (using \"lvremove $rootdev\") and try again"
         exit 1
     fi
+elif [ "$backingstore" = "rbd" ]; then
+    which rbd > /dev/null
+    if [ $? -ne 0 ]; then
+        echo "rbd command not found. Please install ceph-common package"
+        exit 1
+    fi
+    rootdev=/dev/rbd/$pool/$rbd
 elif [ "$backingstore" = "btrfs" ]; then
     mkdir "$lxc_path/$lxc_name"
     if ! out=$(btrfs subvolume create "$rootfs" 2>&1); then
@@ -257,6 +275,14 @@
     mkfs -t $fstype $rootdev || exit 1
     mount -t $fstype $rootdev $rootfs
 fi
+
+if [ $backingstore = "rbd" ]; then
+    [ -d "$rootfs" ] || mkdir $rootfs
+    rbd create $pool/$rbd --size=$fssize || exit 1
+    rbd map $pool/$rbd || exit 1
+    mkfs -t $fstype $rootdev || exit 1
+    mount -t $fstype $rootdev $rootfs
+fi

 if [ ! -z $lxc_template ]; then

If you want to make persistent after reboot, you must add rbd in /etc/ceph/rbdmap and add line in fstab.

© 2016, Red Hat, Inc. All rights reserved.