Replacing a failed disk/OSD

From Ceph wiki

Let's assume you have an osd, say osd.23, whose disk(s) failed.

• Replace the physical disk, and/or deal with whatever hardware level stuff is appropriate (fix up your raid controller or whatever)

• Get a copy of the monmap and authentication key for the failed osd. On a machine that has access to the client.admin key (e.g., a monitor machine)

ceph mon getmap -o /tmp/monmap
ceph auth export osd.23 -o /tmp/keyring.osd.23

• Copy those files to the OSD node

scp /tmp/{monmap,keyring.osd.23} osdhost:/tmp

• Create a fresh btrfs file systems on the new btrfs devs and mount the osd data directory:

mkfs.btrfs /dev/foo
mount /dev/foo osd_data_dir

• Initialize the OSD

cosd --mkfs -i 23 --monmap /tmp/monmap

• Copy the keyring into the osd data directory

cp /tmp/keyring.osd.23 osd_data_dir/keyring

• Start the OSD

service ceph start osd.23