Cpu profiling

From Ceph wiki

The easiest way to profile the CPU consumption of Ceph is by using the oprofile system-wide profiler.

Installing oprofile

If you are using debian squeeze, you can install oprofile by doing the following:

# install oprofile stuff
apt-get install oprofile oprofile-gui

# in my case the kernel source package 2.6.32-5-amd64-dbg
apt-get install `uname -r`-dbg

Compiling Ceph for profiling

The easiest way to compile Ceph for profiling is to do the following:

# clean absolutely everything
make distclean

# build with CFLAGS="-fno=omit-frame-pointer -O2 -g"
# explanation:
# -fno-omit-frame-pointer       required to see callgraph output later
# -O2                           a reasonable level of optimization
# -g                            include symbols in binary
./do_autogen.sh -P

make -j 8

Check Ceph Configuration

Please turn off lockdep. It's also probably a good idea to turn most logging down to production levels.

Running oprofile

Wipe old data from the oprofile session directory. If you forget to do this, data from different runs will be mixed!

sudo opcontrol --reset

Things to do the first time you run oprofile after starting up. You will need to locate the vmlinux image corresponding to the kernel you are now running. (Not vmlinuz-- that does not contain the symbols you need.)

sudo opcontrol --init
sudo opcontrol --setup --vmlinux=/usr/lib/debug/boot/vmlinux-2.6.32-5-amd64 --separate=library --callgraph=6

Start profiling!

opcontrol --start

[ run tests ]

Stop profiling! This will also flush the session buffer.

opcontrol --stop

Reading oprofile results

To see the top symbols in cmon:

opreport -gal ./cmon | less

To see the top symbols in cmon with call graphs attached:

opreport -cal ./cmon | less

Re-running oprofile

Wipe old data from the oprofile session directory. If you forget to do this, data from different runs will be mixed!

sudo opcontrol --reset

You should wipe the data after analyzing it!

Limitations of oprofile

• Inline function samples are not attributed to the inline function but rather to the function the inline function was inserted into. To mitigate this, you can compile your code with -fno-inline. However, this may cause a performance hit.