Memory Profiling

From Ceph wiki

It's possible for Ceph itself to generate heap profiles if you're using tcmalloc. For now this is limited to the MDS and the OSD since those are the only components that actually make use of tcmalloc. For more information about the profiling, see Google Heap Profiler.

To enable continuous profiling, you can start up the daemon with the environment variable CEPH_HEAP_PROFILER_INIT defined.

You can also turn the profiler on and off via "tell id heap [start|stop]_profiler", e.g:

ceph osd tell 0 heap start_profiler

To generate a single profile, you can send the message heapdump to the appropriate daemon. e.g:

ceph osd tell 0 heap dump

This will only work if the profiler is currently running.

You can configure both the "allocation interval" and the "highwater interval" (both discussed in Google's documentation) via the HEAP_PROFILE_ALLOCATION_INTERVAL and HEAP_PROFILE_INUSE_INTERVAL on startup.

There are other options as well:

ceph osd tell 0 heap stop_profiler
ceph osd tell 0 heap release
ceph osd tell 0 heap stats

heap release will release any memory which tcmalloc has allocated but which is not being used by the ceph daemon itself. heap stats will print out the same statistics that are available with heap dump but does not require the profiler to be running and does not dump the heap allocation information to a file.