Shared Memory Tools

This topic is meant to document the tools that has been found interesting during the R&D on shared memory.

GDB for Forked Process

The good news is that GDB 6.8 can be used for debugging forked process. The bad news is that GDB 6.8 is not installed in SLC4. So, if you need to debug forked process install it yourself.

The version 6.8 has introduced four instructions related to the debugging of the forked process (see section 4.10):

• set follow-fork-mode [parent/child]: Choose the process being debugged after fork
• set detach-on-fork [on/off]: if on all the process will be under the control of GDB
• info forks: List of forked processid
• process [processid]: Debug another forked process

For non forked processes you can also use the attach/dettach GDB instructions (see section 4.7).

/proc/pid/smaps Parsing Scripts

The Linux kernel 2.6.16 adds support for smaps, per-mapping, including data on each mapping's RSS usage. This data lives in /proc//smaps. However, the format of the smaps file is hard to digest. In this section we introduce two scripts that has been found in the web in order to parse and digest the smaps files.

From the proc(5) man pages you can get the following information abous the smaps pseudo-files:

=/proc/[number]/smaps (since Linux 2.6.14) This file shows memory consumption for each of the process's mappings. For each of mappings there is a series of lines as follows:

08048000-080bc000 r-xp 00000000 03:02 13130 /bin/bash Size: 464 kB Rss: 424 kB Shared_Clean: 424 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB

The first of these lines shows the same information as is displayed for the mapping in /proc/[number]/maps. The remaining lines show the size of the mapping, the amount of the mapping that is currently resident in RAM, the number clean and dirty shared pages in the mapping, and the number clean and dirty private pages in the mapping.

This file is only present if the CONFIG_MMU kernel configuration option is enabled.=

smem.pl

The first script developed by B. Mauer using perl and requires Linux::Smaps library. In order to install the required Library just execute:

sudo perl -MCPAN -e shell
cpan> install Linux::Smaps

Once the library has been installed you can execute the script:

$ sudo perl smem.pl 19057 
VMSIZE:     101992 kb
RSS:         14012 kb total
             7288 kb shared
                4 kb private clean
             6720 kb private dirty
PRIVATE MAPPINGS
    vmsize   rss clean   rss dirty   file
   1564 kb        0 kb      864 kb   /opt/xdaq/lib/libxdata.so
    752 kb        4 kb      676 kb
   1136 kb        0 kb      656 kb   /opt/xdaq/lib/libtoolbox.so
    944 kb        0 kb      516 kb   /opt/xdaq/lib/libxdaq.so
    708 kb        0 kb      444 kb   /opt/xdaq/lib/libwseventing.so
  11100 kb        0 kb      372 kb
    …

SHARED MAPPINGS
    vmsize   rss clean   rss dirty   file
   3656 kb     1692 kb        0 kb   /opt/xdaq/lib/libxerces-c.so.27
   1564 kb      700 kb        0 kb   /opt/xdaq/lib/libxdata.so
   1136 kb      480 kb        0 kb   /opt/xdaq/lib/libtoolbox.so
    804 kb      436 kb        0 kb   /usr/lib/libstdc++.so.6.0.3
    944 kb      428 kb        0 kb   /opt/xdaq/lib/libxdaq.so
    …   

The output shows the virtual memory and RSS usage. The shared and private RSS tells us how much physical memory is being used for a given process. The private RSS is the physical memory being used exclusively by the process. In forked process is possible to measure how much physical memory has been copied by looking at the private RSS values (see Copy-on-write).

mem_usage.py

This second script was developed by A. Wingo using python. The script only needs python in order to be executed. The output of the script gives better aggregate information but without information per mapping. This script shows RSS usage classified per memory manager flag (VM_READ, VM_WRITE, VM_EXEC and VM_MAYSHARE).

$ sudo python mem_usage.py 19057
Mapped memory:
              Shared            Private
          Clean    Dirty    Clean    Dirty
   r-xp    7284        0        0     5204  -- Code
   rw-p       0        0        0      272  -- Data
   r--p       0        0        0       40  -- Read-only data
   r--s       4        0        0        0
  total    7288        0        0     5516
Anonymous memory:
              Shared            Private
          Clean    Dirty    Clean    Dirty
   r-xp       0        0        0        0
   rw-p       0        0        4     1204  -- Data (malloc, mmap)
   ---p       0        0        0        0
  total       0        0        4     1204
  ----------------------------------------
  total    7288        0        4     6720 

This topic is meant to document the tools that has been found interesting during the R&D on shared memory.

GDB for Forked Process

The good news is that GDB 6.8 can be used for debugging forked process. The bad news is that GDB 6.8 is not installed in SLC4. So, if you need to debug forked process install it yourself.

The version 6.8 has introduced four instructions related to the debugging of the forked process (see section 4.10):

• set follow-fork-mode [parent/child]: Choose the process being debugged after fork
• set detach-on-fork [on/off]: if on all the process will be under the control of GDB
• info forks: List of forked processid
• process [processid]: Debug another forked process

For non forked processes you can also use the attach/dettach GDB instructions (see section 4.7).

perfmon2 on SLC4

I was curious about using perfmon2 on the Trigger Supervisor and XDAQ (i.e. the online software infrastructure being used in the Level 1 Trigger of the CMS experiment). A. Nowak sent me the perfmon_DEluxe.py script that easies the use of perfmon2. After some trials and the invaluable help of Andrzej it seems that it is not possible to use perfmon2 on the "standard" SLC4, at least you need to have the version 2.6.25 of the Linux kernel and apply the perfmon2 patch to it. As the online software of the CMS is distributed in rpm binaries it seems to me that it will take too long to make the test. At least I will need to:

1. Install a Linux version with kernel 2.6.25
2. Apply the perfmon2 patch to the kernel
3. Checkout the online software from the CVS repository
4. Patch the online sources for the new kernel, compiler, etc.
5. Install libpfm and perfmon2
6. Use perfmon2_Deluxe.py

So, for the time being I will not do this... It is too much.

References

Shared Memory Prototyp for LHC, presented at the CERN Multi-core R&D Meeting.

GDB manual, version 6.8.

Ben Maurer, Memory Usage with smaps, March 2006.

Andy Wingo,Reducing the Footprint of Python Applications, November 2007.

smaps pseudo-file documentation, proc(5) - Linux man page, since linux kernel 2.6.14.

---

-- MarcMagransDeAbril - 18 Jul 2008