SysReq Diagnosis

(or how to interpret the weird things that get dumped to the console if you do "sysreq-t")

SysReq-t - show tasks state

(information is from kernel/sched.c:show_task())

Example:

Feb 27 05:47:24 perl          S C8AA82B0  2044 26449  26442         26450 26448 (NOTLB)
Feb 27 05:47:24 e00a0d84 00000082 00000000 c8aa82b0 e84fd230 e84fd230 e00a0d58 00000220
Feb 27 05:47:24        00000000 00000000 c180e7c0 c180dde0 00000000 00000000 322476c0 0011dfe0
Feb 27 05:47:24        e84fd230 c8aa82b0 c8aa841c c011de7e c041e4a0 e00a0000 7fffffff e00a0de4
Feb 27 05:47:24 Call Trace:
Feb 27 05:47:24  [] find_busiest_group+0xdd/0x295
Feb 27 05:47:24  [] schedule_timeout+0xc4/0x159
[...]

Line-by-line:

perl

command name (task_t-\>comm)

               S

status (S leeping, R unning, D uninterruptable kernel, T raced, Z ombified, X ?, t, ? ?? (kernel confused as well))

                 C8AA82B0

program counter thread_saved_pc(task_t) or "runnning"

                                2044

free space on thread stack? difference between p-\>thread_info+1 and next non-null element above it.

                                         26449

PID (process ID)

                                                    26442

PPID (parent process ID)

"eldest child" PID, or empty (as here, 5x whitespace)

                                                                      26450

"younger sibling" PID or empty (7x whitespace)

                                                                               26448

"older sibling" PID or empty (5x whitespace)

                                                                                        (NOTLB)

L-TLB or (NOTLB) (means non-null p-\>mm )

For non-running tasks a stack trace is being printed afterwards (arch/i386/kernel/traps.c:show_stack()). This consists of the addresses on the stack (8 per line, 2nd line and up indented), and the call trace (arch/i386/kernel/traps.c:show_trace()).

SysReq-m - show global memory status

Feb 27 05:45:53 Mem-info:
Feb 27 05:45:53 DMA per-cpu:
Feb 27 05:45:53 cpu 0 hot: low 2, high 6, batch 1
Feb 27 05:45:54 cpu 0 cold: low 0, high 2, batch 1
Feb 27 05:45:54 cpu 1 hot: low 2, high 6, batch 1
Feb 27 05:45:54 cpu 1 cold: low 0, high 2, batch 1
Feb 27 05:45:54 Normal per-cpu:
Feb 27 05:45:54 cpu 0 hot: low 32, high 96, batch 16
Feb 27 05:45:54 cpu 0 cold: low 0, high 32, batch 16
Feb 27 05:45:54 cpu 1 hot: low 32, high 96, batch 16
Feb 27 05:45:54 cpu 1 cold: low 0, high 32, batch 16
Feb 27 05:45:54 HighMem per-cpu:
Feb 27 05:45:54 cpu 0 hot: low 14, high 42, batch 7
Feb 27 05:45:54 cpu 0 cold: low 0, high 14, batch 7
Feb 27 05:45:54 cpu 1 hot: low 14, high 42, batch 7
Feb 27 05:45:54 cpu 1 cold: low 0, high 14, batch 7
Feb 27 05:45:54
Feb 27 05:45:54 Free pages:       21056kB (252kB HighMem)
Feb 27 05:45:54 Active:112475 inactive:111041 dirty:0 writeback:4 unstable:0 free:5264 slab:20161 mapped:222778 pagetables:3271

Feb 27 05:45:54 DMA free:12508kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
Feb 27 05:45:54 protections[]: 0 0 0
Feb 27 05:45:54 Normal free:8296kB min:3728kB low:7456kB high:11184kB active:390008kB inactive:386664kB present:901120kB pages_scanned:8118 all_unreclaimable? no
Feb 27 05:45:54 protections[]: 0 0 0
Feb 27 05:45:55 HighMem free:252kB min:128kB low:256kB high:384kB active:59892kB inactive:57500kB present:130560kB pages_scanned:891 all_unreclaimable? no
Feb 27 05:45:55 protections[]: 0 0 0

So - -machine had run out of "high memory" (free < low), but might have still been trying to do something (not all pages marked as unreclaimable).

Feb 27 05:45:55 DMA: 5*4kB 3*8kB 3*16kB 2*32kB 3*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12508kB
Feb 27 05:45:55 Normal: 1000*4kB 63*8kB 1*16kB 2*32kB 4*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 8168kB
Feb 27 05:45:55 HighMem: 31*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252kB

these lines should help to diagnose memory fragmentation - they would show why order_X_ page allocation failures OK (i.e. "want continuous block of memory of size 2^X")

Feb 27 05:45:55 Swap cache: add 2594130, delete 2593633, find 46099707/46253623, race 0+110
Feb 27 05:45:55 0 bounce buffer pages
Feb 27 05:45:55 Free swap:            0kB

No swap left..

Feb 27 05:45:55 262016 pages of RAM
Feb 27 05:45:55 32624 pages of HIGHMEM
Feb 27 05:45:55 3506 reserved pages
Feb 27 05:45:55 15178 pages shared
Feb 27 05:45:55 499 pages swap cached