Background

(first, Google iSCSI)

It turns out that there are a number of projects that provide iSCSI target code now for Linux (initiators have been around for some time).

• http://unh-iscsi.sourceforge.net/ provides both, and is fairly old, however
• http://iscsitarget.sourceforge.net/ is the "state-of-the-art" target (server), referred to as ST here and
• http://www.open-iscsi.org/ seems to be the "state-of-the-art" initiator (client, if you like, SI).

All these require fairly recent kernels (2.6.10+).

First run

Target system #1

dual xeon 2.4Ghz, SLC3, linux vanilla 2.6.11.12 plus ST as iscsitarget-0.4.11.tar.gz (newer kernels not supported as of 10 Aug 2005) There's but the system disk inside so I'm using a 24GB disk image as an exported LUN. And a small 1GB partition too, to see if it works.

Initiator system #1

dual xeon 2.4Ghz, SLC3, linux vanilla 2.6.13-rc6 plus SI as open-iscsi-0.3rc7-383.tar.gz (older kernels are probably supported but wth)

Findings

Setting up the environment:

fdisk /dev/sdb
mkfs.xfs -l version=1 /dev/sdb1 -f
mount -t xfs -o logbsize=32768,logbufs=8 /dev/sdb1 /mnt
iozone -Mcew -i0 -i1 -i2 -s8g -r256k -f /mnt/iozone.dat

There needed to be some hacking in the client to prevent iscsiadm from segfaulting but otherwise it went OK, discovery of the network and devices available were OK as well as mounting and running. The target code behaves OK and is extremely stable. The client code fell over after writing and reading an 8GB file (iozone), resulting in a kernel panic.

While running, iozone wrote at about 28MB/s and read at the same rate (single stream) - which is OK for an IDE disk, esp. considering that it wasn't even a real disk. CPU (two cpus are 100%) was 55-65% idle on both sides when reading, according to vmstat. Writing was not as consistent; same 50% idle for the initiator, 10-20% idle for the target.

N.B. No tuning here (of vm nor of tcp) since I want stability before performance.

Second run

Target system #1

(see above)

Initiator system #2

dual xeon 2.4Ghz, SLC3, linux vanilla 2.6.12.4 plus SI as open-iscsi-0.3rc7-383.tar.gz (current latest stable kernel)

The initiator code needed to be patched with the supplied backport patch for 2.6.12.

(used the same initialization code as before)

now this is better, it completes:

        Machine = Linux it-adc-test2.cern.ch 2.6.12.4 #1 SMP Wed Aug 10 12:16:53 CEST   Include close in write timing
        Include close in write timing
        Include fsync in write timing
        Setting no_unlink
        File size set to 8388608 KB
        Record Size 256 KB
        Command line used: iozone -Mcew -i0 -i1 -i2 -s8g -r256k -f /mnt/iozone.dat
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd  record  stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read rewrite    read   fwrite frewrite   fread  freread
         8388608     256   27762   26020    28050    28433    6767   20918                                                          

iozone test complete.

Running with 8 threads, initiator CPU is in 92% iowait and the rest is system ... ...and then it crashes with the same kernel panic at the same place.

Third run: opteron client

Target system #1

(see above)

Initiator system #2

IBM eServer 326 (dual opteron 246), SLC3 x86_64, linux vanilla 2.6.12.4 x86_64 plus SI as open-iscsi-0.3rc7-383.tar.gz (current latest stable kernel)

This one took some serious beating (two days on GE) and then

a) got an I/O error b) NMI watchdog killed the machine.

TODO:

• resilience test (cable pulling, daemon killing, machine rebooting)
• test with some "real" disk subsystem that can saturate a GE
• test with 10GE
• test other architectures (amd64 for example)
• test N:1, 1:M, N:M setups

-- AndrasHorvath - 12 Aug 2005