Apple does not provide any iSCSI initiator with Leopard, it’s been rumored that with Snow Leopard Apple might release iSCSI initiator. But till then there are at least 3 iSCSI initiators that exists for OS X users.

I have been told that iSCSI is a very complex specification and the success with a particular (initiator / target) combination depends upon how comprehensive they are (and how lucky you are). And that’s exactly what I found.

As my first test I just downloaded the globalSAN iSCSI initiator and used it to connect to the “iscsitgt” based target from my OpenSolaris 2008.11 Workstation. I was able to connect to the target the read performance was great, write kinda sucked but this was the first time I was playing with iSCSI and i liked the idea of block level access over Wi-Fi

Then I started becoming inquisitive (greedy), and wanted to share multiple thin provisioned iSCSI devices to be used as TimeMachine, iTunes Library, iPhoto Library across my home network to be shared between multiple laptops. And for achieving that i started playing with advanced features like “Access Control List”, “CHAP Authentication” et. al.

So globalSAN iSCSI initiator was able to scan the target’s but as soon as I tried connecting to the target’s, the iSCSI initiator went into some sort of infinite loop and hung.

I came to know about this latest iSCSI implementation by Sun Microsystem which is written from scratch and is blazing fast. I had to try it and I did. Unfortunately with globalSAN iSCSI initiator I was unable to format the block devices.

Then I concentrated the next week getting access to different iSCSI initiators available for OSX and came across three:

• globalSAN (www.studionetworksolutions.com)
• Small-Tree (www.small-tree.com)
• Atto Xtend SAN (www.attotech.com)

I have not performed extensive tests (as of now) so I will just present my finding in form of a small table.

I would also like to state that compatibility table only represents the compatibility between these initiators with OpenSolaris based targets and does not talks about performance only compatibility.

As it’s visible from the table above Atto’s Xtend SAN was the only initiator that has so far been able to connect and access both the iSCSI subsystem’s with all variations (that I have tested so far). It’s also the one that costs the most ~200$ per seat, but hey it’s the only one which works so can’t complain.

If the requirements are not for using ACL’s or COMSTAR globalSAN has the best interface integration with the OSX preferences pane. On the other hand ATTO’s interface show’s age but also has many tunable options available.

These multiple zones provides isolation, security, between each other and yet are very lightweight. Unlike a traditional virtual machines which requires a complete instance of OS running for each instance, the instances of Zones uses the same running kernel efficiently.

This is one of the major motivation behind this exercise, because the resources required for supporting individual instances are equivalent to the primary application that would be running on these zones. But since there is only one kernel instance interpreting all the system calls, it’s not possible to run instances of other operating systems like Windows XP, Vista, OSX et. al.

However there is an interesting project called “BrandZ” which extends Zones, which allows the use of non-native binaries (os). As of the writing (04/09) Linux Zones works pretty well (I don’t think non Unix OS would be supported anytime soon).

I have been using OpenSolaris based workstation as a feature rich, flexible and stable home NAS for serving iSCSI, SMB, NFS et. al. But there are many applications which are either not supported on Solaris but are under Linux. One such application is a DLNA compliant UPnP server.

In this article I will detail all the steps required to:

• Setup a Fedora-10 based Branded Zone on OpenSolaris 2008.11, And
• Configure “Media-Tomb” for streaming content to a PS3 (DLNA Client)

Setting up a zone is fairly straightforward. The command “zonecfg” is used to create zone by specifying the characteristics of the zone. This command creates an xml file under “/etc/zones” which contains all the details that were entered using “zonecfg”.

Creating The Branded Zone

Step 1: Starting a new configuration.

# zonecfg -z Fedora-10
Fedora-10: No such zone configured
Use 'create' to begin configuring a new zone.

Step 2: Using a template to inherit some basic information. SUNWlx refers to Linux based Branded Zones.

zonecfg:Fedora-10> create -t SUNWlx

Step 3: Specifying the location where the files would be installed. The path need not exist as it would be automatically created in the installation phase with the proper permissions.

zonecfg:Fedora-10> set zonepath=/tank/Zones/Fedora-10

Step 4: Setting the autoboot parameter, if set to true the zone would be started automatically whenever the machine is rebooted.

zonecfg:Fedora-10> set autoboot=true

Step 5: Setting the network parameters. And associating the network IP address with the physical Ethernet NIC. In the example below the physical NIC interface “bge0” would be used for networking. The list of available NIC on the system can be probed by using the command “ifconfig”.

zonecfg:Fedora-10> add net
zonecfg:Fedora-10:net> set address=192.168.0.29/24
zonecfg:Fedora-10:net> set physical=bge0
zonecfg:Fedora-10:net> end

Step 6: Adding a local file system to the zone and providing a mount point within the zone. In the example below the local file system “/tank/Content/Media/Videos” would be mounted within the zone at “/local_zfs_volume”

zonecfg:Fedora-10> add fs
zonecfg:Fedora-10:fs> set dir=/local_zfs_volume
zonecfg:Fedora-10:fs> set special=/tank/Content/Media/Videos
zonecfg:Fedora-10:fs> set type=lofs
zonecfg:Fedora-10:fs> end

Step 7: Setting Linux 2.6 kernel parameter. By default OpenSolaris 2008.11 Zones only support 2.4 kernels. For 2.6 there is a template which could be downloaded inside /etc/zones or the following parameters could be changed from the “zonecfg” interface. I chose the latter:

zonecfg:Fedora-10> add attr
zonecfg:Fedora-10:attr> set name=kernel-version
zonecfg:Fedora-10:attr> set type=string
zonecfg:Fedora-10:attr> set value=2.6
zonecfg:Fedora-10:attr> end

Step 8: Setting the upper limit on the CPU utilization by the zone. The example below would cap the maximum CPU utilization at 50% of one CPU on the physical machine. For example my workstation is a dual core 3.0GHz machine, so effectively in worst case scenario the zone would not be allowed to use more than 1.5GHz of processing power.

zonecfg:Fedora-10> add capped-cpu
zonecfg:Fedora-10:capped-cpu> set ncpus=0.5
zonecfg:Fedora-10:capped-cpu> end

Step 9: Capping the maximum memory. This is also neat since there is no actual kernel running for this virtual machines the footprint is actually the size of the applications running inside the zone. So the memory required for supporting a Zone is pretty less.

zonecfg:Fedora-10> add capped-memory
zonecfg:Fedora-10:capped-memory> set physical=512m
zonecfg:Fedora-10:capped-memory> set swap=512m
zonecfg:Fedora-10:capped-memory> end

Step 10: Finishing the configuration, verifying and saving the new changes.

zonecfg:Fedora-10> commit
zonecfg:Fedora-10> info

zonename: Fedora-10
zonepath: /tank/Zones/Fedora-10
brand: lx
autoboot: true
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: shared
hostid:
fs:
        dir: /local_zfs_volume
        special: /tank/Content/Media/Videos
        raw not specified
        type: lofs
        options: []
net:
        address: 192.168.0.29/24
        physical: bge0
        defrouter: 192.168.0.1
capped-cpu:
        [ncpus: 0.50]
capped-memory:
        physical: 512M
        [swap: 512M]
attr:
        name: kernel-version
        type: string
        value: 2.6
rctl:
        name: zone.cpu-cap
        value: (priv=privileged,limit=50,action=deny)
rctl:
        name: zone.max-swap
        value: (priv=privileged,limit=536870912,action=deny)

Step 11: Exiting the interface.

zonecfg:Fedora-10> exit

The flexibility of specifying the amount of resource allocated to a zone comes very handy in providing Quality of Service to different zones. And also saves the other zones in case, if there is a rouge process trying to consume all of the resources. And since there is no or very little overhead in creating/managing a zone itself it might be a good idea to run several application like Web Server, MySQL, Media Server, et. al. in their own separate zones With the help of capped-cpu and memory constructs one zone could be heavily thrashing it’s share of resource without even affecting the process on the other zones.

Now that we have the Zone defined it’s time to perform the actual installation inside this Branded Zone.

Installing BrandZ: Linux (Fedora-10)

Linux OS can be installed on a Branded Zone using two methods:

• Using the installer CD/DVD/ISO
• Using a “tarball” of an installed system

I chose the second option as there are known issues with the first option, and only selected few distributions are known to work. So I created a virtual machine on my “vmware” workstation, installed Fedora-10_i386, and created a tarball using the following command:

# cd /
# tar -jcf fedora-10.tar.bz2 --exclude fedora-10.tat.bz2 \
      --exclude dev --exclude proc --exclude sys --exclude boot *

Remember to unmount any NFS or CIFS share, if you are planning to create a tarball of an existing installation. I have also made the tarball available for download from here [ http://www.abisen.com/pub/fedora-10-brandZ/fedora10.tar.bz2 ].

You’ll also notice that we are excluding all of the device, kernel and sys directories, validating that there won’t be any separate instance of an OS running, only the applications and the system call’s would be interpreted and serviced by the solaris kernel.

These are the steps that I took for installing the linux tarball on my Branded Zone.

Step 1: Installing the OS from the tarball inside the newly created Zone.

# zoneadm -z Fedora-10 install -d /tmp/fedora10.tar.bz2
A ZFS file system has been created for this zone.
Installing zone 'Fedora-10' at root directory '/tank/Zones/Fedora-10'
from archive '/tmp/fedora10.tar.bz2'

This process may take several minutes.

Setting up the initial lx brand environment.
System configuration modifications complete.
Setting up the initial lx brand environment.
System configuration modifications complete.

Installation of zone 'Fedora-10' completed successfully.

Details saved to log file:
    "/tank/Zones/Fedora-10/root/var/log/Fedora-10.install.1075.log"

Note: The above command would also create a new ZFS file system with appropriate permissions.

Step 2: Verify the presence of the newly created zone and all the other installed zones and their current state

# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   2 Fedora-9         running    /tank/Zones/Fedora-9           lx       shared
   3 Fedora-10        installed  /tank/Zones/Fedora-10          lx       shared
   - centos52         installed  /tank/Zones/centos52           lx       shared

Step 3: Booting the newly installed zone. One handy tip is to open a new terminal instance and connect to the console of the installed zone “Fedora-10” using the command “zlogin -C Fedora-10”. This would act as a monitor for all the activities taking place inside the zone while it’s booting up. Following screenshot shows that I mean:

# zoneadm -z Fedora-10 boot

Step 4: Logging into the zone. Also notice that the console of my installation is clean and there are no error messages. In reality the first time you boot there would be many error messages. Which we will fix in the later steps. The following command would get us logged into a terminal session of the zone.

# zlogin Fedora-10

Step 5: In this step we will clean the installation for error free bootup. Also I noticed that for some reason the startup scripts are not getting executed in my installation. Which is fine as one of the positive side effect is that unnecessary services are not started by default :).

So upon further investigation it was found that the culprit for all the error message during bootup were also the init scripts (actually one script in particular: “/etc/rc.d/rc.sysinit”). I won’t go in detail but there were a lot of reference to /dev/urandom, loadable modules for device-mapper et. al. which are not eactly required in a Branded Zone (remember no kernel only applications). So I cleaned my /etc/rc.d/rc.sysinit and at the end I am also using this script to call all the other services that I would like to start during boot sequence.

My modified scrip can be downloaded from the following location [ http://www.abisen.com/pub/rc.sysinit ].

Step 6: Configuring the yum with the rpmfusion [ www.rpmfusion.org ] repository and updating the system.

# rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm \ 
           http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm

# yum update

# yum install media-tomb ffmpeg mplayer vlc-nox

Step 7: Modifying the startup script for media-tomb. The ethernet interface is a fake interface which is basically tied up to the global zone’s NIC. And there is no kernel/os running so some interfaces don’t respond the way they are supposed to in a real environment. And this prevents the media tomb to probe the ethernet nice (in my case bge0). So I modified the file “/etc/mediatomb.conf”, kept everything same but added the following line:

IFACE_IP="192.168.0.29"

And modified the startup by modifying the following lines:

# [ ${NETWORKING} = "no" ] && exit 0
daemon $NICELEVEL mediatomb $MEDIATOMB -i $IFACE_IP $MT_OPTIONS

My version of media tomb startup script can be downloaded from [ http://www.abisen.com/pub/mediatomb ]

Step 8: This sets everything in order for running media tomb. The interface can be accessed using http://IP(of the zone):50500.

In the next part I will talk about customization of media-tomb such that it could be used for real-time transcoding of the format’s that a UPnP player does not understands natively.

Before actually investing in some kind of Solid State Device, it would be nice to actually quantify how much improvement in write performance is possible. That is if I separate the ZIL to a dedicated, fast, transactional device.

A quick and clean test would be to run a similar IOzone benchmark on the iSCSI device and turning off the Intent Log completely from the ZFS file system. This should give me the extreme best case as it would take away the entire overhead of ZIL.

Note: Keeping the ZIL disabled is not at all recommended in any production system as it could severely damage the file system. And the file system is prone to corruption in the event of any crash.

In order to turn off the ZIL dynamically the following command can be executed. I personally feel it’s a nice idea to restart the machine after making such change, especially if I am benchmarking. But I am too lazy so I followed the following steps:

Step I: Disable the ZFS Intent Log

# echo zil_disable/W0t1 | mdb -kw

Step II: Unmount the ZFS file system in the concerned pool

# zfs unmount /tank

Step III: Export the zpool

# zpool export tank

Step IV: Import the zpool back, which would also mount all the file systems

# zpool import tank

Step V: Execute the benchmarking applications

# iozone -ac -s8G -i0 -i1 -y4k -q1M

The results were very interesting, as thought there was a big improvement in writes. But I didn’t expect the performance to increase by 26x on average. The following graph shows the numbers that I got after turning off the ZIL.

Also notice that the Read performance also increased big time, ~110MB/s which is an improvement of around 12%. But it’s hard to say that the bottleneck is the back-end (solaris/zfs/iscsi) or the network itself since the performance is near-wire speed. The results from the perevious tests are mentioned in this entry

Summary

Summarizing all the pervious tests the following graphs shows the improvement I achieved with the two changes:

1. Moving the file system block size from 8k to 128k
2. Running the tests with ZFS intent log disabled (case for SSD)

There is still a lot to work on getting a perfect working system. The next step would be to insert a USB Flash Disk and see where does the performance lands (esp. on writes)

After much discussions, I was inclining towards the theory that the culprit is the high latency and low IO performance of my 5 x SATA-II RAID-Z, which is shared between the data, metadata and the ZFS Intent Log. ZFS Intent Log is the logging system for ZFS file system, responsible for maintaining the file system consistent in the event of crashes etc. ZIL is also used by the file system for providing synchronous writes, for POSIX compliancy. For a good performing system it is recommended that the device that hold’s the log is very fast (in terms of latency for handling many small random writes). Neelakanth’s blog entry has some very good info on ZIL.

Playing with ZFS Intent Log (ZIL)

So as next step, I will try to identify and more importantly quantify the kind of IO operations that are taking place during writes.

Typically Solid State Disks (SSD) with high transactional performance are recommended to be used for the ZFS Intent Logs. The only problem is that the cost of buying an expensive SSD disk would be prohibitively expensive esp. for my home brew NAS.

I still want to not only see how much a SSD device can really help (which is an easy part), but also to optimize the performance of my iSCSI targets for write performance.

In order to have fast Apple Time Machine backups I would need faster iSCSI devices. This would get better once Apple moves to ZFS as their native file system and the backup would truly mean sending the incremental snapshots using “zfs send”.

So I started by monitoring the activity on the logging subsystem using “dtrace”. Dtrace is one of the most exposing tool for realtime analysis, that I have come across.

So I planned a small experiment, where I would write (and re-write) a 128MB file with 256k block size using IOzone on the iSCSI device. And during this time would monitor all the activities on the ZIL. This requires setting two probes on the Solaris workstation before starting the IOzone test.

Setting Probes on OpenSolaris

Probe 1: To collect information on all the ZIL activity.

# dtrace -n zil\*:entry'{@[probefunc]=count();}' -n zio_free_blk:entry'{@[probefunc]=count();}' 
dtrace: description 'zil*:entry' matched 44 probes
dtrace: description 'zio_free_blk:entry' matched 1 probe
^C

  zil_itx_clean                                                     4
  zil_clean                                                        16
  zil_header_in_syncing_context                                    82
  zil_sync                                                         82
  zil_vdev_compare                                               1152
  zil_flush_vdevs                                               31617
  zil_commit_writer                                             32457
  zil_lwb_write_done                                            32764
  zil_lwb_write_start                                           32764
  zio_free_blk                                                  32764
  zil_add_block                                                 32769
  zil_commit                                                    32933
  zil_lwb_commit                                                33106
  zil_itx_assign                                                33585
  zil_itx_create                                                33585
  zil_lwb_write_init                                            34253

Probe 2: To monitor block size of the log writes.

# dtrace -n ::zil_lwb_commit:entry'{@[1] = quantize((lwb_t*)args[2]->lwb_sz);}'
dtrace: description '::zil_lwb_commit:entry' matched 1 probe
^C

        1
           value  ------------- Distribution ------------- count
            2048 |                                         0
            4096 |                                         58
            8192 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   31202
           16384 |@@                                       1643
           32768 |                                         203
           65536 |                                         0        

Note: Notice the ^C character right below the probe; Once the IOzone run is finished the probe must be interrupted for getting the results.

Executing the IOzone test on the client: The following test would basically write 256MB of data to the iSCSI device.

$ ~/Downloads/iozone -r256k -s128m -i0 -w 
     Iozone: Performance Test of File I/O
                Version $Revision: 3.321 $
                Compiled for 32 bit mode.
                Build: macosx
         .
         .
         .
        Record Size 256 KB
        File size set to 131072 KB
        Setting no_unlink
        Command line used: /Users/abisen/Downloads/iozone -r256k -s128m -i0 -w
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.

              KB  reclen   write rewrite
          131072     256     872     869   

iozone test complete.

I still need to completely understand the output from the two ZFS probes, but it is clearly visible that for writing 256MB of data it requires a lot of activity on the logging subsystem. Now that we have quantified the number of commits and block size of commits on the ZIL. It would be very helpful to somehow see how our storage subsystem is performing from IOPS and latency perspective.

I performed a quick test again, using the same IOzone benchmarking tool and iostat. But this time I executed the benchmark locally on the Solaris workstation. Alternatively I also created a small ZFS file system on an old SanDisk Cruzr (512MB) USB Flash Disk. And ran the same test on this file system as well. Just to get an idea on how a USB Flash would fare against my 5 x SATA-II RAID-Z volume.

The video below shows what I noticed, Not only the service time (latency) on the USB Flash Disk was 4-5 times better than 5 x SATA HDD’s, even the IOPS that the old little USB flash could sustain were far greater than what 5 SATA disks could achieve.

So a USB Flash Disk might have some potential for helping in improving the write performance of a home brew NAS. I am hesitant on trying this out right away because, it’s easy to add a new device to a ZFS file system, but I have not been able to find any information on deleting/removing a device from a Zpool.

Next steps:

• Testing the performance by disabling the ZIL completely. This would give an idea on how fast can these iSCSI device get if ZIL is taken out of the picture. A ZFS file system should never be used with ZIL disabled, but it’s a neat feature for testing if separating ZIL would give any performance benefits.
• Testing the performance of the ZFS file system by using a USB based Flash Disk.