As it should be evident from my recent posts, I have been playing with iSCSI and ZFS a lot lately. And the only OS in my home that I work on is OS-X on my laptops. And so started my quest to find a perfect iSCSI solution (based on OpenSolaris) for my OSX environment.

Apple does not provide any iSCSI initiator with Leopard, it’s been rumored that with Snow Leopard Apple might release iSCSI initiator. But till then there are at least 3 iSCSI initiators that exists for OS X users.

I have been told that iSCSI is a very complex specification and the success with a particular (initiator / target) combination depends upon how comprehensive they are (and how lucky you are). And that’s exactly what I found.

As my first test I just downloaded the globalSAN iSCSI initiator and used it to connect to the “iscsitgt” based target from my OpenSolaris 2008.11 Workstation. I was able to connect to the target the read performance was great, write kinda sucked but this was the first time I was playing with iSCSI and i liked the idea of block level access over Wi-Fi

Then I started becoming inquisitive (greedy), and wanted to share multiple thin provisioned iSCSI devices to be used as TimeMachine, iTunes Library, iPhoto Library across my home network to be shared between multiple laptops. And for achieving that i started playing with advanced features like “Access Control List”, “CHAP Authentication” et. al.

So globalSAN iSCSI initiator was able to scan the target’s but as soon as I tried connecting to the target’s, the iSCSI initiator went into some sort of infinite loop and hung.

I came to know about this latest iSCSI implementation by Sun Microsystem which is written from scratch and is blazing fast. I had to try it and I did. Unfortunately with globalSAN iSCSI initiator I was unable to format the block devices.

Then I concentrated the next week getting access to different iSCSI initiators available for OSX and came across three:

• globalSAN (www.studionetworksolutions.com)
• Small-Tree (www.small-tree.com)
• Atto Xtend SAN (www.attotech.com)

I have not performed extensive tests (as of now) so I will just present my finding in form of a small table.

I would also like to state that compatibility table only represents the compatibility between these initiators with OpenSolaris based targets and does not talks about performance only compatibility.

As it’s visible from the table above Atto’s Xtend SAN was the only initiator that has so far been able to connect and access both the iSCSI subsystem’s with all variations (that I have tested so far). It’s also the one that costs the most ~200$ per seat, but hey it’s the only one which works so can’t complain.

If the requirements are not for using ACL’s or COMSTAR globalSAN has the best interface integration with the OSX preferences pane. On the other hand ATTO’s interface show’s age but also has many tunable options available.

OpenSolaris (2008.11) can serve iSCSI target’s easily. I have been using this capability for serving an iSCSI target which is being shared between my Apple Mac’s for TimeMachine Backups. I have been using this for a few months, and now I can backup my laptops using my home 802.11n Wi-Fi.

There are many advantages, like I don’t have to use an external USB disk which always required extra efforts and was not safe (redundant). But I have to be careful so that I don’t double mount the shared device. This is not exactly a problem as I can always create multiple iSCSI targets and bind them to the laptop’s exclusively, this way the host’s would only see and mount the devices for which they have access to.

I will eventually detail out all the steps I followed for configuring the OpenSolaris / iSCSI / ZFS / OSX / TimeMachine setup, in my subsequent blog entries. The basic purpose of this entry is to actually show some performance numbers of iSCSI shared volume to an Apple Mac client.

So first some details about the setup, The OpenSolaris host is configured as follows:

• Intel Core 2 Duo (3.00GHz) with 4GB RAM
• OpenSolaris 2008.11
• Zpool (RAID-Z) consisting of 5 x SATA-II HDD
• Broadcom GbE Adapter

On the client side I am using iSCSI driver for OSX from SNS (Studio Network Solution). All of these machines are connected using a cheap DLINK DIR-655 router. Since this a very small setup with limited number of disks and non enterprise hardware, the performance numbers might not be anything near what it could be with more robust hardware.

What I am trying to achieve is to somehow improve the performance as much as I could with the setup that I have by first identifying the bottlenecks and then optimizing the setup.

Baselining Performance

In order to identify the bottleneck which might exists (and most probably would) on the network subsystem, I started by getting the raw performance of my ZFS file system locally. The tool that I am using for all these test is “iozone ” and I am only limiting my tests to sequential read/write tests. A more exhaustive tests would take a lot more resources and would not be appropriate for the very small setup that I have.

On a five disk RAID-Z, I am getting a very decent performance, around 112MB/s for Writes and 145MB/s for Reads for 512Kb IO size. The IOZone command that used for the benchmark was:

iozone -ac -s8G -i0 -i1 -y4k -q1M

I was using an 8GB file for the benchmark to ensure that i am reading and writing enough data to bypass the caching effect which can spoil the results; 8GB is twice the size of the RAM, so no chances for read or write from cache.

iSCSI Performance

Then I tested a block device powered by the same zpool and exported as an iSCSI target using the default iSCSI subsystem (iscsitadm). The RAW volume was formatted as a HFS+ Journaled File system from the Apple Mac. Following graph shows the bandwidth that I got on my Apple Mac.

I am really disappointed by the write performance that I am getting out of my iSCSI setup. Read performance is actually not bad ~98MB/s on a cheap DIR-655 GbE switch is good. There are couple of different suggestions that I have received from various different places, for improving the performance:

1. Adding a separate dedicated fast block device for storing ZIL “Intent Log”. Typically this is supposed to be a Solid State Disk or something similar like i-RAM. But that would defeat the purpose of a cheap and reliable storage.
2. Trying out the COMSTAR based iSCSI implementation instead of the default “iscsitadm” subsystem. Apparently COMSTAR is much more optimized and offer’s better performance. Unfortunately it’s not provided with Open Solaris 2008.11 distribution and I’ll have to upgrade my installation to the development release of 2009.06.

ZFS uses the Intent Log for all the synchronous writes, so I assume that the performance of writes would improve dramatically if I can get a fast transactional block device (like an SSD) for separating the ZIL. But I am not going to be able to get an SSD disk for these tests as the cost of the SSD disk alone would constitute for more than 50% of the entire setup. I will be writing more about my optimization efforts and results, in the meantime I would greatly appreciate any suggestions or feedbacks.