Anil Gulecha’s Musings

[I've made a few small updates correcting typos and inserted additional information to turn off any existing shareiscsi=on zfs volumes]

Opensolaris has had the capability to share an iscsi target for sometime. This was done in the userland via the iscsi target daemon.

The COMSTAR project on opensolaris was started to build a fast scsi target on the platform. From the project page:

COMSTAR is a software framework that enables you to turn any OpenSolaris host into a SCSI target that can be accessed over the network by initiator hosts. COMSTAR breaks down the huge task of handling a SCSI target subsystem into independent functional modules. These modules are then glued together by the SCSI Target Mode Framework (STMF).

COMSTAR provides

• Extensive LUN Masking and mapping functions
• Multipathing across different transport protocols
• Multiple parallel transfers per SCSI command
• Scalable design
• Compatible with generic HBAs

With intergration of COMSTAR to ZFS, Nexenta Core Platform provides for very easy setup as an iSCSI target. These latest changes to ZFS are available both in the latest beta2 release of NCP, as well as the commercial NexentaStor v1.1.5.

How to setup a NCP2 box as an iSCSI target

First, shutdown older iSCSI target service, and start the new one

svcadm disable -s iscsitgt
svccfg delete -f iscsitgt

Also if you have any zfs volumes with shareiscsi=on, set them to shareiscsi=off. This is required to setup the new zfs integrated scsi correctly.

Enable iscsi/target and it’s dependency

svcadm enable -s stmf
svcadm enable -s iscsi/target

Install the switch by creating the file:

touch /etc/shareiscsi.conf

Now create a dataset (volume) which is to be shared via iSCSI

zfs create -V 500m tank/zvol1

Using -V creates a volume type dataset, and it reserves the given amount of space on the pool for it’s data. In our case we reserved 500 Mb of space for our volume zol1. To create an SCSI target using this, simply run.

zfs set shareiscsi=on tank/zvol1

And we’re now set :). To confirm that the target has been configured, check that output of

itadm list-target -v

Performance gain

This implementation of an iSCSI target provides 2-3 times the performance of the older userland implementation in opensolaris. The numbers stack up as follows:

Parameters:  4 workers x 8 outstanding IOs

COMSTAR iSCSI target

50000 IOPS, 512B, 97% CPU, 24MB/sec  <- Reads
40000 IOPS, 512B, 92% CPU, 17MB/sec  <-Writes

OLD UserLand iSCSI target

17000 IOPS, 512B, 98% CPU, 8MB/sec  <- Reads
15000 IOPS, 512B, 98% CPU, 7.5MB/sec  <-Writes

The tests were conducted with IOMeter, using a 100M zvol with zil disabled. Also these were done over a 1G network between the target and the initiator.

More information on COMSTAR can be found on it’s project page, and the changes can be found in the nexenta-on package in NCP2.