I’ve recently started working extensively with iSCSI based SAN storage, having previously only had a background in Fibre Channel based storage. One unexpected benefit from using iSCSI based storage is the ease of capturing SCSI traffic for analysis. ISCSI, making use of the TCP/IP stack and Ethernet for the physical layer makes traffic capturing as trivial as installing Wireshark. On the other hand, capturing Fibre Channel traffic for analysis typically requires a Fiber tap and expensive pieces of hardware.
I have had experience with SCSI-3 Persistent Reservations in the past, but while recently troubleshooting a SCSI-3 PR issue, I decided to capture sample SCSI-3 Persistent Reservation traffic with Wireshark for reference analysis. I built a CentOS 5u3 virtual machine, installed and configured the iscsi_initiator_utils package, and then installed the sg3_utils package. sg3_utils consists of a number of extremely useful tools which allow you to directly manipulate devices via SCSI commands. The tool I wanted to use is called sg_persist and is used to send PROUT and PRIN commands (more on that in a moment).
Before we go any further, a little background is required…
The SCSI protocol standards, maintained by the t10 sub-committee of the International Committee for Information Technology Standards (INCITS), is split into a large number of specifications covering various aspects of SCSI. The over-arching standard is known as the SCSI Architecture Model, or SAM, now in its 5th generation (SAM-5). The SAM ties together all of the SCSI standards and provides the requirements which the myriad SCSI specifications and standards must meet.
Of primary concern to us now is the SPC, or SCSI Primary Commands standard, which defines SCSI commands common to all classes of SCSI devices. A given SCSI device will conform, in the least, to the SPC and a standard specific to that class of device. For example, a basic disk drive will conform to the SPC and the SBC (SCSI Block Commands) specifications.
Device reservations are handled in one of two ways–the older RESERVE/RELEASE method specified in the SCSI-2 specs and the newer persistent reservation method of SCSI-3. Reservations allow a SCSI device (typically a SAN-based storage array) to maintain a list of initiators which can and cannot issue commands to a particular device. Reservations, whether the older method or PR, are what allows more than one server to access to a shared set of storage without stepping on one another.
SCSI-3 Persistent Reservations offer some advantages over the older RESERVE/RELEASE method–primarily by allowing the reservation data to be preserved across server reboots, initiator failures, etc. The reservation will be held by the array for the LUN until it is released or preempted.
One important thing to keep in mind is that the function of persistent reservations are to prevent a node from writing to a disk when it does not hold the reservation. The reservation system will not prevent another node from preempting the existing reservation and then writing to the disk. It is the responsibility of the server-side application making use of the persistent reservations to ensure that cluster nodes act appropriately when dealing with reservations.
In part II we will look at the two SPC commands dealing with persistent reservations–PRIN (Persistent Reserve In) and PROUT (Persistent Reserve Out) and their associated service actions.