Preparing an LSI SAS 2008 based controller for ZFS

In my last post I wrote about how the Adaptec 5805 RAID controller was the wrong choice for our backup server. So I started some research, based on the following criteria:

Production quality driver (so no more crashes, hangs and other weird behavior)
Pass-through support (/dev/passX), including S.M.A.R.T.
JBOD support (disks should show up as devices /dev/daX)
Decent throughput
Existing success stories with ZFS
Reasonable price and availability

I had positive results using LSI, 3ware and Areca controllers in the past, a lot of our older machines use LSI MegaSAS 1078 based controllers to great success. Thanks to this blog post I learned that using a controller running the firmware of the LSI SAS 9211 IT yields excellent results and fulfills all of these criteria.

The following controllers all based on the LSI SAS 2008 and can all be cross flashed to the LSI SAS 9211-8i IT firmware (only checked for the eight port versions, since this is what was relevant to me).

LSI SAS 9211-8i
LSI SAS 9240-8i
LSI SAS 9260-8i
IBM M1015

Note

Cross flashing controllers might violate the terms and conditions of your vendor and affect your warranty, so if you decide to do this, it's at your own risk.

Fortunately I was able to get a hold of an LSI SAS 9211-8i that shipped with an older version of the IR ("Initiator Raid") firmware. For optimal ZFS performance for this setup I decided to flash it with the latest IT ("Initiator Target") version of the firmware.

Preparation

Most of the instructions below are based on the excellent blog post by Pieter Schaar found at http://www.servethehome.com/ibm-serveraid-m1015-part-4/. You can also find a link there to download all of the files used below (or you download them directly on LSI's web pages, which seems to be a better corporate strategy).

First I downloaded the FreeDOS USB Boot Image from Do Something Every Day.

I flashed the resulting image file to a USB thumb drive using dd, mounted it and copied the files required for the process. At the very least you'll need the following files:

2118it.bin
DOS4GW.EXE
MEGAREC.EXE
SAS2FLSH.EXE
mptsas2.rom

Once copied, you should be able to boot of the USB thumb drive to flash the controller firmware.

Lookup the SAS address of your controller. This is on a sticker on the back of the controller and should start with "500605b". Write it down, you'll need it in the process (in case you're using the Alternative Flashing Procedure described below this is actually not needed anymore).

Original Flashing Procedure

Note

This is the exact procedure for flashing an LSI SAS 9211-8i IR to LSI SAS 9211-8i IT. For more challenging cross flashing (e.g. from an IBM controller) please see Pieter Schaar's blog post. If this procedure fails or anything goes wrong in the process you're on your own, I won't help you and I can't held liable for any harm caused by this ('as-is'). Please make sure to follow the procedure exactly.

Note

In case you're cross flashing an LSI SAS 9211-8i IR the Alternative Flashing Procedure below might give you better results (and save you some time as well).

Warning

megarec supports a wider range of controllers than sas2flsh does. So please check for supported controllers using both commands (-listall) to make sure you're erasing/writing to the correct controller.

Boot FreeDOS from USB thumb drive
Check for available controllers (important):
```
megarec -listall
sas2flsh -listall
```
Clean controller flash memory (0 is the controller number, if your system contains multiple controller please change it accordingly):
```
megarec -cleanflash 0
```
Reboot FreeDOS from USB thumb drive
Flash firmware and OptionROM images (0 is the controller number):
```
sas2flsh -o -f 2118it.bin -b mptsas2.rom -c 0
```
Set controller SAS address (replace 500605bxxxxxxxxx with the actual SAS address of your controller, 0 is the controller number):
```
sas2flsh -o -sasadd 500605bxxxxxxxxx -c 0
```
Reboot into OS

Note that you could omit "-b mptsas2.rom" in step 5. In this case you can't use the controller BIOS from the command line and won't be able to boot of the array. Not what I want in my case, but maybe something you're interested in if you're using pxeboot or other boot media.

Alternative Flashing Procedure

This procedure only uses the sas2flsh utility. It also has the advantage of saving one reboot and not requiring you to write down and reprogram the SAS address.

Boot FreeDOS from USB thumb drive
Check for available controllers (important):
```
sas2flsh -listall
```
Clean controller flash memory (without removing WWN, 0 is the controller number):
```
sas2flsh -o -e 6 -c 0
```
Flash firmware and OptionROM images (0 is the controller number):
```
sas2flsh -o -f 2118it.bin -b mptsas2.rom -c 0
```
Reboot into OS

Note that in case of crossflashing the controller you might also need to erase the WWN of the controller, in this case use the Original Flashing Procedure described above, but replace step 3 (megarec) with sas2flsh -o -e 7.

Conclusion

After flashing the controller it works like a charm and is the perfect basis for a ZFS success story. In my next post I will detail how to install FreeBSD 9.0 from scratch on the disk array using mfsbsd.

Update 2012-12-10: Added *Alternative Flashing Procedure*, added parameters for selecting the correct controller and stressing the importance of checking for installed adapters using the -listall parameter.

FreeBSD, LSI, ZFS | /sysadmin | permalink

Michael's Daemonic Doodles

Thu, 12 Apr 2012