DEVOPS by Jan Gelin

Decided to share since I know it works:) We call these boxes “The BEASTS”. 80 TB/Unit of usable RAID6 storage with ZFS.

You can NFS mount, rsync straight to the device and much more. You probably should build two if one dies. We have four at Rubicon with a total of stunning 320 TB of storage, and I am not going to tell you the cost (thanks Pat).

You need to pick some of your favorite things, like the motherboard you like etc. This is in no way intended to be an enterprise datacenter solution. Use it for what you think it is worth.

So let’s say you have limited budget and you need to get some storage cheap. Here is one way to do it that works:

What do you need:

• Chenbro Chassis
• Favorite server motherboard (multiple pci slots)
• 3ware controller(s), we used 2 x 3ware 9650SE-24M8 Storage controller
• Lots of SATA drives (pref WD Blacks or good quality 2 TB drives or higher drives). we used 2 TB Greens (too high failure rate for my taste).
• Time or a contractor to put it together
• Normal accessory things for putting it together, CPU etc…. You don’t need more than 2-4 gigs of RAM

First kudos to Oliver Cochard-Labbe and team for putting Freenas together. I have used it for years. And yes if you don’t like BSD you may stop reading now:) Freenas is a simple multi protocol storage facility. It is also (for good or for worse) quite a hackable system.

So I am going to assume you figure out how to put this together. On the software side here are some tips:

• Obviously get a cd (I use one of them external CD USB units)
• Install Freenas 8 FreeNAS-8.0.4-RELEASE-p2-x64.iso if you want time machine to work with time machine and OS X
• install 3ware cli 3DM2_CLI-FreeBSD-10.2.1_9.5.4
• if your organization looks like mine get a 4 port intel gigabit card and create a lag group.

I am sure there is a way to get the lag group to work in the GUI. However me and GUI’s tend to not agree most of the time so here is what I did.

1. Shell to the box as root

2. Mount the read only freenas filesystem as r/w with  mount -o rw /dev/ufs/FreeNASs1a /

3. Go ahead and make your changes, it’s just BSD:) here is how to make a lag group:

Edit the file in /conf/base/etc/rc.conf otherwise it will not stick.

ifconfig_em0=”up”
ifconfig_em1=”up”
ifconfig_igb0=”up”
ifconfig_igb1=”up”
ifconfig_igb2=”up”
ifconfig_igb3=”up”
cloned_interfaces=”lagg0″
ifconfig_lagg0=”laggproto lacp laggport em0 laggport em1 laggport igb0 laggport igb1 laggport igb2 laggport igb3 10.10.12.20″
defaultrouter=”10.10.10.1″

And while you are at it install the 3ware CLI, I just used the install shell script and never used the GUI. I found some helpful tw_cli commands on this page:  http://www.cyberciti.biz/files/tw_cli.8.html. There is probably a better page somewhere but this one was enough for me.

You can also be creative and get cron reports from the tw_cli emailed to you daily.

Save and reboot and you should now have your interfaces up and running if you configured the switch right.

Configure your favorite raid type with tw_cli and wait zzzzz for it to finish.

That should do it. Have fun in the FreeNAS GUI setting everything up.

This entry was posted on Tuesday, May 29th, 2012 at 11:09 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.