Notes To Self: Attempting CIFS Performance Tuning

Problem: Abysmal SMB peformance when using W7 as intermediate between EON and FreeNAS: 19.6MB/s.
After Tuning: 25.8MB/s.
(After Turning Direct W7 -> FreeNAS = 42.1MB/s)

Wireshark Capture Filter:
host x.x.x.x and (port 139 or port 445)

Windows Registry:

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanServer\Parameters\TreatHostAsStableStorage = 1
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanServer\Parameters\SizReqBuf = 16384
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanWorkstation\Parameters\DisableLargeMtu = 0
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanWorkstation\Parameters\DisableBandwidthThrottling = 1
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanWorkstation\Parameters\FileInfoCacheEntriesMax = 65536
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanWorkstation\Parameters\DirectoryCacheEntriesMax = 4096
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\EnableWsd = 0
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Executive\AdditionalCriticalWorkerThreads = 2

FreeNAS Server:

• Enable EA Support
• Enable Send files with sendfile(2)
• Enable AIO
• Minimum AIO Read/Write Size: 16,384
• Auxiliary Parameters: max protocol = smb2

FreeNAS Wishy-Washy-Me:

mount -uw /

vi /conf/base/etc/ttys

'on' to 'off' for all but first two. (Logic: board supports max of 8gb; need the extra memory for I/O.)

mount -ur /

To potentially dealing with:
re0: watchdog timeout
re0: link state changed to DOWN
re0: link state changed to UP
Be conservative with:
-rxcsum -txcsum -tso -lro -wol -vlanmtu -vlanhwtag -vlanhwfilter -vlanhwtso

Reference:
SMB Tuning for XenApp and File Servers on Windows Server 2008  - Wireshark screenshots, and settings applicable to more than just XenApp
SMB Block Size Negotiation When Copying Files
SMB/CIFS Performance Over WAN Links
Poor Samba Performance?
Performance Tuning Guidelines for Windows Server 2008 R2 (Has docx with descriptions, page 59 for smb2 start of relevance)
Optimizing Operating System Performance
AdditionalCriticalWorkerThreads Forum Thread
Improve FreeNAS SMB Performance
FreeNAS CIFS Page
ifconfig manpage
Potential Headache:
Bug 8726 - SMB2 protocol crashes smbd inconsistently without any reasons
FreeNAS irritation:
Realtek 8111F fails at high traffic 're0 Watchdog Timeout Error'

Notes To Self: FreeNAS / ZFS Settings / BSD Commands

Commands and output...

 /# camcontrol devlist
<ATA SAMSUNG HD502HJ 00E4>         at scbus0 target 0 lun 0 (pass0,da0)
<ATA SAMSUNG HD502HJ 00E4>         at scbus0 target 1 lun 0 (pass1,da1)
<ATA SAMSUNG HD502HJ 00E4>         at scbus0 target 2 lun 0 (pass2,da2)
<KINGSTON SH103S3120G 502ABBF0>    at scbus1 target 0 lun 0 (pass3,ada0)
<ST3000DM001-1CH166 CC43>          at scbus2 target 0 lun 0 (pass4,ada1)
<ST3000DM001-1CH166 CC24>          at scbus3 target 0 lun 0 (pass5,ada2)
<ST3000DM001-9YN166 CC4B>          at scbus4 target 0 lun 0 (pass6,ada3)
<ST3000DM001-9YN166 CC4B>          at scbus5 target 0 lun 0 (pass7,ada4)
<ST3000DM001-1CH166 CC43>          at scbus6 target 0 lun 0 (pass8,ada5)
<SanDisk U3 Cruzer Micro 3.27>     at scbus7 target 0 lun 0 (pass9,da3)

[root@afarstor /mnt/durance]# for i in $(sysctl -n kern.disks); do smartctl -a /dev/$i >>`date +%Y.%m.%d`-afarstor_smart_dump.txt; done;

Example output:

=== START OF INFORMATION SECTION ===

Model Family:     Seagate Barracuda (SATA 3Gb/s, 4K Sectors)

Device Model:     ST3000DM001-1CH166

Device Id: 5 000c50 04e0962ff

Firmware Version: CC43

User Capacity:    3,000,592,982,016 bytes [3.00 TB]

Sector Sizes:     512 bytes logical, 4096 bytes physical

Device is:        In smartctl database [for details use: -P show]

ATA Version is:   8

ATA Standard is:  ATA-8-ACS revision 4

SMART support is: Available - device has SMART capability.

SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 10

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x000f   100   100   006    Pre-fail  Always       -       121624

  3 Spin_Up_Time            0x0003   091   091   000    Pre-fail  Always       -       0

  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       19

  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       1335

  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       25

 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0

 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       6

183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0

184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0

187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0

188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0

189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0

190 Airflow_Temperature_Cel 0x0022   076   068   045    Old_age   Always       -       24 (Min/Max 23/26)

191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0

192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       5

193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       19

194 Temperature_Celsius     0x0022   024   040   000    Old_age   Always       -       24 (0 21 0 0 0)

197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       40956808134661

241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       21788

242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       60760

 /# zpool status

  pool: chelestra

 state: ONLINE

  scan: none requested

config:

        NAME                                            STATE     READ WRITE CKSUM

        chelestra                                       ONLINE       0     0     0

          raidz2-0                                      ONLINE       0     0     0

            gptid/981738af-cd71-11e2-949a-3085a93e10c7  ONLINE       0     0     0

            gptid/98dab7a8-cd71-11e2-949a-3085a93e10c7  ONLINE       0     0     0

            gptid/9a179f3c-cd71-11e2-949a-3085a93e10c7  ONLINE       0     0     0

            gptid/9ae54d94-cd71-11e2-949a-3085a93e10c7  ONLINE       0     0     0

            gptid/9ba55256-cd71-11e2-949a-3085a93e10c7  ONLINE       0     0     0

        cache

          gptid/9c3aab11-cd71-11e2-949a-3085a93e10c7    ONLINE       0     0     0

errors: No known data errors

  pool: durance

 state: ONLINE

  scan: none requested

config:

        NAME                                            STATE     READ WRITE CKSUM

        durance                                         ONLINE       0     0     0

          mirror-0                                      ONLINE       0     0     0

            gptid/e1f271a9-cc1b-11e2-a9a2-3085a93e10c7  ONLINE       0     0     0

            gptid/e2a6eb5f-cc1b-11e2-a9a2-3085a93e10c7  ONLINE       0     0     0

            gptid/e3541d6e-cc1b-11e2-a9a2-3085a93e10c7  ONLINE       0     0     0

errors: No known data errors

 /# df -h

Filesystem             Size    Used   Avail Capacity  Mounted on

/dev/ufs/FreeNASs2a    926M    381M    470M    45%    /

devfs                  1.0k    1.0k      0B   100%    /dev

/dev/md0               4.6M    3.2M    973k    77%    /etc

/dev/md1               823k    2.5k    755k     0%    /mnt

/dev/md2               149M     15M    121M    11%    /var

/dev/ufs/FreeNASs4      19M    973k     17M     5%    /data

durance                452G     88G    364G    20%    /mnt/durance

chelestra                8T    255k      8T     0%    /mnt/chelestra

Useful Ref:

Seagate's Seek Error Rate, Raw Read Error Rate, and Hardware ECC Recovered SMART attributes

FreeNAS Docs
[ HOW TO ] Manage Automatic Daily Email Reports  - Deal with retarded daily emails. (FreeNAS devs took a lesson from Microsoft; no configurable option, and no trace of logic; instead of sending on an error or condition requiring attention, FreeNAS insists on telling you everything's okay... )

ZFS On Linux - Notes to Self / (aka forget mdadm, this is easier!)

Set up repositories:

deb http://ppa.launchpad.net/dajhorn/zfs/ubuntu natty main
deb-src http://ppa.launchpad.net/dajhorn/zfs/ubuntu natty main
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1EE8660B

Fix "Samba sharing is not supported.":

apt-get install nfs-kernel-server

Create pool:

# zpool create chasm raidz /dev/sda /dev/sdb /dev/sdc /dev/sdd
# zpool list

NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
chasm  1.81T   303K  1.81T     0%  1.00x  ONLINE  -

Create filesystem and set options:

# zfs create chasm/data
# zfs set atime=off chasm
# zfs set mountpoint=/srv/pub chasm/data
# zfs list

NAME         USED  AVAIL  REFER  MOUNTPOINT
chasm        215K  1.33T  41.9K  /chasm
chasm/data  67.3K  1.33T  67.3K  /srv/pub

Set up share with no real security but a decent slice of space:

# vi /etc/samba/smb.conf

[media]
path = /srv/pub/media
comment = Various media.
valid users = greyproc smbadmin uninvited
write list = greyproc smbadmin
read list = uninvited
available = yes
browseable = yes
writable = no
guest ok = no
printable = no
locking = no
strict locking = no

And verify:

$ smbclient -L localhost -U greyproc

Enter greyproc's password:
Domain=[Khazadum] OS=[Unix] Server=[Samba 3.5.8]

        Sharename       Type      Comment
        ---------       ----      -------
        ...
        media           Disk      Various media.
        ...
        IPC$            IPC       IPC Service (Here Be Dragons (Frostfall 3.5.8))

Reference:

https://launchpad.net/~dajhorn/+archive/zfs
http://groups.google.com/group/kqstor-zfs-discuss/browse_thread/thread/c4cd225c45530964?tvc=2
http://zfsonlinux.org/

Paranoia and Filtility

While trying to figure the reason Vista sometimes can't seem to read optical media, I came across this rather amusing nugget regarding mini-filters, aka file system filters:
"A file system filter driver intercepts requests targeted at a file system or another file system filter driver. By intercepting the request before it reaches its intended target, the filter driver can extend or replace functionality provided by the original target of the request. "
Source: http://www.microsoft.com/whdc/driver/filterdrv/default.mspx

Well, golly, seems sort of scary to me; as if it weren't bad enough having keyboard hooks for any aspiring malware writer to whet their skills with, now Microsoft makes it easy for any skript kiddie to get themselves inserted between me and my data.  

I realize this functionality existed before, but, gosh, it's just so EASY now.  What's eye opening to me, though, is I had no idea about it; it's been around since XP!  

Obviously, the first thing I did after reading about it was to see what was hanging about on my system:

C:\bin\si>fltmc

Filter Name                     Num Instances    Altitude    Frame

------------------------------  -------------  ------------  -----

aswFsBlk                                7       388400         0

PROCMON20                               1       385200         0

aswMonFlt                               9       320700         0

luafv                                   1       135000         0

SbieDrv                                 8        86900         0

FileInfo                                9        45000         0

Huh.  Happily, Microsoft provides a handy reference in the form of an excel spreadsheet for any 'registered' filters: http://www.microsoft.com/whdc/driver/filterdrv/alloc-alt.mspx (File System Minifilter Allocated Altitudes); aswsFsBlk belongs to... Avast.  As does aswMonFlt.  The amusing part is that I only use Avast to scan some large data drops... never for residential shield. (For that matter, I don't use any AV... but I also don't use IE, so... it works.).  It looks like Avast needs its wedge to exist constantly.  This, I think, is the final straw regarding antivirus for me; I'm tossing the lot of them (AVG, Avast, Avira... ) into a vm and scanning from there.

I mean, seriously, why does a program need a filter in place to open and read a file?  When that's ALL I want it to do?  

ProcMON20, of course, is Process Monitor's wedge.  This one makes sense; it needs to monitor all file I/O to show me pretty screens of ACCESS DENIED.  I like that.

luafv, on the other hand, belongs to Microsoft.  This blog post: http://fsfilters.blogspot.com/2010/02/deal-with-luafvsys.html does an excellent job of explaining what it is, and its functionality.  

However, I don't need that; I'm running as Admin, and I want my misbehaving programs to do so with wild abandon!  This one was easy to get rid of; I just had to turn off UAC's virtualization. 

SbieDrv is Sandboxie's wedge. This one's existence makes sense, as well --- except, again, I get annoyed at programs that keep running after I've closed them.  For fun, I wanted to see if I could unload it (since it wasn't running):

C:\bin\si>fltmc unload SbieDrv

Unload failed with error: 0x801f0010

Do not detach the filter from the volume at this time.

Huh; doesn't seem I can.  Certainly there must be some performance penalty, to have every filesystem api call traverse a filter chain, no?  Granted, altitudes decide just what they 'see', but, still... mightn't this be just one more drop in the bucket of what made Vista such a dog?

Moving on, to the last one, we've got FileInfo.  This is simply bloat.  It serves both ReadyBoost and Superfetch: both of which I've disabled.  (This is a decent explanation of it: http://winprogger.com/?p=971)

Despite having them disabled, this filter is still loading, and, presumably, being referenced every time filesystem I/O occurs.

Naturally, I wanted to get rid of it, so, expecting it to behave, I bopped into regedit, and set the service start type to 4... and it still started... which admittedly had me scratching my head.

(DISCLAIMER: Following random advice on the Internet is BAD. Don't do this on your own system.)

So, I took the sledgehammer approach, and simply deleted the service key altogether... and it worked!

As I type this, I get the following Kirsten Dunst Turning Japanese (NSFW) warm and tingly feeling giving result:

C:\>fltmc


No filters loaded


Vunderbar! I can't say it feels faster, but I know my bits appreciate not being fondled quite as much; at least, that's what the voices in my head told me, when I was changing my tinfoil headgear.