Tuesday 14 December 2010

SNMP setup on Windows for Zenoss

Intenally we use Zenoss to monitor our servers and systems. In order for Zenoss to query lots of details on the windows servers we are using the free snmp informant program on each Windows host. http://www.snmp-informant.com/ the free version has distribution restrictions but can be used for any purpose for free.

To get this working (based on Win 2003 - other versions of windows might deviate slightly):

  1. In Add Remove Programs, windows components, inside "Management and monitoring Tools" select "Simple network management Protocol". This will probably require the windows installation media.
  2. While in there install the "WMI SNMP Provider" which is also needed by Zenoss.
  3. Windows SNMP service properties, security add community name you want to use to secure this service, read only. Accept snmp packets from specific hosts.
  4. Install SNMP informant
  5. Allow port 161 UDP through firewall for SNMP traffic
A Zenoss server will now be able to report on CPU, HDD, software installed etc.

Troubleshooting:
If WMI not connecting Check firewall
#netsh firewall set service RemoteAdmin enable to enable WMI
A great page for identifying issues is http://community.zenoss.org/docs/DOC-2520 which lists most common errors with monitoring windows devices.

Friday 10 December 2010

HP DL360 G5 very slow hard drive performance

One of our servers became free recently (HP Proliant DL360 G5 with 2 dual core 3GHz processors, 9GB RAM and 4 147GB, 15k rpm hdd in RAID(0+1) - or RAID 10 never sure of the difference). Nice fast server I thought.

So I install VMWare ESXi 4.1 and start testing. Abysmal performance on the hard drives! After much testing I work out that although read speeds are ok, write is useless (26MB/s read, <1MB/s write). I was expecting double that on the read speeds, far more on the write. I try a cheap iSCSI setup with FreeNas and get ~ 12MB/Sec read and write so the server itself is fine. I'm using iometer for this testing, a windows application from 2006 which takes a bit of setting up but is free and very powerful when you figure out how it works.

Eventually after much testing I figure out that although this server has a write cache it does not have a backup battery on this cache. VMWare is detecting the missing battery and deciding not to use the cache in case of power failure which would lead to corrupt data. I've just ordered HP parts 398648-001 (Battery for P400i raid controler cache) and 409125-001 (power cable for this part) which should get this server back up to speed. The battery was substituted with part 381573-001 which is compatible and replaces 398648-001.

I had some trouble finding where to buy the parts but eventually came across www.chilternitparts.com who seem to have lots of parts for HP and Dell systems.

UPDATE:
The battery and cable arrived. Installed with no problems (there is a handy guide printed on the inside of the server cover) and performance has increased greatly. The battery is still charging 4 hours after installing it but VMWare has obviously decided to trust it as write speeds have gone from 0.63MB/s to 23.49MB/s (iometer settings - 1 worker with 50MB file on c:\ drive; 4k; 0%read; 0% random, left to run for about 5 min).

My domain controller (2003 Small Business Server, Poweredge 2600) currently gets 30MB/s during backups - normally it gets nowhere close to this in daily use. However the tests I have been doing are deliberately hard on the throughput - by making the tests easier (32k block size instead of 4k, 32 job queue, 10GB test file to ensure I could out supply the cache and get actual throughput) I get an easy 60MB/sec write speed and 75MB/s read speed. Writes using cache hit a peak at 125MB/s and when I used a smaller file to read pure cache I was getting a rock solid 250MB/s. Tests over - its now time to start thinking about actually getting the Domain Controlelr virtualised. Probably an overnight job to do before Christmas. Luckily with another IT guy one of us can do the 6pm start cold clone part and the other one can do the 6am check clone thoroughly and bring live.
UPDATE: Nope that did not work.  Cold clone over the network took far longer than expected (17 hours for ~ 115GB of files over 3 partitions totaling ~ 260GB), partly due to us resizing the disks, partly due to it just being a known slow process.  Plan B is to find a long weekend, might have to delay as don't want to do it during our busy period (Jan, Feb, Mar).

VMWare iSCSI targets with FreeNAS

Due to budgets (I think I remember what they are, its been so long since I had one with anything left in it..... :) we do not have the money to get a decent SAN system for our VMWare environment at this point. Local storage works fine - its not as flexible as a SAN and we loose some functionality but it is fast enough and reliable.  However as we have just purchased Veeam Essentials for backup of our virtual machines I thought it would be worth trying a few things out to see how it would work. Definitly a productive exercise as although not as fast as the directly attached storage it gives us more flexibility and having a few hundred spare GB for testing is very useful as speed is not such a big issue there.

Steps required to get this going:
  1. Install FreeNas and setup iSCSI target, a good guide (ignoring the first bit which deals with installing it in a virtual environment) is here: http://www.sysprobs.com/nas-vmware-workstation-iscsi-target 
  2. Setup Win7 to connect to iSCSI (if you want to for testing).
  3. Setup VMWare ESXi to connect to the iSCSI target:http://www.sysprobs.com/connect-freenas-iscsi-disks-vmware-vsphere-4-esxi4 
  4. Profit!
If you are using Veeam to do backups you can increase the backup speed greatly by allowing Veeam to connect directly to the iSCSI target rather than using the ESXi host as a proxy. This also reduces the workload on the ESXi host. so is a good thing all round. Handy guide to doing this here:
http://jpaul.me/?p=334