Monday, 29 October 2012

Monitoring Data Centers in the path of Hurricane Sandy

Using The NOC (, I was able to monitor a bunch of data centers in the path of Hurricane Sandy, the results highlighted something I never really considered...

When the charts indicated the data center was unreachable from Sandy I would try and access them from my local machine as well as their status updates on twitter or status websites when available. Mostly when the charts indicated they were offline I could find a status update confirming this, however for example with NYI ( which was online the entire time I have clear indication that they dropped off...

After a quick investigation using BGPlay ( it would seem that the various links between my monitoring system in Kansas and the data centers where offline due to Sandy or flapping and were clearing influencing my access to NYI although other routes to NYI stayed online all the way (clearly the reason for Netcrafts 100% uptime article about NYI). NYI definitely did an amazing job, my main point is from a pure monitoring perspective: 

1. Have more than one monitoring location if possible, especially as certain locations may give you an all clear and others indicate problems (not that NYI could do much about this in any case)

2. Keep in mind your BGP peers are uber important, similar to what happened during Hurricane Katrina a data center may remain online but your peers may fail, especially due to fuel shortages, in Katrina the one data center even delivered fuel to a peer to keep them online, perhaps something to consider in planning.

3. If your data center offers monitoring of your servers/services it should by no means be your only monitoring service, if an entire data center goes down chances are you won't be receiving any notifications.

Thursday, 25 October 2012

Afrihost moves to MTN

Afrihost ( recently announced a move to MTN ( from IS ( for all their ADSL users, on 16th October the planned move was executed... Everything however did not go as smoothly as they must have planned though and latency/speed complaints continue today (1st Nov). With hindsight at 20-20 below are a few things I picked up on from the move...

1. Good communication really helps with calming the crowd, Afrihost managed to do this in large part with social network messages and replies to most user comments. Many people have mentioned dropped calls and long support call queues which can easily happen when many subscribers have the same issue, of course if they can answer the call and put you in a queue they can offer an automated service to help manage the queue better, like offering a call back etc

2. If you promise to call someone, call them... many users complained of someone offering to call them back but then never calling, this just in turn angered their customers more...

3. Social sites play a huge role, although they didn't post much during the stormy days on their facebook page they seem to be mostly in control of Facebook and Twitter, the main problem being those complaints for which they don't have an immediate solution, in which case they can only try offer assurance of future plans although personally I think this should have been part of the original plan.

4. If you own multiple companies (Afrihost owns Axxess, for the love of all that is good don't move both to a new network at the same time... Most users think these are two independent ISP's and are confused why both moved, additionally they have two companies worth of support calls in one, this was just a plain bad move unless unavoidable...

5. They announced carefully planning had gone into the move and all was covered, the day of the move many could not connect though... this was blamed on a lack of IP address pools on the Telkom ( side of things. Telkom provides the ADSL lines in South Africa, Afrihost claim Telkom didn't implement their request correctly, who knows what really happened...

6. IS ( have a Telkom connection for traffic from clients in Cape Town, after the move however all these clients ended up being redirected via the MTN in Johannesburg, adding latency to all those users, especially for their local Cape Town servers... Afrihost say they are now fast tracking a plan for Cape Town but it's to late for many users, additionally they claim this influences only certain users in Cape Town, which can obviously not be the case...

7. Latency increased for some paths and seems like it's going to stay, the graph at the top of the page shows monitoring I configured for work to my monitoring service at from Johannesburg, clearly certain links have remained the same but many have gone up in latency. Many users complained of unacceptable gaming latency, jumps of 60ms+

8. Youtube - users mentioned Youtube being slower and jitter like, I did some testing and it seems like MTN doesn't have a local cache server for Youtube like IS, Mweb, and other local ISP's provide, therefore all the traffic goes overseas and in turns makes Youtube slow. Not sure if they are working on a solution to this?

9. Many users, especially those using uncapped accounts, posted speed test results which show speeds at a fraction of their respective line capacity... I did a quick download test and the speed was on par with the line for local speeds, international seemed to be about half the line speed on average and unstable in both Johannesburg and Cape Town. The same download was running at full speed on my MWeb account.

10. Have a news server ready, with the move the old news server was no longer available, something easily overlooked but many users remain loyal to NNTP

11. They provided another ADSL realm on the previously used IS network to help with the moving pains called, overall a nice gesture but many couldn't as they needed to change router settings and for us to gain access to clients mean't driving out, which is just not a viable option. Afrigamer seems to have moved over to MTN to as many users noted the same speeds/latency issues after the free use of the realm expired.

Friday, 19 October 2012

Linux Mint 13 Cinnamon Upek fingerprint reader config

 I installed Linux Mint 13 Cinnamon on my ThinkPad T520 last night and was tempted to get the fingerprint scanner working... the whole process was almost completely painless and very cool!

USB device details:

147e:2016 Upek Biometric Touchchip/Touchstrip Fingerprint Sensor

Packages required:

sudo apt-get install fprint-demo libfprint-dev libfprint0 libpam-fprint aes2501-wy
sudo fprint_demo

fprint_demo is a GUI application to enroll different fingers etc for your system and worked a charm, there is a console alternative although I didn't give it a try called: pam_fprint_enroll

Next you need to edit: /etc/pam.d/common-auth

Add the following line:

auth sufficient

Note that if you add it say 3 times you can try logging in with your fingerprint 3 times before PAM moves onto asking you for a password. You can also change sufficient to required if you would like fingerprint + password auth.

Friday, 5 October 2012

Linux Time server using NMEA output from a GiSTEQ USB GPS Dongle

It's amazing how accurate the time on a GPS device actually is, with as little as 4 satellites your location can be triangulated and time obtained. The time comes from 4 caesium or rubidium atomic clocks on board each of those satellites, the atomic clocks are so accurate they lose no more than one second in millions of years.

You'll need to install ntpd and gpsd on your system, in Ubuntu:

sudo apt-get install ntp gpsd

For older versions of Ubuntu eg. 11.04, you may need to get a newer version of gpsd, in my case I experienced a silent failure as a result of an older gpsd version. If you cat /dev/ttyUSB0 (assuming thats your device) and don't get any output try:

Setting Serial Baud: sudo stty -F /dev/ttyUSB0 38400
or Initializing the device: sudo minicom --device /dev/ttyUSB0

Spare netbook I have running as a NTP server, indoors but still has a fix from 8 GPS satellites...

To configure the gpsd edit the /etc/default/gpsd file and change it to look like this:

# Default settings for gpsd...

Then edit the file /etc/ntp.conf and change it to look something like this (just so you now, this file will allow remote systems to sync to your server):

driftfile /var/lib/ntp/ntp.drift

# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

# Query the local GPSD daemon
server minpoll 4 maxpoll 4 iburst
fudge time1 0.420 refid GPS

server minpoll 4 maxpoll 4 iburst prefer
fudge refid PPS

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

# Local users may interrogate the ntp server more closely.
restrict ::1

You can go through the ntp.conf file and change it to your own setup, the above worked perfectly for me.

You can check your config on the time server by using: ntpq -p (sample output below)

     remote           refid      st t when poll reach   delay   offset  jitter
*SHM(0)          .GPS.            0 l   15   16  377    0.000   -2.578   5.689
 SHM(1)          .PPS.            0 l    -   16    0    0.000    0.000   0.000

You can query the time server remotely with ntpdate (see below), remove the -qv to set your local clock to the time server clock.

root@etienne-Lenovo:~# ntpdate -qv
 5 Oct 14:22:08 ntpdate[3569]: ntpdate 4.2.6p2@1.2194 Fri Sep  2 18:37:16 UTC 2011 (1)
 5 Oct 14:22:16 ntpdate[3569]: adjust time server offset 0.005287 sec

Overall I'm extremely happy with the GiSTEQ USB GPS dongle, it's a reliable device and a perfect device for the $40 it cost me with shipping to South Africa. For my time server purposes and playing around it works perfectly - that being said - the time keeping club members are very strict! They usually speak of inaccurate in nanoseconds (ie. Linux won't cut it as it uses milliseconds, so they tend to use BSD) and would probably never consider a non Pulse Per Second (PPS) device for time keeping... ;-)

Thursday, 4 October 2012

GiSTEQ USB Dongle in Linux

I ordered a GiSTEQ USB dongle (SKU: C5-GR110-01) a while ago and it finally arrived today! I've been in the mood to play with NTP sync etc from a GPS dongle for a while and ordered one with the SkyTraq chipset instead of the commonly found SiRFstar III. This was mainly due to a reported problem resulting in the NMEA offset drifting and being unreliable...

The main aim being to geek out rather than anything serious as these dongles arn't built with timing as a primary purpose (there are more reliable Stratum 1 time servers available on the net for free...)

I of course immediately started playing around with getting the device working in Linux, the device appears as:

Bus 006 Device 002: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port

Status of the GPS:

Green LED continually on: Location not fixed yet, may take a while
Green LED flashing: Location has been fixed

If you cat /dev/ttyUSB0 and see characters than don't make any sense, ie. lines that don't start with $GP your baud rate might be a problem... You can adjust the baud rate in Linux to 38400 with the following command: stty -F /dev/ttyUSB0 38400

The lines you want to be seeing:


The above lines were the main ones I saw coming through, now to make sense of the important (well to me at least) ones... You can find a nice list of all the codes and analysis of each at:


GPGGA - GPS Location Fix Data
Time (UTC/Zulu): 143830.462 => 14:38:30 462 millisecomds
Latitude: 2547.2945
North/South of Latitude: S
Longitude: 02815.8548
East/West of Longitude: E
Fix Quality: 1 (0 - None, 1 - GPS Fix, 2 - DGPS Fix)
Number of Satellites in Use (not View): 05
Horizontal Dilution of Precision: 1.7
Altitude: 1441.4
Altitude Unit: M - meters
Geoidal separation: 17.8 (WGS-84 Earth ellipsoid - Average sea level)
Geoidal separation Unit: M - meters
Age of Differential GPS data (seconds): blank
Differential reference station ID: 0000
Checksum: *74


GPVTG - Track made good and ground speed
True Track degrees: 119.8
Relative to True North: T => True
Magnetic Track degrees: 99.1
Magnetic Track true: M
Speed over ground: 000.0
Speed in units: N => Knots
Speed over ground: 000.0
Speed in units: K => kilometers/hour
Checksum: A*0C

I'll blog about the Gisteq as a time source soon but in the meantime I wrote a quick BASH script to display the results in a loop:

# Gisteq GPS USB Dongle

echo "Setting the baud rate to 38400..."
stty -F /dev/ttyUSB0 38400

while read input; do
 if [ "$line_id" = "GPGGA" ]; then
  value_array=(`echo $input | tr "," "\n"`)
  gps_time=`echo $gps_time | awk '{for(i=1;i<=length($0)-4;i+=2){printf("%s:",substr($0,i,2))}}'|awk '{sub(/:$/,"")};1'`
  echo "UTC Time: $gps_time"
  echo "Latitude: $latitude $latitude_direction"
  echo "Longitude: $longitude $longitude_direction"
  if [ $gps_fix -gt 0 ]; then
  echo "GPS Fix: $gps_fix"
  echo "Satellites in use: $satellites"
  echo "Altitude: $altitude meters"
 elif [ "$line_id" = "GPVTG" ]; then
  value_array=(`echo $input | tr "," "\n"`)
  echo "Speed: $speed kph"
  echo "######################"
done < /dev/ttyUSB0