Sunday, 30 December 2012

ONMS.Net - Update on progress


I managed to do some work on The NOC at ONMS.Net with all the holidays in december. Above is an example of a graph showing the speed of a fan within the server hosting the application, imagine being able to track every minute or second of a specific device/object/event and keep that information for the future or take immediate action when something goes wrong... it amazes me to have the ability to monitor things so closely, pretty cool!

We need to be able to handle a maximum of 2.5 million updates per key though for every month, definitely time for some additional server power!

Saturday, 29 December 2012

NGINX fpm-php 504 Gateway Time-out

I recently encountered the 504 error on NGINX due to a php script fetching data from a slow database server, after 60 seconds of runtime the execution simply timed out... I tried all the suggested solutions I found on forums but they didn't seem to work, the solution that worked here:

Within nginx.conf under the location section and under the server section add the lines starting with fastcgi below:

server {
 location ~ .php$ {
  fastcgi_connect_timeout 60;
  fastcgi_send_timeout 180;
  fastcgi_read_timeout 300;

The timeout values above are all in seconds and may need some tweaking, also remember to check the max_execution_time setting in php.ini if not set within the script itself to extend the default php execution timeout. Remember to restart nginx to load the changes made to your nginx.conf

Thursday, 20 December 2012

Incredible - 6.28 years and rebooted 12 times

Our one server hosted with Hetzner is a recycled one purchased on the Hetzner market. It naturally runs Linux and uses two Samsung drives in a software RAID config. I was amazed to see a power cycle count of 12 on drives with a total runtime of 6.28 years so far, the statistics for both drives:

root@onms ~ # smartctl --all /dev/sda
smartctl 5.41 2011-06-09 r3365 [i686-linux-3.2.0-24-generic-pae] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint P80 SD
Device Model:     SAMSUNG HD160JJ
Serial Number:    S08HJ10L455334
Firmware Version: ZM100-41
User Capacity:    160,041,885,696 bytes [160 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 4a
Local Time is:    Wed Dec 19 23:12:39 2012 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         ( 3677) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (  61) minutes.
SCT capabilities:            (0x003f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   100   100   051    Pre-fail  Always       -       6
  3 Spin_Up_Time            0x0007   100   100   025    Pre-fail  Always       -       5760
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       12
  5 Reallocated_Sector_Ct   0x0033   253   253   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   253   253   051    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0025   253   253   015    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       55101
 10 Spin_Retry_Count        0x0033   253   253   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0012   253   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       12
187 Reported_Uncorrect      0x0032   253   253   000    Old_age   Always       -       196608
190 Airflow_Temperature_Cel 0x0022   118   070   000    Old_age   Always       -       40
194 Temperature_Celsius     0x0022   118   070   000    Old_age   Always       -       40
195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       115166123
196 Reallocated_Event_Count 0x0032   253   253   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   253   253   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   253   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x000a   100   100   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   100   100   000    Old_age   Always       -       0
202 Data_Address_Mark_Errs  0x0032   253   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     49333         -
# 2  Extended offline    Completed without error       00%     49277         -

Note: selective self-test log revision number (0) not 1 implies that no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@onms ~ # smartctl --all /dev/sdb
smartctl 5.41 2011-06-09 r3365 [i686-linux-3.2.0-24-generic-pae] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint P80 SD
Device Model:     SAMSUNG HD160JJ
Serial Number:    S08HJ10L455779
Firmware Version: ZM100-41
User Capacity:    160,041,885,696 bytes [160 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 4a
Local Time is:    Wed Dec 19 23:13:28 2012 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         ( 3699) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (  61) minutes.
SCT capabilities:            (0x003f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   100   100   051    Pre-fail  Always       -       7
  3 Spin_Up_Time            0x0007   100   100   025    Pre-fail  Always       -       5632
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       12
  5 Reallocated_Sector_Ct   0x0033   253   253   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   253   253   051    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0025   253   253   015    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       55099
 10 Spin_Retry_Count        0x0033   253   253   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0012   253   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       12
187 Reported_Uncorrect      0x0032   253   253   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   130   076   000    Old_age   Always       -       36
194 Temperature_Celsius     0x0022   130   076   000    Old_age   Always       -       36
195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       109023062
196 Reallocated_Event_Count 0x0032   253   253   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   253   253   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   253   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x000a   100   100   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   100   100   000    Old_age   Always       -       0
202 Data_Address_Mark_Errs  0x0032   253   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     49331         -
# 2  Extended offline    Completed without error       00%     49276         -

Note: selective self-test log revision number (0) not 1 implies that no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@onms ~ #

Tuesday, 4 December 2012

Intel SSD 520 series unboxing

I had the opportunity to help unbox an Intel SSD 520 series drive recently and being the the first SSD drive I physically encountered I was very excited to view it from every angle... In the years before SSD geeks quite quickly encountered new drive types in systems, servers, etc - SSD however seems to be coming in at a much slower pace, one reason I suspect is simply the cost of the drives being in my view to much!

 
 
The Intel SSD drives are still quite expensive and hard to source from large local suppliers, especially when looking for a specific model. Overall I was amazed at how much less the drive itself weighs, especially when compared to it's spindle driven brothers already in field. 

Intel has always been one of my favourite brands, the drive seems to be packed with Intel branded chips as can be seen in some online teardowns. The drive comes with an amazing 5 year warranty, which alone is simply awesome considering previous solid state drive longevity being well rubbish.


The SSD connects to your system using SATA and runs at 6Gb/s, this drive's performance does not disappoint, my uncle (the owner of this baby) has noticed a definite increase in performance. In addition Intel has also developed software allowing users of their SSD to monitor the drive's statistics in depth! I would love to get a busy database server running off of one of these drives to really give it a run for it's money!

The only con so far... Intel needs to do a better finishing job on the back of the drive, not that it matters beyond cosmetic but it does give a bit of a "why the hell did I pay so much and they couldn't take time to finish the back properly" feeling...