Thinking problem management!

Seeking to eliminate the "techie curse"

2012-01-18T06:15:00.000+02:00

I wrote about the "techie curse" some time ago. Techies don't create documentation and most environments are a SOP (Standard Operating Procedures) desert. In every job in which I have started, no-one has ever given me a SOP of the services being provided. I would have to "sink or swim." Most techie managers who were treated in this manner, reinforce the behaviour by treating their subordinates in a similar manner. In reality they even go further to ridicule those who create documentation as bureaucratic. By this they imply that it is a waste of time but often the underlying reason is to be able to be a hero, as when the sh1t hits the fan, they are the only ones able to resolve it, as all the knowledge is in their heads and not on paper for anyone else to attempt to resolve. I have even been told that it stops their ability to be "business flexible." What a load of codswallop! Imagine if a new hire was given a SOP of the service they are responsible for delivering on day one of their tenure. Alas, most techies spend the first three weeks of their job being unproductive.

I have been meaning to update the orginal SOP template ever since it became a reference in Wikipedia's entry for callcentres for a short time period. Here is the updated SOP template!

Documentation - the techie curse

2012-01-18T06:12:00.000+02:00

Techies are cursed by the reluctance to create documentation. This is so bad when a member of the techie sect actually starts becoming diligent and creates some documentation he or she is ridiculed. My opinion is that technologists have evolved into a mindset where the expectation is to be spoon fed and mollycoddled. In the good old days, it was common practice to tell people to RTFM. This was the "old school" way I learnt Computer Science. Nowadays this is deemed rude. "RTFM or Google it are two inappropriate responses to a question. If you don't know the answer or don't wish to help, please say nothing instead of brushing off someones question. Politely showing someone how you searched or obtained the answer to a question is acceptable, even encouraged. If you wish to remind a user to use search tools or other resources when they have asked a question you feel is basic or common, please be very polite. Any replies for help that contain language disrespectful towards the user asking the question are unacceptable and will not be tolerated." (from the Forum Policies and Expectations - Ubuntu Forums). If these wimps are not expected to do some of their own reading and research there is no chance in hell that they will create any documentation. It is time to be a traffic cop.

Being involved in problem management I have gathered some stats about the major incidents that have crossed my desk. The service that causes the most major incidents is data networks. This is no surprise as in this neck of the Kalahari the former telecommunications monopoly that dominates the market is not renowned for service excellence. This is followed closely by messaging and then voice. Again no surprises as the email and voice systems are by far the most used IT systems.

The statistic that is of more interest in this blog is the second one. The cause of most major incidents is around mis-configurations. This is often because techies do not configure a system or solution correctly. Further analysis usually indicates that the techies never bothered to document it in the first place and the reason for the incorrect configuration was that they were flying blind. Next is component failure. Again this is traced back to not knowing what systems are where and what the warranty and maintenance schedules happened to be! Yes, you guessed right! Again insufficient documentation. The next category are the vendors (carriers and service providers). I will deal with vendor management in another blog post.

The documents that should exist and are never created are Standard Operating Procedures, also known as SOPs. The term is used to describe a best practice approach to executing tasks related to hardware and software maintenance, as well as incident and change management. SOPs are created to provide specific documentation for various processes, usually highly-technical processes. Businesses are finding that they lack this kind of documentation; either there has been inadequate process definitions altogether, or they have no documentation in place. When reviewing the existing documentation, it is important to know where a particular SOP may be needed, or that the SOP available is detailed enough or not current.

The reasons for a SOP are:

To provide techies with all the environmental and operational information necessary to perform a job properly.
To ensure that IT operations are performed consistently to maintain quality control of people, processes, partners and products.
To ensure that processes continue uninterrupted and are completed on a prescribed schedule.
To ensure that no failures occur in the live environment that could result in major incidents.
To ensure that approved procedures are followed in compliance with corporate due diligence and compliance to regulations.
To serve as a training document for teaching newbies about IT processes.
To serve as a checklist for the techie's superior when job performance is being evaluated.
To serve as a checklist for auditors and they become an important part of compliance due diligence.
To serve as an historical record of the how, why and when of steps in an existing process so there is no hearsay when reviewing a process when a process has changed. As people move from job to job within and between companies, unwritten knowledge and skills disappear from the workplace. Properly maintained written SOPs can chronicle the best knowledge that can serve new workers when older ones move on.
To serve as an explanation of steps in a process so they can be reviewed in major incident analysis.

The SOP should take the following format:

Title and description (include reference numbers)
Definitions
Scope
What Vital business functions are being serviced?
Lessons learned
Equipment, hardware and software lists
Daily availability checklists
Diagrams
Full system configuration
Testing plans
Schedule of administrative and maintenance tasks
Costs and charge back models
Risk mitigation actions
Continuity plans including disaster recovery and methods for restoring data
Escalations

The following are guidelines that apply. Use pictures, diagrams, tables, and bulleted lists for quick reference. Often screen shots are great. Keep the steps involved to a maximum of ten. If the steps are more than this amount, break it up into logical sub procedures. Review the effectiveness of SOPs after a few weeks and make necessary changes if in-the-field practice suggests that descriptions should be improved. A SOP template is available here.

Some of the important Infrastructure SOPs required are:

SOP for major incidents and problem management
SOP for handling security incidents
SOP for messaging
SOP for directory services
SOP for WAN quality of service
SOP for video conferencing
SOP for corporate voice, least cost routing and branch to branch links
SOP for call centre
SOP for voice recording
SOP for SAN and storage
SOP for backup and restore
SOP for virtual servers
SOP for patch management
SOP for server images
SOP for hosting
SOP for access management
SOP for hardware management
SOP for anti-virus
SOP for network management
SOP for cabling
SOP for licensing and asset management

and my final opinion on the subject is, "READ THE ~!@#$%^&*()_ SOP."

NB: The SOP template has been updated!!! Read about it here.

Radwin video

2012-01-17T07:56:00.000+02:00

At least you guys could have cleaned up the cables at 0:57!

Love of my life

2011-08-19T21:21:00.001+02:00

Testing webradio

2011-08-01T15:20:00.000+02:00

Listen:

Africa's fibre routes

2011-07-27T12:08:00.000+02:00

African Undersea Cables -- A History

View more presentations from Steve Song

The final landing...

2011-07-23T07:43:00.002+02:00

This day in history: Tranquility Base, the Eagle has landed!

2011-07-20T09:03:00.002+02:00

Cellular providers in South Africa

2011-07-13T20:05:00.002+02:00

Thinking networks

2011-07-10T07:58:00.000+02:00

Thinking Networks by Prof. Simon Saunders

View more presentations from Zahid Ghadialy

Happy Birthday, USA

2011-07-04T07:54:00.001+02:00

1984 reunion

2011-07-03T15:21:00.000+02:00

Ultrabroadband and QoS

2011-06-18T13:54:00.000+02:00

Yeha Noha

2011-06-06T05:05:00.002+02:00

Artic sounds

2011-06-04T08:24:00.000+02:00

How I Reached the North Pole by Frederick A. Cook.
Listen:

Neotel strategies

2011-06-01T21:04:00.000+02:00

Telepresence? The opinion remains the same, stated five years ago.
Remember this? It was posted on 26 January 2008.

Never mind deploying to a relief aid area in a developing country, I doubt this stuff would be deployed to a hurricane or tornado disaster area in the first world!

A foundation for RCA

2011-05-29T12:40:00.000+02:00

A Foundation For Root Cause Analysis 2011.05.25

View more presentations from William Corcoran

Think Pink

2011-05-23T17:25:00.000+02:00

wifi interference (with Wavion)

2011-05-18T09:13:00.001+02:00

Ray and Callie

2011-05-16T07:51:00.000+02:00

Radwin PtMP

2011-05-15T15:56:00.000+02:00

Motorola Defy

2011-03-06T14:33:00.000+02:00

It need one of these! I lost my BB to a cup of Milo.

How do you use OpenDNS?

2011-03-06T14:11:00.003+02:00

Poll about LinkedIn

2011-02-14T17:29:00.001+02:00

Sinclair ZX Spectrum

2011-02-08T12:29:00.000+02:00

My Mate Simon emailed me a recent discovery of a boxed ZX Spectrum!

Remembering Challenger

2011-01-30T09:07:00.002+02:00

Problem management on Practitioner Radio

2011-01-26T05:42:00.000+02:00

pr 2 from ServiceSphere on Vimeo.

Pingtest and Speedtest

2011-01-17T06:11:00.000+02:00

Scanner on ABSA ATM

2011-01-09T06:54:00.000+02:00

First time I saw one of these in real life!

Ninja iPhone mode

2011-01-04T09:15:00.000+02:00

Is is not all it's cracked up to be!

Pingtest and Speedtest

2011-01-04T05:50:00.002+02:00

Pingtest and Speedtest

2011-01-02T08:21:00.000+02:00

Pingtest and Speedtest

2011-01-01T10:39:00.000+02:00

Pingtest and Speedtest

2010-12-30T05:37:00.000+02:00

Merry Christmas!

2010-12-25T10:41:00.000+02:00

Christmas is cancelled

2010-12-23T13:54:00.002+02:00

Christmas card string

2010-12-23T12:28:00.000+02:00

Product amplifier

2010-12-21T09:56:00.000+02:00

The concept of creating custom sound and audio using an amplifier with sliders is a common concept. Certain slider settings are even associated with specific genres like pop, rock, classical, voice, flat, etc.

Internet broadband products are no different. There isn't a single product that will suit the market and debating about uncapped versus usage based products misses the point. That is but a single variable slide of any Internet broadband product.

So how does a product amplifier look for an Internet broadband product?

The sliders with variable settings are:

Price - is directly influenced by the position of the other sliders.
Speed - a slider that indications throughput.
Symmetry - an indication of the difference between upload and download speeds (typically a business requires better upload ability).
Connections - the network connection count, i.e. how many sessions (e.g. browser windows you can open).
Bucket/usage - an indication of the bucket size with uncapped being the extreme.
Time period - used to indicate peak times and off peak times.
Availability - amount of expected downtime, often categorized as gold, silver and bronze.
Content - a master slider consisting of a subset of sliders that adjusts content. This content can include: video, browsing, email, P2P, VoIP and social network as an example.
Bursting - ability to go beyond the expected throughput or usage limits.

The product amplifier is a method to create very cheap or expensive products dependant on the consumer appetite. As with audio, consumers will have genres where a certain slider setting will be preferential for a large common base. These product slide differentials are achieved using products like a service control engine.

Smokeping

2010-12-19T08:38:00.000+02:00

DSLReports.com has a Smokeping tool that can be used to monitor international broadband performance to the US.

Pingtest results

2010-12-18T09:34:00.000+02:00

Storm in Magaliesberg

2010-12-17T07:51:00.000+02:00

We went for an overnight stay to the Magliesberg and woke up to a flood.

Wireless

2010-12-16T13:29:00.002+02:00

Speedtest and Pingtest results

2010-12-16T13:28:00.000+02:00

South African weather photos

2010-12-13T08:19:00.000+02:00

South African weather photos is a great site to view current weather conditions.

Rain

Clouds

Lightning

A egte South African tjorr

2010-12-11T18:53:00.000+02:00

Wear sunscreen

2010-12-10T17:32:00.002+02:00

Mamma Mia

2010-12-10T16:21:00.000+02:00

Where the family will be tonight, celebrating Kath's 40th birthday!

Layer 2 data cente connectons

2010-12-06T05:34:00.000+02:00

Download#102: Packet Pushers show on data centre communications at layer 2

Another great podcast on data centres...
You can find Packet Pushers here.

I have always been of the opinion that layer 2 is acceptable in this application and there are no real requirements for layer 3.

Listen:

Updated results - Speedtest/Ptingtest

2010-12-05T09:08:00.000+02:00

Speed and ping tests

2010-11-30T05:56:00.000+02:00

Chicken Excellence Officer

2010-11-28T09:48:00.000+02:00

Madge Networks Farm - August 1995

2010-11-24T08:08:00.000+02:00

Andy Cranston writes: "I worked at Madge from May 1995 to February 1996. Initially at Lodge Farm near Chesham and then at Wexham just north of Slough. I was one of the software testers working for a chap called Dominic Wilde who reported to one of my top five bosses called John Grubb.

Attached is a photo taken at Lodge Farm in August 1995. Robert Madge's mother is in it (with one of her beloved canines) but my memory, alas,cannot recall any of the other faces! "

The Pareto Principle

2010-11-24T06:54:00.000+02:00

Pareto Principle Explained

View more presentations from Ed Simpson.

Roald Amundsen

2010-11-23T20:48:00.000+02:00

ROALD ADMUNSEN

View more presentations from Bemaguali .

Another Madge photo from Woodside, August 1993

2010-11-23T20:45:00.000+02:00

Henry Ford

2010-11-23T07:47:00.000+02:00

Henry Ford By: Paul Yamane

View more presentations from Fordlovers.

Starbucks

2010-11-23T07:42:00.002+02:00

Starbuck's experience

View more presentations from Mohamad Faried Aguslemi.

Product management

2010-11-21T16:31:00.000+02:00

The Myths of Innovation

View more presentations from Benjamin Joffe.

Spud premeires on 3 December

2010-11-21T07:36:00.000+02:00

My only friend through teenage nights

2010-11-18T20:39:00.000+02:00

This song popped into my head again tonight! I think that it is the one I enjoy the most!!!

Pingtest and Speedtest

2010-11-15T06:14:00.000+02:00

Neotel:
Download Speed: 1301 kbps (162.6 KB/sec transfer rate)
Upload Speed: 475 kbps (59.4 KB/sec transfer rate)

Neotel:
Download Speed: 1648 kbps (206 KB/sec transfer rate)
Upload Speed: 501 kbps (62.6 KB/sec transfer rate)

IP Addressing

2010-11-14T08:41:00.000+02:00

Part 1

Part 2

Part 3

Part 4

Part 5

Part 6

Part 7

Part 8

Part 9

Pingtest and Speedtest

2010-11-14T06:54:00.000+02:00

Network management

2010-11-13T11:36:00.002+02:00

SNMP Tutorial

2010-11-13T11:32:00.002+02:00

Pingtest and Speedtest

2010-11-13T07:14:00.000+02:00

Speedtest.net and pingtest.net results

2010-11-11T06:46:00.000+02:00

Podcast on Internet Exchanges

2010-11-01T18:42:00.000+02:00

Download#101: Packet Pushers show on Internet Exchanges

Another great podcast on Internet Exchanges...
You can find Packet Pushers here.

This refers to the current debates about peering in ZA.

Listen:

MyBroadband conference

2010-10-23T15:30:00.002+02:00

Session 2

Session 1 Part 1

Session 1 Part 2

iBurst Technology video

2010-10-23T15:22:00.002+02:00

W is for wifi

2010-10-17T17:55:00.000+02:00

Interesting statistic

2010-10-12T07:40:00.000+02:00

Gadget writes in Broadband myths busted: "Walter says that according to internal research conducted by Nashua Mobile 53% of 3G/HSDPA subscribers use less than 250MB of data a month and that 71% of 1GB Capped ADSL users never hit their cap."

and elsewhere...

Jan Vermeulen on MyBroadband writes in Slow ADSL speeds? Uncapped to blame?: “Overall though, uncapped has been putting a lot of strain on DSLAMs [Digital Subscriber Line Access Multiplexers] around the country, which has unfortunately been affecting users on all networks, especially during peak hours,” said WebAfrica. "In many areas Telkom is now only allowing 384k connections to limit the congestion.”

A ride on SpaceShipTwo

2010-10-11T06:22:00.000+02:00

Tomato QoS example

2010-10-09T08:00:00.000+02:00

Global MBB in Oslo

2010-09-28T08:37:00.000+02:00

Here are my photos of my trip to Oslo where I viewed the first commercial LTE installation in the world.

BTW: All the base stations have wifi offload.

RADWIN Solutions - Streakwave Webinar

2010-09-24T13:34:00.000+02:00

Part 1

Part 2

Part 3

Climbing a tower

2010-09-18T14:02:00.000+02:00

Someone pointed me to a youtube video of climbing a tower. However, it was taken down. However, a quick search on google discovered a leak.

Fibre channel over Token-ring

2010-09-15T08:16:00.000+02:00

Beating life into a good old friend: Fibre Channel over Token Ring

Madge photos from Martin Taylor

2010-09-05T20:17:00.000+02:00

Asian sales conference

Cow and lamb

Ringswitch team

Hitler is throttled (shaped) by his ISP!

2010-08-27T06:40:00.001+02:00

I think this one is still one of the best I created.

Hitler is told about free internet

2010-08-27T06:40:00.000+02:00

Hitler is humiliated by Winston the Pigeon

2010-08-27T06:39:00.000+02:00

Stop the pigeon!

2010-08-27T06:38:00.000+02:00

Eating Winston the pigeon

2010-08-27T06:37:00.000+02:00

Jason's gateway

2010-08-22T09:43:00.000+02:00

My mate Jason has a very interesting device that can be used in a network or data centre, no only as a serial console device but for environmental management. He designed and built the box from scartch all by himself! The device has a built-in battery backup and thus is perfect to use when utility power is unavailable. This is a more advaned form of monitoring power than Greg's way.

"Smartsavy’s Serial GSM Gateway’s are a family of rack mountable devices that provide for local and remote serial device access, and building management in a single appliance.

Available as either a STD or PRP model the Serial GSM Gateway family creates a 1 to 4 serial switch which can support up to four serial devices such as routers, switches, PABX’s, radio’s etc. All external serial devices are then accessible via four different console access methods. The access methods are Local Serial connection, Local USB connection, Remote GPRS connection and Remote LAN connection (Pro Model Only) and each method displays an easy to use software menu. A good analogy would be a KVM device, except in this case only serial data is switched. The Serial GSM Gateway is unique in that it extends the serial console port to the end user over four different communication mediums, which are: Local Serial Port, and Local USB Port using a terminal or a PC with terminal emulation software, Remote Internet over GPRS Port, and Remote LAN Port (PRO version only) using a PC or Smartphone with telnet virtual terminal connection software.

Smartsavy’s Serial GSM Gateway family also creates a building management and telemetry solution. Built-in General Purpose Input and Output (GPIO’s) ports are provided, to connect external devices to the Serial GSM Gateway, these sense and react to the outside environment. Connection of external devices such as movement sensors, sirens, generators, lights, door locks, flood sensors etc... Provide for building automation and management with comprehensive alerting and management available on the local and remote consoles as well as SMS.

Location awareness is available via GPS technology. Site location co-ordinates can be requested via a simple SMS message to the Serial GSM Gateway and a response SMS with GPS navigator compatible co-ordinates. This feature is especially helpful to large organizations with many remote sites."

Data centre checklist

2010-08-22T09:11:00.000+02:00

This data centre checklist is based on the SANS checklist and the LBNL technical best practices.

Property

Site location

Natural disaster risks
The site location should be where the risk of natural disasters are acceptable. Natural disasters include but are not limited to forest fires, lightning storms, tornadoes, hurricanes, earthquakes and floods.
Man-made disaster risks
The site location should be located in an area where the possibility of man-made disaster is low. Man-made disasters include but are not limited to plane crashes, riots, explosions, and fires. The site should not be adjacent to airports, prisons, freeways, stadiums, banks, refineries, pipelines, tank farms, and strike routes.
Infrastructure
The electrical utility powering the site should have a 99.9% or better reliability of service. Electricity must be received from two separate substations (or more) preferably attached to two separate power plants. Water should be available from more than one source. Using well water as a contingency should be an option. There must be connectivity to more than one access provider at the site.
Sole purpose
A data center should not share the same building with other offices, especially offices not owned by the organization. If space must be shared due to cost then the data center should not have walls adjacent to other offices

Site Perimeter

Perimeter
There should be a fence around the facility at least 6 metres from the building on all sides. There should be a guard kiosk at each perimeter access point. There should be an automatic authentication method for data center employees (such as a badge reader reachable from a car). The area surrounding the facility must be well lit and should be free of obstructions that would block surveillance via video cameras and security patrols. Where possible, parking spaces should be a minimum of 8 metres from the building to minimize damage from car bombs. There should not be a sign advertising that the building is in fact a data center or what company owns it.
Surveillance
There should be surveillance cameras outside the building monitoring parking lots and neighboring property. There should be security guards patrolling the perimeter of the property. Vehicles belonging to data center employees, contractors, guards, and cleaning crew should have parking permits. Service engineers and visitor vehicles should be parked in visitor parking areas. Vehicles not fitting either of these classifications should be towed.
Outside Windows and Computer Room Placement
The site location must not have windows to the outside placed in computer rooms. Such windows could provide access to confidential information and insight into the locations layout and processes. The windows also cast sunlight on servers unnecessarily introducing heat to the computer rooms. Computer rooms should be within the interior of the data center. If a computer room must have a wall along an outside edge of a data centre there should be a physical barrier preventing close access to that wall.
Access Points
Loading docks and all doors on the outside of the building should have some automatic authentication method (such as a badge reader). Each entrance should have a mantrap (except for the loading dock), a security kiosk, physical barriers (concrete barricades), and surveillance cameras to ensure each person entering the facility is identified. Engineers and cleaning staff requiring badges to enter the building must be required to produce picture ID in exchange for the badge allowing access. A log of equipment being placed in and removed from the facility must be kept at each guard desk listing what equipment was removed, when and by whom. Security Kiosks should have access to read the badge database. The badge database should have pictures of each user and their corresponding badge. Badges must be picture IDs.

Computer Rooms

Access
There should be signs at the door(s) marking the room as restricted access and prohibiting food, drink, and smoking in the computer room. There should be an automatic authentication method at the entrance to the room (such as a badge reader). Doors should be fireproof. There should only be two doors to each computer room (one door without windows is probably a violation of fire code). Access should be restricted to those who need to maintain the servers or infrastructure of the room. Access should be restricted to emergency access only during moratoriums for holidays. Service Engineers must further go to the Network Operations Centre to obtain access to computer room badges.
Infrastructure
Computer Rooms should be monitored by surveillance cameras. Each computer room should have redundant access to power, cooling, and networks. There should be at least an 45cm access floor to provide for air flow and cable management. Computer rooms should have air filtration. Computer rooms should have high ceilings to allow for heat dispersal.
Environment
Each computer room should have temperature between 13 and 24 degrees Celsius and a humidity of between 20 and 80 percent. Environmental sensors should log the temperature and humidity of the room and report it to the Network Operations Centre for monitoring and trend analysis.
Fire Prevention
There should be Inergen or Eckoshielda fire suppressant solutions in place in each computer room. There must be fire extinguishers located in each computer room. There must be emergency power off switches inside each computer room. There may be respirators in computer rooms. There must not be wet pipe sprinkler systems installed.
Shared Space
If the space is being leased then the computer room will probably be shared space. A clause should be entered into the lease stating that competitors of the business may not have equipment located in the same computer room. Lists of clients utilizing the same room should be monitored to ensure compliance. Computer equipment in shared spaces must at a minimum be in a locked cabinet.

Facilities
- Cooling Towers
  There must be redundant cooling towers. Cooling towers must be isolated from the Data Centre parking lot.
- Power
  There must at least be battery backup power on site with sufficient duration to switch over to diesel power generation. If there is no diesel backup then there should be 24 hours of battery power. There should be diesel generators on site with 24 hours of fuel also on site. A contract should be in place to get up to a week of fuel to the facility.
- Garbage
  All papers containing sensitive information should be shredded on site or sent to a document destruction company before being discarded. Dumpsters and bins should be monitored by surveillance cameras.
- Network Operations Centre
  The Network Operations Centre must have fire, power, weather, temperature, and humidity monitoring systems in place. The Network Operations Centre must have redundant methods of communication with the outside. The Network Operations Centre must be manned 24 hours a day. The Network Operations Centre may monitor news television stations and websites for events which effect the health of the data center.

Disaster Recover

Disaster Recovery Plan
The data centre must have a disaster recovery plan. Ensure that the plan addresses the following questions: What constitutes a disaster? Who gets notified regarding a disaster and how? Who conducts damage assessment and decides what back-up resources are utilized? Where are backup sites located and what is done to maintain them on what schedule? How often and under what conditions is the plan updated? If the organization does not own the data center what downtime does the service level agreement with the center allow? A list of people within the organization to notify must be maintained by the Network Operations Centre of the data center including work, home, and mobile numbers. (Including Skype and other Instant Message Names if available) How often are those people updated?
Off site backup
There must be regular off site backups of essential information. There must be a backup policy in place listing the procedure for restoring from backup and allowing for the scheduling of practice runs to test that the backups work.
Redundant site
Redundant servers may be set up in another data center. If these are set up then they must be tested during a "dry run" to ensure that they will switch over properly during a disaster.

People

External

Security officers and guards
Security guards should submit to criminal background checks. Security guards should be trained to follow and enforce physical security policy strictly (for example ensuring that everyone in the facility is wearing a badge).
Cleaning staff
Cleaning crews should work in groups of at least two. Cleaning crew should be restricted to offices and the Network Operations Centre. If cleaning staff must access a computer room for any reason they must be escorted by Network Operations Centre personnel.
Service engineers
Service engineers must log their entering and leaving the building at the entrance to the building. The Network Operations Centre should log their badge exchange to access a computer room.
Visitors
Visitors must be escorted by the person whom they are visiting at all times. Visitors must not be allowed access to a computer room without written approval from data center management. All visitors who enter computer rooms must sign a register or log.

Users

Education
Users must be educated to watch out for potential intruders who may shoulder surf or directly attempt social engineering. Users should be educated on securing workstations and laptops within the facility and laptops outside the facility, awareness of surroundings, and emergency procedures.
Policy
All users at the facility must sign Non Disclosure Agreements. A Physical Security Policy should be signed by each user and enforced by security guards.

Disaster Recovery

Organizational chart
An organizational chart should be maintained detailing job function and responsibility. Ideally the org chart would also have information on which functions the worker has been cross trained to perform.
Job function documentation
It is important to document what employees currently know about existing systems and hardware. Also all new work requests and changes, must be documented.
Cross training
Data Centre employees should be cross trained in a number of other job functions. This allows for a higher chance of critical functions being performed in a crisis.
Contact information
A contact database must be maintained with contact information for all data centre employees.
Telecommuting
Data centre employees should regularly practice telecommuting. If the data centre is damaged or the ability to reach the data centre is diminished then work can still be performed remotely.
Disparate locations
If the organization has multiple data centres then personnel performing duplicate functions should be placed in disparate centres. This allows for job consciousness to remain if personnel at one center are incapacitated.

Airflow management (mechanical)
The efficiency and effectiveness of a datacenter conditioning system is heavily influenced by the path, temperature and quantity of cooling air delivered to the IT equipment and waste hot air removed from the equipment.

Eliminate mixing and recirculation of hot equipment exhaust air

Hot aisle/Cold aisle
Rigid nnclosures
Flexible strip curtains
Blank unused rack positions
Design for IT airflow configuration
Select racks with good internal airflow

Maximize Return Air Temperature by Supplying Air Directly to the Loads

Use Appropriate Diffusers
Position Supply and Returns to Minimize Mixing and Short Circuiting
Minimize Air Leaks in Raised Floor System
Optimize Location of Computer Room Air Conditioners
Provide Adequately Sized Return Plenum or Ceiling Height
Provide Adequately Sized Supply
Use an Appropriate Pressure in Underfloor Supply Plenums

Air Handler Systems (mechanical)
The air handler fan is typically the second largest energy use in the mechanical system, and can even exceed the energy use of the cooling plant in some cases. Optimizing the air handler system for datacenter use, as opposed to relying on traditional air handler design rules developed over years of office system design, is essential to achieve an efficient and cost effective system.

Minimize Fan Power Requirements
- Low pressure drop system design
- Use Redundant Air Handler Capacity in Normal Operation
Use an Optimized Airside Economizer
- Implement an Airside Economizer
- Design for Medium Temperature Air
- Control to Avoid Unnecessary Humidity Loads

Use Large Centralized Air Handle

Use Load Diversity to Minimize Fan Power Use
Optimize Air Handler for Fan Efficiency and Low Pressure Drop
Configure Redundancy to Reduce Fan Power Use in Normal Operation
Use Premium Efficiency Motors and Fans
Control Volume by Variable Speed Drive on Fans Based on Space Temperature

Humidification (mechanical)
Humidification specifications and systems have often been found to be excessive and/or wasteful in datacenter facilities. A careful, site specific design approach to these energy-intensive systems is usually needed to avoid energy waste.

Design System to Actual Equipment Requirements

Use Widest Suitable Humidity Control Band
Specify Humidity Sensor Calibration Schedule
Provide Appropriate Sensor Redundancy
Control Humidity with Dedicated Outdoor Air Unit

Eliminate Over Humidification and/or Dehumidification

Ensure Proper Economizer Lockout
Maintain coil temperature above 13 degrees Celsius
Centralize Humidity Control

Use Efficient Humidification Technology

Use Waste Return Air Heat to Humidify
Use Adiabatic Humidifiers for Humidity and Evaporative Cooling
Use Lower Power Humidification Technology

Plant Optimization (mechanical)
When a chilled water plant is used, all the standard design best practices apply, with a few additions. The unusual nature of a datacenter load, which is mostly independent of outside air temperature and solar loads, makes free cooling very attractive and increases the importance of efficiency over first cost. Also, the typical level of redundancy and reliability can influence the value of various design options.

Maximize the chiller system efficiency
- Select Chiller for High Efficiency
- Implement an Aggressive Condenser Water Reset
- Minimize Tower Fan Power and Size Towers for Close Approach
- Use Free Cooling / Waterside Economization
- Use a Medium Temperature Chilled Water Loop
- Use Primary Only Variable Flow Chilled Water Pumping
- Consider Thermal Storage
- Monitor System Efficiency
- Rightsize the Cooling Plant

IT Equipment: Selection
The IT equipment is the reason for the facility. Increasingly, there are reasonable opportunities to increase the efficiency of IT equipment, reducing the need for mechanical infrastructure and ongoing energy use directly at the load level through the selection of IT equipment.
- Specify Efficient Server Equipment
  - Specify High Efficiency Power Supplies
  - Consider Equipment Power Consumption in Specifications
- Use Cooled Equipment Racks

Electrical Infrastructure
Protection from power loss is a common characteristic of datacenter facilities. Such protection comes at a significant first cost price, and also carries a continuous power usage cost that can be reduced through careful design and selection.

Design UPS System for Efficiency
- Maximize Unit Loading
- Select Most Efficient UPS Possible
- Specify Minimum Unit Efficiency at Expected Load Points
- Evaluate UPS Technologies for Most Efficient
- Do Not Overspecify Power Conditioning Requirements
Use Self-Generation for Large Installation

Eliminate Standby Generator
Recover Waste Heat for Local Heating Uses
Recover Waste Heat for Datacenter Cooling Use
Eliminate UPS Systems

Lighting
data centres are typically lightly occupied. While lighting is a small portion of the total power usage of a datacenter, it can be often be safely reduced through mature, inexpensive technologies and designs.

Use Active Sensors to Shutoff Lights When Datacenter is Unoccupied

Occupancy sensors

Design Light Circuiting and Switching to Allow for Greater Manual Control
Bi-Level Lighting
Task Lighting

Commissioning and retrocommissioning
An efficient datacenter not only requires a reliable and efficient design, it also requires proper construction and operation of the space. Commissioning is a methodical and thorough process to ensure the systems are installed and operating correctly in all aspects, including efficiency.

Engage Additional Design Expertise for Review and Guidance.

Perform a Peer Review
Engage a Commissioning Agent

Perform System Commissioning

Document Testing of All Equipment and Control Sequences
Measure Equipment Energy Efficiency Onsite
Provide Appropriate Budget and Scheduling for Commissioning
Perform Full Operational Testing of All Equipment

Perform Retrocommissioning

Perform a Full Retrocommissioning
Recalibrate All Control Sensors
Where Appropriate, Install Efficiency Monitoring Equipment

The bovine who wants to be eaten

2010-08-15T23:35:00.000+02:00

Rayner believes the cow is a hoot! I wonder if it is an advanced lifeform descendant of Ivan?

A Technical Services Manager Speaks (via Packet Pushers)

2010-08-15T22:30:00.000+02:00

Download#99: Runt Packet No 5 – A Technical Services Manager Speaks

This is a great podcast on troubleshooting...buried in this blog is something in the same vain, the network troubleshooting checklist!
You can find Packet Pushers here.

Listen:

Deja vue

2010-08-09T13:38:00.000+02:00

Change

2010-08-01T07:33:00.002+02:00

If there is no change you are stagnant!

Facetime

2010-07-24T21:59:00.000+02:00

Dog versus Vuvuzela

2010-06-27T11:47:00.000+02:00

Hitler and the vuvuzela - brilliant

2010-06-26T13:57:00.000+02:00

I have made my own set of Hitler parodies about broadband in South Africa. This is by 51project and is brilliant.

...and another one...

...and some history, first use of vuvuzela in 1936...

Shosholoza!

2010-06-13T13:15:00.002+02:00

Vuvuzela Orchestra plays Shosholoza from Pedro Espi-Sanchis on Vimeo.

This was a collaboration between the Vuvuzela Orchestra and the famous South African opera singer Siphiwo Ntshebe for the gala evening of the WEF held in Greenpoint in 2009.

Shosholoza can be said to be South Africa's Soccer Anthem. As is usual with African music the band members have to dance at the same time as playing a syncopated arrangement for seven notes.

The song should be heard loud and clear during the 2010 Soccer World Cup.

Falcon 9 successful flight

2010-06-13T08:13:00.002+02:00

Double-Wide Rack Cooling

2010-05-24T15:47:00.000+02:00

Monitoring downtime (it isn't about uptime)

2010-05-24T10:33:00.000+02:00

All network management products miss the point. Their primary focus is on monitoring uptime. This serves no business purpose and cannot justify any return on investment. All the development is in looking at when the situation is acceptable but none when you are in dire straits. How is this monitoring? All it does is give you a comfortable feeling. With this approach there is no difference in the value proposition of 'ping' or a network management framework product worth a million bucks.

What is required is to monitor downtime. Outages require a special view that is serialized and has the outage time. However, it is not the length of time of the outage that is important but the crucial time periods within the outage the provide the metrics. These metric times are aligned with the expanded incident lifecycle.

Where is the real worth in reporting 99.9% availability? This would translate to 0.1% outages but this is meaning less because:

do we know which services were affected
what was the business impact
what were the prevailing conditions during the outage
what was the resolution (and was a workaround implemented)
did we annotate the visual, proximate and root causes?
do we know the resources assigned to work on the outage
how was the outage classified and prioritized
what was the risk assessment and mitigation

A metric of 0.1% answers none of these questions and no network management product in existence reports on them. As an example, does a historical snapshot exist of other outages at a certain time of a major incident?

Crucially, the monitoring of downtime needs to record the delta in the time when the actual outage happened against when the outage was acknowledge and escalated to the responsible operator via a notification process. If there was an outage with no notification then this needs to appear in an exception report. Every network management product vendor bases their sales pitch on the fact that they will provide a faster detection time for an outage. This is absolute bull and the reason is that in most cases, when the downtime of an outage is analyzed, the detection time is the smallest contribution to the overall length of the outage downtime. Why would you buy a tool that's primary purpose is to address the smallest requirement?

The other misnomer is that a monitoring tool will somehow miraculously prevent an outage. The reality is that shit happens, and if your tool is only focused on prevention it will fail when you try to use it to cure. The value of a tool is how it deals with an outage because the reality is that this is the major factor in dealing with service improvement. Optimal management of outages results in two trends:

the time period between outages increases
the length of the outage reduces

There is no value in your monitoring tool if it cannot influence these trends and worse if it cannot even report or measure them.

The next step is diagnosis. Diagnosis monitoring is not that difficult. What often happens is that an outage occurs; a techie diagnoses it and after an investigation comes up with a solution. Often this knowledge remains in the techie’s head and is not annotated in a separate knowledge base. If this was consistently done, then the diagnoses time would reduce and immediately a tool that provided this ability would have a justified return on investment.

The Repair-recovery-restore time periods are not difficult to measure if a network management product is service aware, as these metric times will be obvious.

The output required is an analysis of the consequences of downtime:

what is the total outage time and what was the time period of degradation (brown outs versus black outs)
what is the deviation off the norm for this outage compared against the histrical trend
what are the average incident times for: detection, diagnosis, repair, recover and restore.
what are the top services affected by outages
what are the top causes of outages
what is the MTTR, MTBF and MTBSI?
what is the Manhattan analysis?

Network management needs to step up a gear from Red-Amber-Green, to different shades of grey.

A brand new bakkie

2010-05-20T19:06:00.000+02:00

Do you know that when a woman wears a leather dress, a man's heart beats quicker, his throat gets dry, he gets weak in the knees, and he begins to think irrationally???

Ever wonder why?

It's because she smells like a brand new BAKKIE!

(via amaree of KSS)

Thinking problem management!

Seeking to eliminate the "techie curse"

Documentation - the techie curse

Radwin video

Love of my life

Testing webradio

Africa's fibre routes

The final landing...

This day in history: Tranquility Base, the Eagle has landed!

Cellular providers in South Africa

Thinking networks

Happy Birthday, USA

1984 reunion

Ultrabroadband and QoS

Yeha Noha

Artic sounds

Neotel strategies

A foundation for RCA

Think Pink

wifi interference (with Wavion)

Ray and Callie

Radwin PtMP

Motorola Defy

How do you use OpenDNS?

Poll about LinkedIn

Sinclair ZX Spectrum

Remembering Challenger

Problem management on Practitioner Radio

Pingtest and Speedtest

Scanner on ABSA ATM

Ninja iPhone mode

Pingtest and Speedtest

Pingtest and Speedtest

Pingtest and Speedtest

Pingtest and Speedtest

Top content on TPM - 2010

Merry Christmas!

Christmas is cancelled

Christmas card string

Product amplifier

Smokeping

Pingtest results

Storm in Magaliesberg

Wireless

Speedtest and Pingtest results

South African weather photos

A egte South African tjorr

Wear sunscreen

Mamma Mia

Layer 2 data cente connectons

Updated results - Speedtest/Ptingtest

Speed and ping tests

Chicken Excellence Officer

Madge Networks Farm - August 1995

The Pareto Principle

Roald Amundsen

Another Madge photo from Woodside, August 1993

Henry Ford

Starbucks

Product management

Spud premeires on 3 December

My only friend through teenage nights

Pingtest and Speedtest

IP Addressing

Pingtest and Speedtest

Network management

SNMP Tutorial

Pingtest and Speedtest

Speedtest.net and pingtest.net results

Podcast on Internet Exchanges

MyBroadband conference

iBurst Technology video

W is for wifi

Interesting statistic

A ride on SpaceShipTwo

Top network problems

Tomato QoS example

Global MBB in Oslo

RADWIN Solutions - Streakwave Webinar

Climbing a tower