Why Don’t CIOs Enforce Rigorous Performance and Stress Testing BEFORE New Systems Go Live?

I recently spoke at the Next Generation Telecom conference in Rio de Janeiro, Brazil.  This blog captures a few key themes from my speech and highlights some innovative work my team is doing in helping telcos deal with exponential customer growth.

But first, a quick sidebar to draw out parallels between musical rehearsals and the need for IT to perform rigorous, multi-faceted testing for new systems or software releases before they go live.

Every musical group – rock bands, church choirs, heavy metal groups, classical orchestra – has rehearsals before they perform in public.

Why?  The answers and the lessons learned for IT are as follows:

1) Get the basics right. Each individual musician practices on their own – to first be proficient in their part. In a choir or an orchestra, groups of similar instruments (basses, sopranos, violins, cellos, etc.) have ‘sectional’ rehearsals to practice and get used to performing as a group. In an IT environment, this is functional testing.

2) Get the group to ‘be in sync’ with each other.  The group practices starting and finishing together, handoffs of the main melodic phrases from one musician to another, and general musicianship. For new IT systems, this is End-to-End testing.

3) Obtain input on the performance.  Sometimes, the group invites a ‘friendly’ audience to a rehearsal – in the garage or rehearsal area.  This helps the group obtain feedback regarding the musicianship and overall aura of the performance.  An IT analogy is Friendly User Testing.

4) Schedule a Technical and a Dress Rehearsal in the Actual Venue. A Technical rehearsal is a complete run through of the performance to check sound levels, lighting, and performance space for the musicians. A Dress rehearsal is a repeat of the Technical rehearsal, with performer wearing the exact clothes to be worn for the performance. The IT parallel is User Acceptance Testing, Performance Testing and Stress Testing.

Why go through all the effort for Technical and Dress Rehearsals?  Just Google ‘wardrobe malfunction concert’ and you get 3 million hits… everything from pop stars and TV performers all the way to Janet Jackson’s infamous Super Bowl 2004 incident.

But due to a variety of pressures (schedule being the most frequently cited), many IT organizations fall short in testing systems. They perform functional, end-to-end, user friendly testing – and then skimp on user acceptance testing.  Performance and Stress Testing is a rare occurrence… with the IT philosophy succumbing to the approach “make it work first, then make it work fast later”.  This philosophy is also quickly followed by another faulty strategy: “if there are performance problems, we can always throw hardware at it”.  The old adage “pay me now or pay me later” certainly applies to testing.

About the Author: Laddie Suk

Laddie leads a cross-functional Dell Technologies Consulting team focused on digital transformation and industry solutions. He is a seasoned industry veteran with deep experience across multiple industries, solutions, and technologies. As a former Verizon Network CIO and Network Executive at AT&T and Bell Labs, he has extensive hands-on experience in leading strategic network and IT development projects and managing communication service provider environments. He has also led strategic and tactical engagements in network transformation, IT transformation, and business process and performance improvement for clients throughout the Americas.