The SSP disaster and what you need to know about uptime

Home > Insight > Blogs > The SSP disaster and what you need to know about uptime

29th November 2016

In August 2016, that is exactly what happened for around 1000 insurance brokerage firms using Software Solutions Partners

In August 2016, that is exactly what happened for around 1000 insurance brokerage firms using Software Solutions Partners (SSP) Worldwide’s Pure Broking platform, setting in motion a disastrous chain of events.

As reported in Insurance Age and Insurance Business, amongst others, for at least ten days, a power surge leading to a massive outage caused irreparable hardware damage to SSP’s SAN in Solihull.

When generators and other back-up systems also failed, brokers were unable to do any work at all, with some firms reporting weeks of disruption and lost earnings.

As Pure Broking was a cloud-based platform, many brokers hadn’t felt the need to back up their data elsewhere. Emails couldn’t be accessed neither could records, clients’ contact details or other services.

This impacted both renewals and finding quotes for new business. All they could do, for days on end, was twiddle their thumbs.

Most worryingly, perhaps, was that as a direct consequence of the outage, there were likely to be motorists unknowingly driving without car insurance. They couldn’t be contacted because their details were stored in the inaccessible SSP system.

Lenny Kerkhove, general insurance advisor at Warwick Davis in Sussex, said: “If someone needed to do a change of vehicle … or someone’s renewal was due … we couldn’t do it.”

Regarding the damage to his business, Kerkhove continued: “It’s a loss of business and obviously it reflects badly on us. They call us expecting a quote and we can’t do the very basic of offering them one.”

Richard Stevens, principal at Trelawney Insurance Solutions, talked about the ramifications of his inability to work for such an extended period of time.

Stevens explained that he and his colleagues expected to be able to catch up on a day or two’s missed work over a weekend. But the longevity of the outage meant the entire business was forced into inactivity, leaving them with a huge amount of backlog once the system was returned to a workable state.

He said: “It seems bizarre to me that their backup generator didn’t work. I would have thought that’s the only job of a backup generator – to not fail when it’s needed.”

 

The Fallout from a Failed Failover

For Insurance brokers such as Kerkhove and Stevens, the fallout from the SSP outage meant expensive lost business and a drawn out recovery time, where catching up with the backlog slowed down processes and impacted performance and reputation.

For SSP, it has meant a huge back-up operation retrieving data for their clients, crisis-managing an avalanche of bad press, permanently closing its Solihull data centre, prematurely relocating to west London and now dealing with repercussions from the FCA and potentially enormous client compensation pay outs.

Whether SSP will retain its UK client base when the dust has finally settled is certainly debateable. There are many questions being asked through the Insurance sector:

How could this have happened? What happened to the second data centre cited in SSP sales material that would ensure a speedy recovery? How could the failover system have failed so dramatically?

While some of these questions are already being answered, for the brokers concerned it is now a case of recovery and damage limitation on a grand scale.

Stan Crosby, managing director at Crosby Insurance, said: “It’s a lamentable state of affairs … We just have to take it and vote with our feet when the contract is due for renewal.”

It is completely understandable that the insurance world is irate over the SSP outage: if it can happen to this, apparently reputable SaaS provider, surely it could happen anywhere?

The answer is yes, power outages can happen anywhere and to any business. It’s how they are anticipated using planned failover systems, disaster recovery and business continuity measures that count.

It’s very unfortunate that so many brokers are asking pertinent questions about what could have been done to minimise this disaster now, when it’s already too late.

 

Disaster Recovery and Business Continuity: Don’t Wait til it’s Too Late

Uptime is the key consideration of any business reliant on data for any of its core processes or customer data storage.

The infrastructure and standards in place to maintain uptime should be your key focus when it comes to your choice of data centre.

Focusing on due diligence, realistic risk and responsibility levels and complete dissection of contractual detail is where you should always start.

 

1. Don’t assume a big name is a sure thing

Assumption is Kryptonite to avoiding risk.

It’s incredibly common for businesses to sign IT contracts based on reputation and client list, without employing proper due diligence.

This is particularly true when the vendor is a big industry name and other, similar businesses are already signed up to their products or services.

However, as the SSP case unfortunately demonstrates, supplier ubiquity or prominence doesn’t necessarily equate to resilience.

According to SSP figures, 40% of UK brokers rely on its systems to track insurance renewals, source quotes and purchase products from third parties. It’s Pure Broking cloud-based platform, used by up to 1000 brokers, was the one affected by the outage.

When everything was working normally, SSP was an industry favourite. When the lights went out at Solihill, however, they could have taken half of that industry with them.

2. Be realistic about your risk and responsibility levels

Understanding your risks, their potential impact on your business and how you can minimise them is key.

While it’s tempting to want to pass over total responsibility for risk management to a third party, you will always remain responsible on some level.

Disaster recovery and business continuity, as SSP customers discovered so dramatically, is not always provided to the levels you expect. Even with the most advanced data centres, there will always be a low level of risk.

Acknowledging unavoidable risks, particularly any single points of failure, is the first step in understanding their potential impact on your business. Only then can you ensure the right responses are in place as part of your DR protocol.

Key risk mitigation elements to look for in a data centre are dual-powered equipment, effective distance between primary and secondary sites, multiple independent distribution paths serving the IT equipment and at least 99.9% availability.

Many brokers assumed that, as the Pure Broking platform is cloud-based, they didn’t need to back up their data. Unfortunately, they learned the hard way that this was not the case.

It’s always a good idea to have a secondary backup plan, for rather obvious reasons that conjure images of eggs and baskets.

3. Take expert advice if you’re not sure about any contractual points

Data storage and management is a serious business. In fact, it is your business; don’t gamble with it by signing a contract you don’t fully understand.

Realising that you can’t abdicate total responsibility to a third party for the safety of your data is vital; you have to be realistic about what a large scale data outage could mean for your business and ideally set contingency budget aside in advance.  

Following the August outage, credit card insurer, CPP Group announced it will spend £6.5million developing its own IT platform rather than continue using SSP as a partner. This on top of an undisclosed cash payment to end its contract with CPP.

Other brokers are facing shortfalls of up to £100,000 as a result of the loss of power, with little hope of receiving compensation any time soon and only an expensive exit option from their contracts.

Lessons Learned

While the business world watches the fallout from the SSP outage, it would be foolish not to learn as much as possible about why this happened, how damage could have been minimised, why the disaster was so extreme and how effected businesses will recover.

Through better due diligence, risk management and clear contractual terms, you can work more effectively with data centres to maximise uptime and minimise disruption. It’s too late to mitigate the impact of the SSP disaster. But by learning valuable lessons, we can certainly take steps to make sure it doesn’t happen again.

If you would like to find out more about Sovereign please get in touch.

You may also like

21st January 2016

About Us

Technology is continually advancing and transforming the way you work. Businesses need IT to react quickly to change and help drive opportunity. Sovereign can help make it happen.