At 9:13am she received a call from her operations manager about an issue with the job candidates database. Apparently, the system is not responding to users who try to access it using their web browsers. The problem allegedly started before 8am that day but nobody knew when exactly as there had not been any monitoring of its accessibility.
The CEO was strongly vocal about the issue as, in his words, “We are losing hundreds of thousands of pounds per day as our clients cannot post new jobs and review candidates. Fix it!” The head of marketing added fuel to the fire, “I have just launched new campaign to promote our system, and you are now not delivering on the accessibility of the portal as promised!!!” Sarah wanted to say something but let it go this time. She excused herself from the staff meeting and called her own staff to work on the issue at hand.
During the next 2 hours it became clear that the issue is not with the internal company systems but with the cloud service provider who had been hosting their HR servers for the past 2 years. Whilst Sarah’s company is responsible for the development of the application, the cloud provider hosts the servers, network connectivity and databases needed for the application to work properly.
To make matters worse, the cloud provider had gone into administration late last week, with all staff being dismissed by the new company administrators. Naturally, nobody bothered to inform “Jobs Are Us” about it.
A quick brainstorming session with the operational manager, chief technical architect and security manager revealed that:
1. There are no contingency plans that detail what to do if the cloud provider is not available
2. The backup of all systems data is hosted by the very same cloud provider, and the last offsite copy is some 6 months old
3. There is no one answering the phone in the cloud provider’s offices.
Suddenly, Sarah realized that this is probably a good time to freshen up her CV.
In the end, “Jobs Are Us” had to find another hosting company, restore data from 6 months old backup media and spend a considerable sum of money on the data retrieval exercise, laboriously going through individual recruiters’ mailboxes. The company’s reputation was damaged and few big clients walked away. And Sarah? She is now managing IT teams in another company...
By subscribing to our early morning news update, you will receive a daily digest of the latest security news published on Help Net Security.
With over 500 issues so far, reading our newsletter every Monday morning will keep you up-to-date with security risks out there.