This post is an example of corporate disaster and two examples of leadership related to it.
I did work as admin for development tools and version control in one of my earlier employers. The new trendy name for the job would probably be devops.
We had huge disaster with our version control tool.
Sysadmin had made major mistake when they allocated more disk space and it looked like we lost all source code.
Timing was extremely bad since the spring had been busy with several major projects for main products.
You could ask where is the backup?
There was good regular backup that was taken every day and it had passed successfully, but since it was never tested, it turned out that the backup was empty. It had passed successfully, but no data was backed up.
It looked like several months of work is gone and the future maintenance of the products would need some magic.
Situation was really bad. To me it looked like that if the data are not recovered, it will affect the financial result of the company and for stock exchange company it would mean stock exchange announcement.
With their experience sysadmins took the responsibility and they were able to recover data as data, but to the version control system data looked completely corrupted.
I was against an impossible task. I needed to figure out some way to get the corrupted data in shape. Of course I did not have any experience. You don’t deal with corrupted data every day.
Now you will see two leadership lessons – One for good and one for bad leadership.
Good leadership lesson
New CEO was recently nominated after Vice president had run the company already several months.
I was in the machine room with sysadmins to figure out some way to get the corrupted data in shape and something unexpected happened.
Vice president of the company arrived to the machine room and sat next to me.
He asked calmly “What do you think, are we able to recover”.
I answered, “I don’t know sir, but I am sure I will figure out some way…”
I am still today impressed. He did really see what was the most important thing at that moment for the company and future of it. With his calm presence he was able to put power to me, young technical guy.
This was the example of the “good leadership”.
Now comes the example of the “bad leadership”.
Click here to find out how you can become a better leader.
Bad leadership lesson
While we were with Vice president figuring out how we could survive, the rest of the company was having corporate party where the CEO was present.
Having a party or attending it was not such a bad thing. They could not work without the source code. What the CEO did was really bad judgment of the situation.
He made jokes of the vice president sitting in the machine room with us who were working to get the solution. Luckily we were not listening the speeches in the party, since we had hands full of problems, otherwise it would have been extremely unmotivating.
What did happen to the data?
After few hours we had recovered some of the data as proof of concept and we could test how does it look and it looked good. With additional manual work we were able to get most of the source code back in use.
I was never contacted by the CEO. I did never hear from him. I never received comment or thank you of the work we did.
That was first of his leadership mistakes, but not the last one.
My boss and vice president thanked me several times after that and I felt respected.
After several years from the incident I still remember the good and bad example. It was one of the toughest technical situations I have had.
Since then I have faced several smaller scale catastrophes. There are always few people who take the responsibility and leadership. They are always the same people, often too little rewarded.
Think about it. Next time your company or team is in the trouble.
Who are the persons who stand with you?
Have you rewarded them ? Have you even said thank you?
Photo credit Flickr and klaserfilms