Crisis time - an analysis

-

 Every software project has crisis now and then. It might be an error in production or last minute issue which is going to derail the plans. Crisis always leaves bitter taste in your mouth. 

T0: This is the time when issue is discovered. This is the time when you get that feeling in gut. Something is really wrong and you do not know what to do. It is the time when ... hits the fan. This phase may last for 1 hour to one day. Depends on the team environment and organizational culture. At the end of this phase team informs next level of management. 

T1: This is a messy phase. Now issue is escalated. People who do not know anything about the issue are involved and they try to gather information. This can also be called a stampede phase. It is a mad rush with lots of activity. Do this, try this, check this Or checklist followed, process followed etc etc. This is the phase which pulls away key players in different direction. They are not allowed to focus and find a resolution. Towards the end of this phase some one takes charge. If they are in position of power they will issue an order with clear directions to come out of the situation. If not in a position of power they will isolate from the stampede and focus on the issue. This phase can last from hours to days. 

T2: This phase information trickles in. Analysis is done why the issue happened. Doing a root cause analysis by 5 why is helpful. This leads towards the solution. Starting with the first why gives some direction for the team and brings in the focus needed to solve the issue. 

T3: This is the solution phase. Either issue is found or a workaround is decided. This starts the de-escalation of the crisis. 


A good leader takes team to T2 phase quickly. It will feel like crisis did not happen. 

Comments

Post a Comment

Popular posts from this blog

ChatGPT for cavemen

-
Image
I heard a lot about ChatGPT and thought to come out of my cave to find out what the hype is about. Is it going to eat all my lunch or eat me for lunch?  Like any product I gave it all my details and signed up. I asked it about quantum computing an it gave some answer. So far so good. Next day I wanted to check its blogging skills. So I opened chatGPT and I got this . I checked throughout the day and it is still down. So it won’t eat me till it is fed few more servers.  Continuing after it was up.  I asked it to write a blog about itself. Here is what was generated by chatGPT.
Read more

Greedy computing with API

-
Image
 We all know computing resources have a cost. Memory, long term storage, transportation and computing all have their own costs. There are lot of scenarios where your API end point won't be able to respond in time. These are scenarios where you need to use more resources to provide a response. Let us be greedy with our resources and explore different patterns which can solve this. Polling  This is a commonly used pattern where the caller initiates a process. Then keeps on checking if the process is complete. If the process is complete then the caller reads the data needed.  This is a good pattern when both systems are part of the same organization. At least developers of caller and callee should be able to discuss this implementation.  Call back This is an improvement over polling pattern. Here the caller supplies a call back URL to the long running process. Once the process is complete it calls the callback URL and informs the caller that process results are available now. 
Read more

Event driven architecture for micro services

-
This is a work in progress. I am designing an event driven architecture to be used in systems using micro services. Micro service architecture is popular because it solves few problems in development process. While it solves some problems it also brings in new problems to the table. Bad designs impact micro services harder than monoliths. But this is thought is for another time.  How micro services talk to each other. In a monolith you have one gigantic database, or you can easily call another component to process the information. In a microservice you can use a Que (or event bus) to pass the information. This is the problem I want to discuss. Main issue with que is it is not designed for this scenario. There are few products which give a good solution by providing a subscription-based approach. But these are large implementations. Using these is similar to bringing an excavator to plant a sapling in your garden. Yes, many people go with using these large implementations to resolve thi
Read more