Scrum in Action (behind the Scene)

After going through 4 projects using Scrum as the software development methodology, my experience about Scrum can be summarized as follows:

Scrum helps to lower the project's risks, bring happiness to customers and improve team collaboration.

One of the greatest things when applying Scrum is that it can be applied without even knowing that it has been applied in a project and still have all the nice outcomes that Scrum could bring into a project.

Below is a use case of Scrum in a six-months project that my Team and I have delivered to one of the largest banks in China.

Use Case

We have being working with the bank to draft and define precise requirements for a new hedging fund system for 2 weeks, however the requirement documentation was still very vague that we cannot estimate the man-days required to deliver the project. Nevertheless, the customer asked us to sign a contract that bounded us to agree on delivering a system as described in the requirement documentation. After a few days of negotiation, the customer agreed that they would play "fair" to us because they knew that the requirement documentation was not up to the point that a vendor could develop a system out of it. We took the risk by signing the contract with the guesstimated man-days and launched our quest to this mysterious land.

I was leading a team of 3 to deliver the system. I began by using the traditional waterfall development process to guide the other 2 team members to deliver the well-defined component of the system. After the evaluation of the scope of the component, I expected that it could be delivered in 2 weeks. With the support of my team, we achieved the first goal of the project before the Christmas time. Customer and we were very happy about the outcome but the challenge didn't start to crop up until the next stage.

The second stage of the project was chaotic. We continued to apply the waterfall development process during the beginning of the second stage of the project but we soon found out that it wasn't appropriated. This was due to the fact that the requirement was so vague that we couldn't proceed any further. I started to call for requirement meetings with the customer and told them that we needed to collect more requirements before we could proceed. However, the customer was not clear about what they wanted also since they were not the end-user of the system. I immediately knew that problems were coming and I turned to my manager to ask for his advice. My manager really wanted to bring the customer to the house, so he asked the team to try their best to help the customer to succeed. This was the turning point of the project. I knew the fact that we were not going to deliver the right system in the first attempt, therefore I had to deliver it in batches and adapt the system overtime during the remaining period of the project. Luckily, we were working onsite and therefore we had direct access to the customer and the end-user to collect user requirements. This was when I started to think about using Scrum for the second stage of the project.

Applying Scrum is straightforward if the team members already know about the philosophy, the roles and the practices of Scrum. However, my team members had no past experience on Scrum. Therefore, I had to tweak the game just enough to get it going. I started to change my way of leading the project. I wrote the product backlog for the project and prioritized the user story in the backlog. Also, instead of managing the project like before, I started to hold daily meeting with them to discuss about obstacles they were facing when designing the system. Casually, in a daily basis, I would inquire them about what they did yesterday and what they were planning to do in the next few days. It was just very natural for senior engineers to have a clear picture of what they were working on and what was next things to do so my conversation with them didn't annoy them too much. However, the team members expressed that they were quite uncomfortable of this project because they didn't know if the project would ever end. I told them we would going to have bi-weekly meeting with the end-user until the end of the project in order to make sure that the system we developed was what the end-user wanted. However, this required us to prepare for demo every 2 weeks. They agreed on it and we proceeded. It was true that for the first few bi-weekly meetings, the end-users made a lot of changes in their requirement documentation. But after they saw the system 2-3 times, they started to know what were MUST in their system and what were just NICE-TO-HAVE. I negotiated with the customer saying that if our common goal was to deliver the project on-time we needed to focus on items that were MUST have. Since the customer knew that the NICE-TO-HAVE functionality could be developed by themselves after the foundation was completed, they agreed to take the risk.

The Scrum-behind-the-scene continued until the end of the project. The backlog was updated and prioritized everyday to reflect the changes. We continued the 2 weeks development iteration and bi-weekly meeting until the end of the agreed duration of the contract. The system was delivered on-time and the customer was very happy with it.

For the past 2 months, the system was working day and night to process over 200000 jobs for their pricing and hedging computations and no single bug was found for the past 2 months. The benefits for the company and the customer are clear but the most important thing is that the team is proud of their work. The efficiency and productivity of the team have been improved after the project and we get to know better each other after the intense collaboration that was required in the project.

Thanks to Scrum, It brings order from chaos.

How to be an Engineer, IMHO (Basic Part I)

My favorite quote in my engineering career is "Asking good questions is the first step to solve great problems". One of the interesting characteristics of an engineer is to ask a lot of questions. A good engineer selects a subset of those questions that are most relevant to the problem to be solved. So, what are good questions? IMHO, there is no definition for it because it depends on what approach you choose to solve the problem. Different approach has different questions to ask to meet different objective.

When a person is facing a problem, one of the sensible approaches for problem solving is by "searching" for solutions. This approach works fine when the problem has to be solved in a timely manner (i.e. within minutes or days) because the quality of the solution is a ratio of K * (Quality of the Solution)/(Time taken to solve). The approach makes use of past experiences as the initialization points of a solution set. Solutions that are closed to the initialization points might also be included into the final solution set. To solve the problem, pick one of solutions from the solution set that can maximize the objective.

There is an assumption in the above approach that usually goes unnoticed and that is the knowledge about the problem at hand. For instance, your boss says "We have a problem of not meeting the deadlines aggressively in the team". One might attempt to solve the problem by using the approach above. If it works fine every time, then I wouldn't blog about it :). But what if this approach fails even you and your team are trying very hard? Have you ever wondered what is the cause of the failure and explore other approaches to solve this problem?

Another approach for problem solving that I always keep in my solution bag is to redefine the problem. It goes a little bit deeper than what it sounds like. It uses common senses as the starting point of the solution set and reshape the problem by asking relevant questions. Common sense is an interesting thing; it means "what people in common would agree on". The actual definition of common sense is not very important for this approach to work well. Common sense serves primarily as an initialization tool; it allows us to look at the problem from many different perspective. Asking good questions to reshape the problem can lead to what many people called "thinking outside of the box". This is trivial using this approach because there is no box to begin with. The questions are aimed to elicit the real problem without introducing a priori assumptions about the problem at hand. Using this approach might simplify the problem tremendously especially for problems that are ill-defined.

The first two approaches might sound common to many problem solvers. The third approach that I will also consider is to use the first and second approach altogether. The problem with the second approach is that it might take very long to obtain a relatively good solution and it requires a lot of skills in order to ask those "good" questions. The third approach starts by using the first approach to acquire experience about the problem and then it uses the second approach to fine-tune the problem. Once the problem is understood better, then the first approach can be used to solve the fine-tuned problem. This process continues until a solution is found. The third approach requires an unlearning process when switching between the first approach and the second approach. Also, it requires a lot of adaptations during the process of problem solving.

Which approach you choose to solve a problem depends on what problem you are facing. However, there are some general rules to follow which I might blog about it in the future post :)

References

- How to be an Engineer, IMHO (Introduction)

My Thought on Extreme Transaction Processing (XTP) in Financial Services

Today, I have a discussion with a customer through emails regarding Extreme Transaction Processing (XTP) system for Order Management System (OMS) using GigaSpaces XAP. I found the discussion interesting and would like to also share it with you all. The discussion was about measuring system performance in XTP which usually involves Latency and Throughput. Here is the detail in the email:

Although latency is an important performance metric for an OMS, we also need to consider other aspects such as the system throughput. Measuring independently the latency and the throughput using different test cases will not reflect the performance of the system in the real world as the test case might aim to optimize the performance of a specific system parameter (in this case the latency) by trading off other important aspects of performance (such as the throughput and the ACID properties). In optimization theory, the optimization complexity increases with the number of variable in the system. In the real-world application, this complexity arises frequently and GigaSpaces does a good job, if not the best, to solve this problem.

We believe that the single-threaded approach to optimize out the object locking in order to shave off any possible latency will actually impact the throughput of the system as this approach can only update ONE order information at a time; limiting the system concurrency and utilization. Also, the single-threaded approach DOES NOT satisfy all the ACID properties even in this simple test case which affects the system reliability.

In fact, GigaSpaces can achieve the same latency using the single-threaded approach (i.e. polling container with single consumer in GigaSpaces). It can achieve even lower latency using embedded space. Also, in addition to the single-threaded approach, GigaSpaces provides standard transaction support. It has several implementation of Spring's PlatformTransactionManager which allow user to utilize Spring's rich support for declarative transaction management (which is reliable and standard), instead of coding an in-house transaction manager which might be error-prone and complex.

In reality, there will be more than 1 application using the XTP. The denominator for all these applications is the same which boils down to data consistency. It is very easy to achieve weak consistency in which many XTP solutions can/only provide. However, for some applications, strong consistency is a must. Therefore, we need to evaluate an enterprise XTP solution from a broader perspective and how much flexibility that it can provide in order to achieve the desired performance, manageability and security. When the usage of a XTP solution is beyond the basic and many different applications rely on it, these functionalities are not just "nice-to-have" but essential foundations for any enterprise application.

At the end of the day, what we would like to achieve in Extreme Transaction Processing is to keep latency low and throughput high while the processing (the business logic) is done transactionally (i.e. to provide ACID guarantee). Therefore, the performance of a XTP should be measured against all 3 properties as a whole in order to get a better idea of the capability of the solution.