High Performance Computing with RESTFul Excel in Financial Services

If you have ever worked for financial services to develop systems for their front office, you know that traders love MS Excel. There are a lot of reasons to use Excel as the viewer; manipulating and visualizing data in a grid fashion are both sound and logical. Still, computations that require more than 1 computer could handle are better to offload to a grid computing platform and that's what we have done for one of the largest banks in China.

Today, I'm not going to talk about the grid computing platform we've modernized for the bank, instead I will focus on version control for MS Excel. You might think that is easy because you can use Sharepoint. You are absolutely right and that's what we have selected as a technology to manage different version of the Excel. However, this is only part of the story. There is another part of the story which is constantly being overlooked and that is to manage the different version of "the data" in the Excel.

Version control of data in the Excel is a problem that even Microsoft leaves the door opened for others to contribute because it requires the domain knowledge of the field (in this case, the financial services) in order to understand what the data is about to version control. In the financial services, the quants build their pricing models into the Excel spreadsheet. The pricing models are mathematical models that required model parameters and some predefined inputs in order to calculate the outputs for the pricing models. Once the quants have validated the models, the Excel spreadsheet is hands-off to the traders and the sales for perform their daily operations. Problem is that when the quants update the pricing models with new parameters and new inputs, all of a sudden the traders and the sales are facing a difficult problem; all the current deals that are made in the past using the old models are needed to port to the new version of the Excel spreadsheet. This porting activity include the inputs, the model parameters and the outputs of the model, PLUS all the information about the deal. They need a flexible way such that they can use the new Excel spreadsheet with the data that is in the old version of the Excel. The solution we have proposed and implemented uses RESTFul services to facilitate the version control and migration of data in the Excel spreadsheet.

RESTful services are great for system integration. As discussed in this presentation, it allows each system to upgrade at its own pace without concerning the other depending and dependent systems as long as the upgraded system maintains the older versions of the services. In our case, the data in the Excel is stored in GigaSpaces which provides real-time risk hedging functionalities based on the market changes. We developed a XML schema for the data and represented the data in XML format so that all client applications know exactly what is in the XML and how to use it (we build a library that uses XPath to get the data we need from the XML and populate the Excel spreadsheet on-the-fly) . If the new version of the Excel spreadsheet does not required new inputs and model parameters, the traders and the sales can benefit from it immediately. If the new version of the Excel spreadsheet requires new inputs and new parameters, the quants will need to add new entries in the XML to describe the new data. When the traders and the sales open the new spreadsheet, the data in the old version of the Excel will be filled-in to the new version of the Excel spreadsheet. After the traders and the sales have filled-in the new data and submit the new version of the data back to the data grid, everything is up and running again. This is just one of the use cases for the RESTful Excel but you can imagine that there are other use cases that can make good use of this technology.

Bear in mind that system integration is crucial in financial services and therefore, having a good technology that can facilitate system integration will allow the banks to adopt new technologies much faster which might improve their system reliability and performance. It has a direct impact to everyone's life (assuming you also put your money in a bank).

Repost: Virtual Identity v.s. Real Identity (dated on the 25th February 2007)

This is a post that I wrote 2 years ago on my previous blog hosted on a different site. I'm going to repost it here because the topic becomes even more interesting than 2 years ago and I think it will become the topic for the future as well.

Living in a virtual world with a virtual identity allows a person to explore his/her life in a totally different way than in the reality. Moreover, the person can have multiple identities in the virtual world that can be completely different than what he/she actually is in reality. With the possibility of multiple instances of the same person with possible different personalities, the security in the virtual world is in doubt. How trustworthy is a person who you can only interact in the virtual world? It will be desirable to enforce the virtual identity, to some extents, closer to the real identity. Current technologies such as username and password are not enough. What seems to be missing is a physical linkage between the real person and his/her virtual counterpart. This physical linkage is difficult to establish if we are allowed to have multiple instances of the same person in the virtual world. Therefore, if the goal is to match the virtual identity closer to the real identity, it will be sensible to find a solution that can limit one virtual identity per person to begin with.

Scrum in Action (behind the Scene)

After going through 4 projects using Scrum as the software development methodology, my experience about Scrum can be summarized as follows:

Scrum helps to lower the project's risks, bring happiness to customers and improve team collaboration.

One of the greatest things when applying Scrum is that it can be applied without even knowing that it has been applied in a project and still have all the nice outcomes that Scrum could bring into a project.

Below is a use case of Scrum in a six-months project that my Team and I have delivered to one of the largest banks in China.

Use Case

We have being working with the bank to draft and define precise requirements for a new hedging fund system for 2 weeks, however the requirement documentation was still very vague that we cannot estimate the man-days required to deliver the project. Nevertheless, the customer asked us to sign a contract that bounded us to agree on delivering a system as described in the requirement documentation. After a few days of negotiation, the customer agreed that they would play "fair" to us because they knew that the requirement documentation was not up to the point that a vendor could develop a system out of it. We took the risk by signing the contract with the guesstimated man-days and launched our quest to this mysterious land.

I was leading a team of 3 to deliver the system. I began by using the traditional waterfall development process to guide the other 2 team members to deliver the well-defined component of the system. After the evaluation of the scope of the component, I expected that it could be delivered in 2 weeks. With the support of my team, we achieved the first goal of the project before the Christmas time. Customer and we were very happy about the outcome but the challenge didn't start to crop up until the next stage.

The second stage of the project was chaotic. We continued to apply the waterfall development process during the beginning of the second stage of the project but we soon found out that it wasn't appropriated. This was due to the fact that the requirement was so vague that we couldn't proceed any further. I started to call for requirement meetings with the customer and told them that we needed to collect more requirements before we could proceed. However, the customer was not clear about what they wanted also since they were not the end-user of the system. I immediately knew that problems were coming and I turned to my manager to ask for his advice. My manager really wanted to bring the customer to the house, so he asked the team to try their best to help the customer to succeed. This was the turning point of the project. I knew the fact that we were not going to deliver the right system in the first attempt, therefore I had to deliver it in batches and adapt the system overtime during the remaining period of the project. Luckily, we were working onsite and therefore we had direct access to the customer and the end-user to collect user requirements. This was when I started to think about using Scrum for the second stage of the project.

Applying Scrum is straightforward if the team members already know about the philosophy, the roles and the practices of Scrum. However, my team members had no past experience on Scrum. Therefore, I had to tweak the game just enough to get it going. I started to change my way of leading the project. I wrote the product backlog for the project and prioritized the user story in the backlog. Also, instead of managing the project like before, I started to hold daily meeting with them to discuss about obstacles they were facing when designing the system. Casually, in a daily basis, I would inquire them about what they did yesterday and what they were planning to do in the next few days. It was just very natural for senior engineers to have a clear picture of what they were working on and what was next things to do so my conversation with them didn't annoy them too much. However, the team members expressed that they were quite uncomfortable of this project because they didn't know if the project would ever end. I told them we would going to have bi-weekly meeting with the end-user until the end of the project in order to make sure that the system we developed was what the end-user wanted. However, this required us to prepare for demo every 2 weeks. They agreed on it and we proceeded. It was true that for the first few bi-weekly meetings, the end-users made a lot of changes in their requirement documentation. But after they saw the system 2-3 times, they started to know what were MUST in their system and what were just NICE-TO-HAVE. I negotiated with the customer saying that if our common goal was to deliver the project on-time we needed to focus on items that were MUST have. Since the customer knew that the NICE-TO-HAVE functionality could be developed by themselves after the foundation was completed, they agreed to take the risk.

The Scrum-behind-the-scene continued until the end of the project. The backlog was updated and prioritized everyday to reflect the changes. We continued the 2 weeks development iteration and bi-weekly meeting until the end of the agreed duration of the contract. The system was delivered on-time and the customer was very happy with it.

For the past 2 months, the system was working day and night to process over 200000 jobs for their pricing and hedging computations and no single bug was found for the past 2 months. The benefits for the company and the customer are clear but the most important thing is that the team is proud of their work. The efficiency and productivity of the team have been improved after the project and we get to know better each other after the intense collaboration that was required in the project.

Thanks to Scrum, It brings order from chaos.