Moving a Monolith to Micro Service

The Beginning

Creating a web based application has it’s own challenges but the ones that need to be scaled up to almost infinite audience needs to be designed from ground up to be scaleable — or so I thought till I was left with the Baton for a product created 20 years ago which needed to be modernised fast with the minimum cost and minimum go to market time.

So there I was trying to understand, how to renew an ageing product and trying to see how it has been adapting to the evolving world around it. Numerous discussions with various stake holders including Customers, Project Managers, Architects and Developers all led to the same conclusion — the product has not required to be scaled on a global level as we never had more than 120 licensed corporate customers. It’s not a B2C product and thus scaling costs were born by the organisations where the product was deployed and managed. Product maintenance costs were still additional revenue stream for our organisation.

The change that the organisation wanted was to move away from the current business model into a SaaS based business model with revenue stream recognition via monthly subscription costs in the B2B space — the only targeted consumer space where our product made sense and some money.

The Preparation

Going over the critical paths in the workflows and the in-demand or most used features of the product; the team was able to identify the core parts of the system which were essential to be moved out of the ageing product in order to at least have the core systems functional in the rewritten product. We had a prototype of separated standalone code that could work as a disparate system component within 3 months of the kick-off. Now we needed to figure out how to move a behemoth into more manageable chunks and create a distributed system.

The major challenge here was to change the mindset. We were still thinking about just breaking things apart into smaller chunks — but not about making the chunks integrate together and work as a distributed system. Disparate, Smaller chunks of sub-systems but integrated together to work as a single distributed system. That’s something I would like to change the next time I am doing this again.

So, the plan was to have a singular database system and a single DAO layer which would be able to easily scale up on-demand. Interfaced with this single DAO layer would be the various middle-ware sub-systems that interact with each other via message passing over HTTP. We also wanted to rewrite the Integrated UI layer as a SPA which would be easily convertible to mobile applications — if needed.

The Design

We split the sub-systems based on functionality and converted the monolith’s modules into individual micro-services. This was a mistake and I would request people doing this today; to think really hard whether you want to really just convert the modules into micro-services.

This are the micro-services that we planned first:

1. Authentication & Authorisation
2. Order type 1 processor
3. Order type 2 processor
4. Order type 3 processor
5. Configurable Content system
6. User Preferences system
7. Batch processor system
8. Administrator system

We ended up with the following:

1. Authentication & Authorisation service
2. User Preferences, Customer Preferences and Feature enabling service
3. Data Validation and Ingestion service
4. Order Orchestrator service
5. Workflow service
6. Batch Processor service
7. Data processor service
8. Business data analytics service
9. Business data Audit Service
10. Microservice monitoring service
11. Log data analysis + system data analysis service
12. Configuration management service
13. API Gateway service
14. Developer Portal service
15. Metering and billing service

The Learnings

What we had not planned for was scaling a distributed system is really a different ball game compared to scaling a monolith. We had to learn the hard way that performance testing, BDD and TDD are all required to save time, money and effort. Also, when modernising a similar application, please consider the additional effort that would be required to manage containers and multiple clusters of micro services. The IT Operations team needs to be skilled to manage the individual service clusters. Also, the monitoring and management of distributed system is more complex compared to a single cluster of monolithic applications.

Hope to have helped you design better Micro Service based distributed systems when you start with an ageing monolith. I would really like to hear your experience and improve this article to be helpful for others — hence please comment. :)