02 October 2020
Think about a car. A car has these functions: It moves the driver around, it moves passengers around, it moves things around. It can also carry things. It can carry clean things like books and files and food, and dirty things like potting soil and fertilizer. A car can also tow things. A car can also serve as meeting room, a private space, a place to keep you out of the rain or the cold. Most families would have one or maybe two cars and the cars are expected to do all of these things.
The sad truth is that when you drive your SUV to work alone in the morning you are taking with you some 1600 kgs of metal, rubber, plastic and glass. You are taking with you the space for four other people, and an empty trunk. You are also taking with you an engine that is powerful enough to go up to what maybe 200 kms an hour. It is all pretty silly. And there is something else, depending on the car you have it may not be very good at doing some of the things you need it for. If you need to carry a tonne of dirt or more than five people, the car you have may not be adequate.
What we really need is transportation in a variety of configurations.
On the weekends you might need a truck if you are taking stuff down to the tip, or you might need minibus if you are taking a big group of people to the football. During the working week you probably only need a small one-person vehicle that goes a maximum of 100kms per hour and keeps you warm and dry and fits into a tiny car park or even better folds away to be stored under your desk. You only really need a trunk if you are carrying things, and you only need passenger seating if you are carrying people. It would be good to be able to upgrade the boot or the seating without having to change the whole car.
In the software world a car can be seen as an application. It is the monolithic thing that you go out and buy in the hope that one day you will use all of its features and that you will not need something that it cannot do. Microservices tells us that this is no longer how we should be looking at software.
A car built on microservices principles would allow us to swap the seating space for a big cargo area on the weekends or when we are moving to a new house. It would allow us to add passenger space in accordance with the passengers we needed to carry and it would allow us to go to work using a small car with a top speed of 110 kms per hour without anycargo space or passenger space when we drove to work. This car would allow us to upgrade the cargo space on its own without needing to change the engine or the seating
The problem with microservices is that the various pieces that do the job do not just fall together as, and when, the need arises. To assemble the parts into a solution they must be clear, intuitive and well understood by everyone and the nuts and bolts that need to be tightened and the settings that need to be configured to ensure that everything works must be fool proof and easy to use.
It turns out that building a solution out of a bunch of dynamically configurable, separately upgradeable modules is not quite as easy as it should be. This is why a number of companies have failed to move into microservices despite the promise it offers. The bad news is that not every problem lends itself to microservices.
The good news however is that the data management problem really does lend itself to microservices. Data are alive and well and living in your organisation. No matter where you work you will have a variety of systems that gather and process data. Whether or not you know it those data are part of the data system of your organisation. Getting control of the data in your organisation is not a matter of replacing all of those systems with the one true system to rule them all. It is more a matter of learning to understand and publish where those data come from and where they are going.
Every data system in your organisation should be seen as a replaceable part of the data ecosystem of your organisation. If it is currently doing its job as it should then it should continue to do the job, at least until something better comes along. All you need to do is ensure that its inputs and outputs are understood, and that authorised people can find the data they need. When data systems need to be upgraded then they should be upgraded with the specific purpose in mind.
Datalign is a cloud-based serverless framework that takes control of the
data in your organisation. There is no need to remove the old trusty database that has been doing its job forever, instead it is registered as one of the “modules” performing a function in your organisation. The inputs and outputs are registered too, giving data managers an easy to use top down view of the data systems in place.
Data can be transformed, cleansed, manipulated and loaded into a variety of platforms and formatted as required using the Datalign toolbox or by using the tools your team are familiar with.
As a microservices framework Datalign does not dictate who makes the components that perform the functions, but it can provide feedback on whether the system is working as it should, whether the data is reaching its destination and whether people are using the data.
Written by Trevor Christie-Taylor
