Many years have passed since the dawn of modern computing broke … and with it many batch processes have been spawned and continue to survive.

Like a plague of locusts.

It is time for software engineers the world over- I am talking about you over there in the corner pretending not to be noticed with your geek sensibility and painful shyness – yes you, to stand up, take a deep breath and open your eyes to the modern world of event driven architecture or EDA.

As life becomes increasingly recorded by Big Data miners and such like, the sheer mass of data being collected has gone from a light breeze of disparate facts to a veritable  snow storm of related information.

Classic data handling eco-systems accumulate data at one point in the application constellation and greedily hog that data from downstream systems, until the clock strikes the magic hour from Whence The Batch Load Begins.

Given that batch loading – moving a large chunk of data, usually relating to a period/epoch whatever – was invented in the 1970s, it is no surprise that modern information demands are straining this paradigm beyond its limits.

Read as this:  some clients experience batch loading scenarios where their batch load for the previous day takes 25 hours to process.

Hmm. not sustainable. Some would say “Suboptimal.”

But make the batch run faster, I hear you cry.  I can assure you that many have tried and partially succeeded in doing so , but really have only kicked the can of catastrophe down the road 6 months or so.  This remediation is akin to pushing runny stuff up a hill with a garden rake. Eventually you are back to where you started, only sweatier.

The solution is before us and some would be surprised to learn, always has been. Simply take a look at Nature. All bio-systems appear to exhibit near real-time interactions.  The dog sees the cat, the cat the dog.  The cat runs. The dog runs and, well, Nature takes its course.

Boffins would call this “cause and effect”.  I call this event based behaviour.  A Thing happens which triggers another Thing to happen and so on. Much like what our application sub systems need to do with one another.

Our humble application constellation needs sub system A to tell sub system B (eventually) about that Thing That Happened.  Why not pass on the Thing straight away? Why wait, holding all the cards to your chest, sub system A?

There is no logical reason.  However, there are plenty of other reasons, such as legacy methods, sunken investment, complexity of the batch process and the like.  But under close scrutiny these arguments don’t hold water and are actually just excuses born from Fear and Loathing.

Migrating from batch to an EDA has a number of advantages. There are more, but for the sake of brevity I have listed the main advantages here:

•     Liveliness appears within the application constellation for free.  This is an excellent thing to market to your information partners as you can provide  real-time services – reports become heatmaps; changes to the nature of the data can be detected as they happen, great for limit, fraud, regulatory or breach detection.
•     Gone is the need for heavy, complicated data-loaders,  staging ares and massive files which corrupt easily.  Data comes over the wire in a nice steady stream of bite-sized (sic!) chunks which flow river-like downstream to all relevant sub systems.
•     The architecture is simpler to manage: you don’t suffer an angry business partner when your EOD batch fails 9 hours into the process and you have to take the whole day to recover.  If there is an outage systems only need to recover from the point in time when the outage occurred, giving the opportunity to fall back on manual systems while recovery is underway.  The missing dataset is by definition smaller.
•     File corruption becomes a thing of the past.
•     All sub systems involved in the event flow can be made aware of each other’s health through judicious use of heart-beating or request-response paradigms.
•     Monitoring UI’s can be put in place to show what is happening where and when. This also serves as an alerting platform.
•     Optionally a smart routing system can be deployed to filter, wire-tap or simply forward messages to whoever is interested.

It is remarkably simple to establish a common interface language between systems.  There are many existing standards already in place that make just this so simple. There are many messaging infrastructure solutions available  – XML e.g. FPML, Fixed Length e.g. SWIFT/BACS, Protocols such as FIX.  Doubtless one of these will suit your budget and complexity.

The main  questions that remain then are:

1. do we need durable messaging?  If a subsystem goes offline should it be able to publish/consume waiting messages? If so, use a Point to Point messaging technology. If not, then use the broadcast radio-style Publish/Subscribe methodology where another event just like the last one will be coming along soon enough, so don’t worry.
2.  how we fast do we need to push events about the place?  Most systems accept latencies of around 1 second.  Do bear in mind that when you start asking for more speed, the cost of messaging goes up dramatically
3.  should we be able to handle receiving duplicated messages?  Generally the answer is a resounding Yes!
4.  should we be able to get source subsystems to resend the same message one demand? This relates to point 3 above and is also usually Yes! This feature is invaluable should the unfortunate happen, you loose connectivity and need to recover a missing message or two.
5.  how much will this cost?  My experience is that the maintenance costs of batch loaders are similar to the development costs of an EDA, but once running the EDA is so much more flexible and has a much lower cost of ownership
6.   would we benefit from a simple EDA routing engine where I can pass the same message to many places at once? Or pass on only certain messages to certain subsystems.  Most cases again the answer is Yes!  Doing so brings in an intelligence into your system never before seen.

The trick is to start small, pick on a key data flow and migrate that first, preferably with the inclusion of a routing broker.  Once you have achieved the first EDA flow, you will doubtless be eager for more.

Lastly a word on brokers.  I have used Complex Event Processors very successfully as routing brokers to support an EDA in small and large Enterprise contexts.  They are invaluable. Based on your budget I would recommend that you look at Spring Integration, Esper and Oracle Event Processing.  Other solutions are available 😉

Happy eventing!