Category Archives: Server Side

Server side notes on Java programming

Java, Server Side

How to unit test CQL statements in an Oracle Complex Event Processor version 11.1

Although the Oracle Complex Event Processor version 11.1 is a very powerful  tool, it doesn’t come with a mechanism for testing CQL statements in isolation to the rest of the application.

A while ago a client wanted me to implement an application to leverage the CQL language shipped with the Oracle Complex Event processor version 11.1.1.7 for routing important financial messages around an internal business domain.

This meant that there was a lot of emphasis placed on the routing rules themselves, therefore there was a need to perform  rules testing.  Naturally, common sense dictated that a semi-automated test harness was the best was forward – quick, robotic therefore reliable, robotic therefore deterministic, robotic therefore executable at any time, robotic therefore … you get the picture.

Naturally the rest of the application’s java code was covered by unit and unit integration tests which were easy to write and exercised, well, robotically.

The wrinkle in our robotic CQL testing concept was that it required a fully operational Oracle Complex Event Processor to be running,  in order for the rules to be fired in something that approximates production. As we all know from TDD and other testing approaches, our CQL testing  concept is well beyond the unit test scope where mocking resources is the norm.

The solution? Hack a running CEP, replacing the internal publishers and consumers around each CQL processor within the EPN with dummy producers and consumers.  The  dummy producer then injects a pre-constructed canonical object into the feed channel to the CQL processor in question while multiple consumers are attached to the outbound consumers after the CQL processor.

The stage is then set to inject and  retrieve crafted canonical objects designed probe a particular CQL rule.

Easy, right?

Well, in fact its not as hard as you think.  Dynamically changing the behaviour of an application by INJECTING new code during runtime is not a new idea (see aspect oriented programming and ilk), being able to do code injection through a web interface by dropping in another application into an application container is new indeed. And I suspect, may be useful in AI operations – more on this in a later blog.

OSGi with resource sharing is the key here . For those unfamiliar with OCEP, it is based on a standard weblogic application container, but is flavoured with an OSGi application management layer.

What is this OSGi thing? Wikipedia tells us it is “OSGi (Open Service Gateway Initiative) is a Java framework for developing and deploying modular software programs and libraries“. Not a very useful explanation.  Practically, OSGi allows multiple programs to share resources through public interfaces, meaning that an app can be sand-boxed for all intents an purposes, except when you don’t want it to be. This is “advertising” in OCEP/OSGi speak.

In this case, when you set a channel to be “advertised”, another application via a discovery mechanism which leverages the subscription mechanism in the OSGI BundleContext, get access to that channel and reprogram the channel to use a set of publishers and consumers different from that which was originally deployed.

OK. Baffling. Here is the howto:

  1. within the META-INF/spring/[myapplicationcontext.xml] set all channels around your CQL processors in your application using the wlevs:channel attribute setting “advertise=true”
  2. create a new application, the one which contains all of the testing artefacts – dummy publishers and consumers, test canonical object construction code, test data that is converted into the test canonical objects
  3. finally write a utility in the new application which gets references to all live channels, reconfigures them to use new publishers and listeners, injects the test canonical objects and grades the routing outcome by detecting which listener should and should not have received the canonical object.

Again, easy right?

In my code repository you can find code snippets which you can use, free, to reconstruct the test harness. Sadly I can’t give you cut and paste code, as the implementation is customised to your real application test cases.  But you will get the idea.

The client took some convincing that it was worth spending the 5 or so days building this test harness, but the results speak for themselves 4 years on. Given the 150+ routing pathways through the application and the many corner cases associated, being able to prove and re-prove the rule behaviour during each release has eliminated hundreds of hours troubleshooting and thousands of dollars in commercial side effects by avoiding production incidents when compared to the previous non-OCEP implementation.

Happy testing!

Server Side

RMDS RFA OMM Madness

So recently I was engaged by a client to lifecycle some complex event processor applications, migrating from the old and creaky object-based TibMsg interface to the new and shiny OMM interface to listen to Reuters prices over the Reuters Robust Foundation (RFA) API version 7.6.

So the implementation was done – which required a complete change in thinking. It was a sharp departure from the warm and familiar world of over-the-wire object transmission to the far less reassuring domain of asynchronous messages.

The TibMsg interface requires actors (consumers and producers) to interact with RFA using preconstructed objects. Login and subscription/publication objects are constructed by the actors application code and then listeners are implemented to receive responses from RFA in the form of TibMsg objects. The listeners only need to be implemented for market price data and system events. Nice and easy.

The OMM interface uses the same login/subscription/publication pattern, but requires the caller to “message” RFA instead. That means that the login step is independent of the subscription/publication step and so on. Added to the mix is the fact that each message will be responded to asynchronously, using a listener running on a different thread which only must exist until the desired response is received and then the thread must exit –  with the notable exception of the market price listeners. The upshot is that the caller is forced to write a bunch of Futures (i.e. threads that return a value later on) to monitor response messages. It got quite messy to orchestrate.

Anyway, the OMM interface was successfully implemented and deployed to a development linux server for two components .  One component was (accidentally) configured to log each RFA call at the finest trace level, producing very large and useless log files for, get this, both components!

Yep, configure OMM for one component, and the other “automagically” takes on that unrelated components OMM config too. Bah!

After much head scratching and a quite a bit of blue air, it became apparent that this automagic was in fact due to a hidden configuration directory which had been snuck into the root of the app users home directory i.e. ~/.java/.userPrefs/com/reuters/rfa/RDMS.  Here there were two directories: Sessions/ and Connections/. The directory names under these sub directories were mad: “/::|&%%’ and so on.

So whenever one of the components was (re)started, RFA would copy the config shipped with the application to this hidden location and then start.  It was not clear to me if the copying action was exclusively additive, so if a config property, such as logging, was removed from the component config, the hidden properties location did not remove the unwanted property.

Anyhow, this is explained why when only one component was started it took on the (bad) configuration of the other application!

The cure was simple: fix the component config, removing the unwanted logging properties, shut down all relevant components, delete the Sessions/ and Connections/ directories in the hidden preferences directory, set the hidden preferences directory to read only AND then start a component.  Piece of cake.

Suffice it to say that this spooky action at a distance was not well documented … Thanks RFA!

Server Side

The BigDecimal rolls into town …

Like most programmers, I am probably the laziest person I know.  That means that when I can be bothered to do something, I want to take the minimum amount of time to do it, naturally with the best possible results.  This thought occured to me the other day when working on a collection of Domain Objects that had, you guessed it, similar but slightly different implementations of the same method.  Although that is bad (See my post in “Pet Hates” about this particular problem) the main issue here was how a misunderstanding of the internal workings of the BigDecimal can really screw things up for you.

The issue in this case was a utility method that attempted to determine whether any data in the current Domain Object differed from that seen in a another Domain Object.  Simple stuff.  Not so when you consider that the DO in question used a number of closely typed fields, including the infamous BigDecimal type. 

The problem arose when the values contained in DO #1’s BigDecimal fields were compared with that in DO #2’s BigD’s.  The BigDecimal.equals() method does a sterling job of comparing the two BigD values, right down to the scale (or mantissa if you are of a mathematical bent) i.e. the number of decimal places.  So 2.00 is NOT equal to 2.0, even though it clearly is to us Humans.

Every now and then, depending on how the BigD was populated, the equals() would return a false result when to the naked eye the data should be equal in all cases.  The “every now and then” part was worked out to be dependent on whether the source DO was obtained directly out of the DB via Hibernate or hauled out of Hibernate 2nd level cache. Turns out that the scale differs when reading from the DB as opposed to pulling an existing object out of the 2nd level cache.  Although this issue was limited to our code, you can see how easily an issue like this can creep in.  Scary stuff.

So the scale variation depended on whether the object had been cached or not by Hibernate, which in turn depended on whether the VM had been restarted recently or not.  The net effect was that if the DO existed before the VM restart, all was well with the equals() evalutation.  If it was created after the VM restart, the equals() call ALWAYS failed, despite the numbers in question appearing to be equal.

To the rescue came the BigDecimal.compareTo() method, which effectively normalises the scale without enforcing any potentially damaging data truncation.  Once implemented, our system became a lot more predictable.