Taming the IT Beast with IDIOM

Increase Business Agility, Reduce System Complexity

Over the past 15 years IDIOM has conceived, evolved, and demonstrated the effectiveness of it’s ‘decision centric’ development approach, which leverages both decisioning and agile approaches to radically simplify and strengthen commercial systems development. This advertorial describes the IDIOM products and how they can be used to implement the decision centric approach.

Development of systems using the approach, and the IDIOM tools, has been repeatedly shown to:

• simplify systems and their related development practices to reduce development time, cost, and risk by substantial amounts 
• increase business agility
• improve the alignment of business policy and computer systems
• improve runtime performance and reduce operational resource requirements.

The evolution of the approach has been chronicled in a series of published articles that remain available as published. For convenience, we have included a précis of the more important articles in the section titled ‘The IDIOM Approach – A Chronicle’. These précis’ provide a shortcut into the background of the methodology and its rationale. Links to the original articles are also provided for the interested reader.

The IDIOM products are derived from, and closely aligned with, the concepts described in the articles, and together they provide a cost effective and risk averse path to successfully building decision centric systems. An outline of each of the IDIOM tools is offered, followed by discussion on how they are used to support the decision centric development approach – that is, the IDIOM approach.

This is followed by further discussion about the implications of nimble, continuous, perpetual development, including versioning and effective dating, and how this is addressed by the tools.

The approach can address significant problem complexity, huge operational scale, and a wide range of business domains. Real projects are presented that demonstrate these capabilities.

Everything presented herein is based on common sense and a desire to simplify our overly complex IT world; there is no magic. But it does work, and it does deliver outstanding results.

The following two diagrams provide a high level view of the IDIOM approach.

The first diagram describes the policy development life-cycle, including development and testing of past, present, and future business policy using the IDIOM Decision Manager and the IDIOM Decision Manager Workbench. 

The development and testing of decision models that codify business policy is a nimble, continuous, perpetual process that is performed by the business ‘subject matter experts’ (SMEs) using the IDIOM Decision Manager – hands-on!

In day-to-day development, the IDIOM Decision Manager Workbench helps to support this nimble, continuous, perpetual process by allowing the SME’s to perform regular, production scale regression tests to confirm the efficacy of the evolving business policies in their codified form. The performance is such that this can be a fully automated, daily occurrence if needed.

And when looking back to audit and/or remediate past policy implementations, the Workbench provides the opportunity to audit and if needed, to remediate, existing implemented policy of any complexity and on any scale – actual examples include decades of payroll, or pension trust deeds.

At the same time we can look into the future, using the Workbench to perform policy simulations and what-if analysis to assess and verify the effects of proposed policy implementations. The IDIOM approach and toolset become part of the policy development cycle itself.

The single blue line in the first diagram indicates the extent of this process. We have customers that use the IDIOM Decision Manager and the Workbench as policy prototyping tools to develop and prove new business policies, which they then subsequently translate into formal policy narratives. The blue line represents IDIOM generated ‘logical English’, which is inserted into the policy narrative as an explicit and exact specification of the policy as it will be implemented. There are no ‘fingerprints’ between this logical English and the executable code – so that the business policy and computer systems are exactly aligned when implemented.

The second diagram outlines how the business policy can be deployed and executed using an IDIOM supplied decision engine. The decision engine is primarily an interface (provided in source code form for peace of mind) that executes decision models generated by the IDIOM Decision Manager (also in source code form). At execution time, the compiled JAR or dotNET assemblies for each decision model are executed behind the ‘decision engine’ interface.

The decision models shown in the diagram are examples for illustration only; decision models can address substantially complex business problems. In more mature environments, decision models can be reused across related problem domains, giving rise to the multi-modelling concept implied by the illustration. In response to this, IDIOM evolved the ‘control model’, which is a special instance of decision model used to orchestrate multiple (other) decision models at execution time.

The green arrow linking the two diagrams is used to indicate the flow of decision models from development to production. Using the IDIOM tools, these models are exported in a comprehensive, digitally signed manifest that includes: generated source code for each model; compiled code; both XML and ‘logical English’ descriptions of all rules; change logs; dependency tracking to other requirements source documents; etc. It is this manifest which becomes the ultimate ‘source of truth’ for the business and its policies; as such, it is usually fully integrated into the organization’s source control regime, at which time the handover from business to IT control is complete and assured.

“Decisioning: A new approach to Systems Development” [1]

This seminal article (published in the Business Rules Journal, Jan 2007) provided the background and conceptual building blocks for decisioning and decision centric development. It highlighted the importance and benefits of decision analysis to good systems design, and at the same time, noted its absence from the then current development methodologies. It drew important distinctions between decisions and ‘business rules’, and noted the relatively greater importance of decisions over processes as analysis subjects.

It laid the conceptual foundation for decisions (now defined as follows) to be a fundamental unit of requirements specification.

Decision: A single definitive datum that is derived by applying business knowledge to relevant data for the purpose of positively supporting or directing the activity of the business.

According to the article, these decisions are defined in the context of a decision model, which is defined as:

Decision Model: An ordered assembly of decisions that creates new and proprietary information to further the mission of the business.

And decisioning itself was defined as:

Decisioning: The systematic discovery, definition, deployment, and execution of automated decision making.

Requirements and the Beast of Complexity [2]

This popular article (published in Modern Analyst, 2010, ~23,000 downloads as of May 2015) updated the above article, and targeted requirements specifications as a significant weak link in the traditional systems development cycle. It presented a case for a new approach to requirements gathering and specification that is based on analysis of business policy to identify core business entities and their respective state changes, which are then captured and described as decision models and ultimately implemented as transactions. From the paper’s conclusion:

“This ‘decision-centric development approach’ . . . focuses on a ‘missing link’ between business strategy and operational systems – the Decision Model. The Decision Model is a new and important requirements artifact that is accessible and understood by both business and development practitioners, giving rise to a previously unobtainable level of shared understanding that can help bridge the gap between business strategies and the systems that support them. The Decision Model gives the business 'hands-on' control over the definition and implementation of its most critical IP – its decision making know-how.”

Decisioning – the Next Generation of Business Rules [3]

This article (published in both Modern Analyst and Business Rules Journal, 2013) noted that Charles Forgy, the inventor of the rete algorithm (which powered a generation of business rules engines), listed criteria that more or less excludes ‘transactions’ as an appropriate use-case for the algorithm. Notwithstanding, rete aligned, constraints based rules engines continue to be used in transactional systems, adding significant cost and complexity for little value.

The article then expands the scope of IDIOM’s business rules/decisioning concepts to include the ability to dynamically create data that is aligned with the idiom [4] of the business; that is, with the nouns and noun clauses of the proprietary language (the idiom) that w/e asserted is always present when describing business rules. By definition, the idiom is always proprietary to the author of the business rules because it is the need to define the proprietary business rules that gives rise to the idiom.

Dynamic transformation from raw data to data that is compliant with the business idiom is needed because a) it is rare that the real-world data is already in the required state for adjudication by the rules, and b) the business idiom is fluid and ever changing – it is the essence of business rules, changing whenever the business changes.

The article also highlighted the separation of the doing and decisioning components within a system:

“Service oriented architectures are inexorably trending towards separation of ‘doing’ components from ‘deciding’ components. The ‘doing’ components [can] be commoditized and outsourced. The truly proprietary code, the code that captures business knowledge and IP, is being increasingly concentrated in the ‘deciding’ components.”

This observation is expanded upon in the more recent Taming the Beast Article [5].

The Role of SQL in Transaction Centric Processing [6]

The logical disconnect described in the above paper, between the original design intent of the technology (rete in that case) and its current use, is echoed with SQL. This recent article (published in Modern Analyst (2014) and the Business Rules Journal (2015)) noted that SQL was designed from the outset to address a specific class of problem – set processing. This class of problem does not fit easily with transactions, and is to a large extent redundant if the data is hierarchically structured around core business entities (and is stored as such). In essence, the article made the observation that the ubiquitous SQL ‘join’ is fundamentally flawed when used in a transactional context: dependent tables should be joined onto their parent entity; the parent entity should not be joined on to the dependent tables. Again, and in apparent contrast with its original purpose, complex ‘join’ based SQL is routinely used to underpin modern day transactions.

And to close the circle on the business idiom described in the Decisioning paper above, we observed that the labels that are used to reference data in the business idiom must describe that data in context; that is, the labels used for the data values must include the context qualifiers. If we are going to be able to implement business rules that are described using the idiom, then it is necessary that we also describe the data in terms that align with the idiom, so that the rules of data context are matched with the vocabulary of the business idiom and vice versa. This correlation between the business idiom and data context is the essence of the linkage between business policies and their implementation in computer systems. It is also the backbone of business transactions, and in our experience, usually represents the major part of a transaction’s internal work effort.

The SQL paper further extended the scope of IDIOM’s business rules/decisioning concepts by highlighting data context as an integral part of a decisioning ‘requirements specification’. With a decision authoring tool that understands context, we can quickly and completely define decision models that address the full scope of policy driven decision-making, including validation, transformation, calculation, adjudication and workflow, and which require only ‘simple SQL’ (i.e. no joins) to provide complete and error free business transactions of any size and complexity. And these transactions will often execute much faster than solutions developed using more traditional approaches to use of SQL. This performance gain is primarily a reflection of the way that SQL is used to support the transactions.

Taming the IT Beast with Decision Centric Processes [7]

This most recent article (published as a companion to this advertorial) builds on the earlier articles to address the process perspective of the approach. It describes a process architecture and a matching development approach for development of decision centric systems. It uses ‘business transactions’ as the core building blocks for the system, which are defined as:

The activity that is initiated by an event that changes the state of a business entity, and which when complete leaves all systems compliant with all business policies relevant for that entity in that state.

A business transaction embodies the complete life cycle of core business entities, and is called and executed for each and every event that affects each entity.

The benefits are substantial. The architecture and its associated development approach collapse the number of moving parts in the system, leading to a dramatic reduction in the quantity of bespoke coding required.

Throughout the article there are observations regarding the simplification of systems development and execution that are attributable to the approach. When the approach is used to address complex business transaction processing, the demonstrated results include substantial improvements in the key development metrics of time, cost, and risk; and often, significant improvements in runtime performance.

These already encouraging results are accompanied by improvements in business agility, better alignment between business strategy and operational systems, and systems that are more transparent and more durable.

In the approach described by the article, SMEs assume hands-on responsibility for the definition, capture, and deployment of the business policies that manage the entity life-cycles within the business transactions. By using the IDIOM tools to implement the approach, these business SMEs become empowered to take full and complete ownership of the implementation of business policy within the operational business systems.

IDIOM Tools

As you might expect, IDIOM has been fine-tuning its flagship IDIOM Decision Manager product since its launch in 2001 to fully support the concepts outlined in our series of articles.

In fact, the name of our company was an early reflection of the importance of the business ‘idiom’, and its relationship to both business rules and data. The ‘idiom’ is the proprietary language that is used to describe the essence of any business – it is used to describe its business policies, its decision making, and its business rules. And in so doing, the idiom refers to the data ‘in context’. The challenge lies in mirroring this context within the various executables that must implement it.

Defining, capturing, validating, and automating the business ‘idiom’ is the essence of our business and of our flagship product, the IDIOM Decision Manager.

Furthermore, the high performance SQL to XML mapping tool alluded to in the Taming the Beast Article [8] exists as the IDIOM Mapper, and is in part responsible for the high performance of many of our applications.

The entire Mapper cycle is also embedded into a comprehensive management application called the IDIOM Decision Manager Workbench. The Workbench is an industrial scale, ready-made batch application processor for large scale, high performance batch processing.

A transaction often requires dialog with a human actor. A dialog is easily managed by building an IDIOM Form over the business transaction’s entity and event data – the same data that is available to the decision models that interpret and maintain it. An IDIOM form includes the ability to execute IDIOM decision models inside the form on a large scale, field by field if necessary, to ensure a fully reflexive user experience.

Finally, the transaction end state might need to be reported via one or more business documents. The IDIOM Document Generator can be used to generate complex business documents under the control of decision models, using only Word authored text and images as additional design input.

The IDIOM Decision Manager

• IDIOM Decision Manager is a tool for graphically modeling, testing, and deploying business decision-making logic – without programming!
• A tool for the policy maker, not the programmer.
• IDIOM automates complex decision-making at the enterprise level, deployable as industrial strength stand-alone components.
• In day-to-day practice it is used by SMEs, or analysts working closely with them.
• Together they model the business policies in terms of both data and decisions (see Decision Model below) before moving on to define the underlying logic that binds them together (see Formula below).
• The decision logic and data are usually modeled together in a single combined process of analysis and definition.
• The data model and the decision model share the same ‘context awareness’, with current-context and context boundaries visually highlighted at all times within the development palettes.
• Testing of the models is available at all times within the development palettes themselves; full regression testing (incl. ‘model answer’ differencing) is available in an adjacent ‘Test Executive’ (not shown).
• Deployment as 100% generated software components is fully automated and ‘without fingerprints’.

IDIOM Decision Manager – decision palette showing a Decision Model

• Example above is a small but real model drawn from a city council implementation of policy that calculates financial contributions to be paid by property developers.
• The problem domain is decomposed using a ‘mind mapping’ approach until we reach the atomic units that we call decisions (rounded boxes).
• This ‘decision model’ and the adjacent data model (left hand panel above) are demonstrably aligned and integrated through shared context – validating and strengthening both.
• The data model defines the entity at rest; the decision model defines the valid state transitions. Together they completely define the entity life-cycle and required business policy.
• The atomic ‘decisions’ provide an easy entry point for specification of the underlying rule details via the Formulas (see next).

IDIOM Decision Manager – formula palette showing a Formula

• The underlying rules details are easily captured using a ‘Lego like’ drag-and-drop approach that is ‘more fun than playing golf’ according to the CEO of one of our largest customers – there is no scripting or coding required to build formulas.
• The rules can be tested immediately within the IDIOM Decision Manager palettes.
• When finished, IDIOM Decision Manager generates computer source code (C# or Java) with a single button click, to be called by any application at run-time using any of a wide variety of supplied interfaces and wrappers (in-line, dll, web service, queue service, many more).
• And at the same time, it generates the model into business readable documentation (PDF).

Key Points of Difference

• IDIOM’s decision models do for decisions what data models do for data – a powerful abstraction that makes the underlying complexity visible and manageable.
• The models allow data transformations and more traditional business rules to be intermingled. Business rules acting alone are severely limited in their ability to fully resolve complex commercial problems – invariably, in-line data transformations are necessary to facilitate the calculate/adjudicate/act behavior of business rules.
• Context is continuously displayed and actively managed.
• Decision models that incorporate both data and rules behavior enable a further critical capability that is unique to IDIOM Decision Manager – the models can be fully tested using real-world cases directly in the builder palettes. No external technology or application support is required to empirically prove the completeness, consistency, and correctness of the models.

Key Points of Innovation

• Fundamental redesign of the traditional SDLC by fully separating the development and automation of business policy (deciding) from development of the systems that support it (doing).
• Use of IDIOM is effective in spawning a ‘Business Policy Development Life Cycle’ that is managed independently of and alongside the traditional System Development Life Cycle.

Key Value Propositions

• 100% alignment of systems based decision making with business policy, because the business owners have hands-on custody and control of the policy definitions actually used by the system.
• Increased agility with reduced business risk through business modeling and empirical testing of policy definitions prior to automated generation and implementation.
• Significant reduction in the business cost of developing and implementing automated business policy.
• Further reduction in software development cost, time, and risk through reduced system complexity and clear separation of concerns.

Further Benefits

• Full auditability of policy changes, and visibility of policy implementation through the graphical models and logical English PDF generation.
• Decision model artefacts can be traced to/from ‘sources of truth’ in underlying business policy documents (Word, Excel) using the IDIOM Tracker for requirements traceability.
• The IDIOM Decision Manager Workbench is available to experiment with and further develop policy independently of the underlying systems and the SDLC.
• Automated, robust, industrial strength deployment on any scale that can be supported by the ‘host application’ and its underlying platform.
• Simple injection into legacy systems leading to eventual legacy replacement.

IDIOM Forms

IDIOM Forms is a tool for generating web2.0 forms that embed the power of IDIOM decision models into the form itself. As the user navigates through an IDIOM Form, the integrated decision models can be executed on an element by element basis to make various business calculations and assessments; and to modify the form’s meta-data. Control of the form’s meta-data allows the decision models to dynamically adjust the visible shape and content of the form on an element by element basis.

A single IDIOM Form can be used to process complex business transactions (e.g. an insurance application, a clinical pathway, a loan application) through to closure, including such functions as validation, transaction acceptance, costing or pricing, up-selling, and determination of subsequent workflow amongst others. Usually, a single form is used to manage the full life-cycle of the subject entity (i.e. not just individual state changes).

The IDIOM Forms Engine (the runtime component of IDIOM Forms) is production hardened after many years use on thousands of servers throughout the NZ health sector. The deployed IDIOM Forms Engine became the basis for the NZ Health Information Standards Organization (HISO) forms standard and is currently the most compliant under that standard. The Forms Engine is now also widely used in the finance sector for insurance underwriting and claims management.

IDIOM Decision Tracker

The IDIOM Decision Tracker is a tool to map Microsoft Word and Excel policy documents to IDIOM decision models for full bi-directional traceability between corporate policy definitions in the Microsoft documents and their actual implementation as IDIOM generated decision engines.

IDIOM Mapper

The IDIOM Mapper generates ‘simple SQL’ from an XML configuration document, and then executes a full round trip transaction cycle: from database to XML; then rules execution (multiple decision models); and from XML back to the database if required. All done extremely quickly, either in single transactions or in batch. The Mapper process is thread-safe and can execute in many process streams for scalability that is limited only by the database itself.

IDIOM Decision Manager Workbench

The IDIOM Decision Manager Workbench is a user operable application for running decision models across enterprise databases on a large scale without the need for IT technical support.

• A generic batch processing platform for use by IT and/or business operations.
• A platform for the auditor, the business policy manager, the product manager, the corporate business strategy manager.
• An audit tool for identifying, reporting, and managing anomalies, errors, and issues of concern in any database.
• A simulation tool for comparing the performance of current and ‘to be’ versions of any automated policy.
• A tool that can intelligently modify the inbound data to simulate changes in the make-up of inbound transactions over time.
• A data masking tool, able to replicate large scale databases with all personally identifying details masked from its own library of several million fake identities. Relationships between identities in the source data can be maintained, notwithstanding the use of fake identities.
• A conversion tool: able to read data from one system in its proprietary format; intelligently transform it through one or more decision models; and then output it to a new system also in its proprietary format. Millions of entities described by hundreds of tables can be supported.

Examples of Use

Superannuation Fund Administrator: Perform all period end processing, including fee’s, insurances, member adjustments, entitlements, and reporting, for several hundred thousand fund members.

Insurer: Compare the current underwriting policy with a proposed underwriting policy across a recent portfolio of 500,000 insurance policies to determine potential changes in the rate of referral and its attendant costs.

Superannuation Fund Administrator: Run 100’s of distinct audit tests across 1,000,000 member accounts on a daily basis to independently verify the production data.

City Council: Compare this year’s rating policy with next year’s proposed rating policy for each property in a city of more than 500,000 ratepayers to identify outlier changes in the rates actually charged.

Insurer: Dynamically modify key attributes of real transactions to simulate changes in the make–up of in-bound business, and assess the impact of these changes across an entire portfolio.

Enterprise Class Features

• Full authorization and audit controls down to the field level for all users of the platform.
• Seamless operation across multiple user definable environments, for instance Development, UAT, Simulation, Production.
• High performance, including parallel processing on a large scale across multiple machines, for instance running up to 24 Processes on each i7 class CPU, with multiple CPU’s possible, each logging back to the Workbench for centralized management.

Measured Performance

• Daily pass of 1,000,000 pension fund members each comprising of a join of 20+ complex member related tables for a total of ~one billion rows.
• 10’s of decision models implementing hundreds of individual member tests.
• Alerts captured and reported daily for start of business.
• Run daily in 2 hours using one i7 class processor, with data drawn from an IBM iSeries.

IDIOM Document Generator

The IDIOM Document Generator is a tool to collect and collate Word documents (which are used as document templates) and many associated Word text fragments (‘blurbs’), and to assemble these into complete documents under the control of a decision model.

At document design time, the Document Generator inspects the templates and blurbs and builds a list of all blurbs (including nested blurbs) that could be used for each candidate document; and it also builds a list of all of the variables declared within them. It then places these two lists (blurbs and variables) into an XML document.

A decision model is then built that will extend this document at runtime when the specific business entity context is known. This decision model will analyse the business entity data in order to set flags to control the insertion (or not) of each blurb; and it will derive the substitution values for each of the listed variables.

At runtime, when the actual business entity data is provided and the decision model has executed, the Document Generator will read the tags to insert the desired blurbs into the template, followed by substitution of all variables, so that large and complex documents can be generated automatically under the control of the decision model to reflect the exact circumstances of the underlying business entity.

The Document Generator also allows for insertion of images and for calling bespoke programs to insert additional generated text or images.

Documents can be generated into a number of formats, including Word and PDF.

In day to day use by SME’s, the Document Generator only requires standard Microsoft Word and IDIOM Decision Manager skills.

Putting the Approach and the Tools Together

Using the Approach

We have highlighted the role of the business ‘idiom’ as the language of business policy, and the importance of its automated equivalent in a decision centric system. In order to automate the idiom, we need to view data in context. We have also described how the XML Schema provides us with a partial ‘context map’.

The IDIOM approach uses the IDIOM Decision Manager to provide the SMEs – the subject matter experts who actually define the decision models in accordance with business policy – with a drag and drop GUI to graphically build decision models, including the validations, transformations, adjudications, calculations and workflow logic (collectively, the decisions) directly over the XML Schema without needing to know any XML navigation syntax or other programming skills.

The schema provides our basic context map, so that the complex task of understanding and controlling context is largely managed by the tool. The SME is only able to reference data that are valid for the current context, which is the locus of the decision that is currently in focus, and is always a single, unique point on the schema.

The decisions and their related logic are graphically captured using the Decision Manager’s decision model and formula palettes. A decision model is strictly hierarchical, executing left-to-right/top-to-bottom so that the decision execution sequence is graphically displayed and easily controlled.

The IDIOM Decision Manager allows the SME to dynamically create new (temporary) data at any time. These plug into the schema as required, and are usable in the same way as any other value.

The IDIOM Decision Manager also allows the SME to create test data and to test any part/whole/group of decisions directly in the development palettes during the development process, with context being graphically displayed as the test execution takes place. Real XML entity records from operational business systems can also be used immediately and directly for this purpose.

When finished, entire libraries of test cases can be regression tested within the IDIOM Decision Manager, before generating the source code and deploying it with a single click for system testing. Again, this can be a use for the XML entity records described above.

Because the decision models and their associated schemas provide a complete specification, a high performance and well documented computer program can be completely and automatically generated ‘without fingerprints’.

The following outline summarizes this new decision centric approach:

1. Develop an XML Schema to describe the ‘real-world’ entity that is the subject of the transaction, to be used as a ‘context map’ for subsequent decision model development.
2. If an existing database exists, use the IDIOM Mapper to generate and execute a series of ‘simple’ SQL queries to collate a complete, in memory XML object as defined by the schema. The reverse mapping can also be generated for write back if required.
   Or, you can use XML entity records directly, either from the file system or a database.
   The database design described in the ‘Simplified Data Model’ section of the Taming the Beast Article [9] supports an efficient and simple hybrid relational/XML approach.
3. Use IDIOM Decision Manager’s drag and drop GUI to declaratively build out the decision model(s), using the schema defined ‘context map’ as an active and extensible requirements template.
4. Test the evolving transaction ‘in situ’ to ensure completeness, consistency, and correctness at all times.
5. Regression test in the same tool to confirm absence of unintended consequences; and/or run simulations to verify that changes in the underlying business policy achieve the desired objectives.
6. Generate and deploy without further manual intervention.

Further Effects of the Approach

Building a complex transaction using the above approach has many positive downstream effects.

1. The ‘decision model’ is tightly structured, and able to be rendered into many formats, including logical English, computer source code, and XML. It is therefore transparent, auditable, and reliable as a specification of the executable code.
2. This is a tool for the SME. The high degree of automated assistance and reduced development complexity means that the SME is only contributing what they already know – their own proprietary business knowledge. Productivity is quickly (days, not weeks) and reliably achieved; arcane IT skills are not needed and offer no advantage.
3. Generation of code means that the code per se does not need to be tested; only the logic supplied by the SME needs to be tested. This means a substantial reduction in testing effort.
4. The automated tool assistance and reduced human input reduces the potential for errors. This gain is compounded by the extensive on board testing support already described, so that it is rare for a decision model to be anything other than complete, consistent, and correct when released into later-stage testing cycles.
5. All aspects of the decision model support continuous, perpetual versioning. When the models are implemented as ‘content’ in computer systems, they provide a practical and robust mechanism for SMEs to independently develop, manage, and deploy the organization’s codified knowledge (including complex transformation, adjudication, calculation, and workflow policies) on a daily basis – continuously and perpetually.
6. The decision models are technology agnostic and technology independent, forming a complete historical record and the ultimate ‘source of truth’ for the organization’s proprietary knowledge; extracted, tested, confirmed, and documented by that organization’s SMEs in the normal course of business.
7. A tool assisted approach as described allows SME’s to capture, test, and deploy this knowledge extremely quickly. In a 100 developer year project, 80% of the system code was generated from decision models that were built in less than 20% of the total development hours – all of which were contributed by business analysts drawn from business branches (i.e. not IT specialists).
8. This development efficiency is sustainable over the long term, offering huge business agility with reduced cost, time, and risk
9. A decision model is a durable life-long artefact that defines the business in perpetuity. As a complete record of an organization’s knowledge, and with multiple and extensible source code generation options, there should never be another ‘legacy system’.

The ‘Make a Loan’ Transaction

The following diagram shows the example ‘make a loan’ transaction as it was presented in the Taming the Beast Article [10]. We have overlaid the original diagram with the IDIOM tools that would participate in its development and operation as an IDIOM derived application.

The majority of the declarative development effort is in the decision models, which are compiled and executed behind the standard interface that is built into the generic business transaction activity. Independently measured projects have assessed this declarative development effort as being at least 20 times more efficient than traditional SDLC approaches.

The generic business transaction container would use the event context to look up and load the relevant decision models based on the nature of the event that it is responding to.

An IDIOM solution could also use the IDIOM Forms engine to run the web forms for the users of the system. And the Document Generator would be deployed as a service within the host application to generate and serve business documents. The runtime components of the IDIOM Forms and the IDIOM Document Generator are included in the diagram.

Note that IDIOM Forms and IDIOM Document Generator both use IDIOM Decision Models to drive their respective runtime meta-data, which is what makes these runtime engines truly generic.

The ongoing development of the decision models, forms, and documents requires the builder versions of each of these tools. This is shown inside the ‘Financial Product’ Builder Platform. The Workbench is also available in this platform to assist with policy modelling on the one hand, and regression testing on the other.  

This entire transaction can be built with only a handful of bespoke code, much of which is available in our existing supplied libraries. The web services, shown as the pink arrows would need to be built bespoke according to the requirements of the other parties; this typically constitutes the largest bespoke coding effort implied by the diagram.

Continuous, Business Led Development

One of the important objectives of a transaction/decision centric application is continuous development and release of business policy by business owners, or more specifically, the SMEs. This business policy is usually focused on management of entity life-cycles, whether as formally defined ‘business products’ or as case management or other services.

Policy adjustments should be able to be developed, tested and deployed at any time. The intent is not to actually deploy changes (say) daily, but to be able to deploy new policy at any time the business requires, without the delay and risk inherent in traditional ‘code-drops’.

All parts of the business transaction, including all policy changes, are effective dated and cumulative forever. The business transaction is always able to operate ‘as at’ any prior date. The IDIOM approach even allows effective dating that spans changes in the entity schemas.

Continuous development and deployment is helped by IDIOM’s zero script, fully declarative approach, coupled with a strong focus on versioning of and within all of the declarative IDIOM components, so that continuous, rapid change can be sustained – perpetually.

With a transaction/decision centric application there is a new dichotomy in versioning: the schemas, forms, rules, and business documents, and all of their associated reference data, can all be deployed externally to, and independently of, the code-base of the host application. There are now two distinct classes of versioning – changes that are applied as new declarative content (in the business transaction layer); and those that require system code changes in the host application layer, via a traditional ‘code drop’.

Broadly speaking, the former can be managed by suitably trained business SME’s and policy experts using the new “business policy life cycle”, while the latter follows established IT systems development best practice. While the traditionally expensive IT development practice is still critical in a decision centric application, it is now dealing with a much smaller part of the total installed code-base – in one carefully measured ‘green-fields’ project that developed a full insurance system, the native coded ‘host application’ was less than 20% of the entire run-time image of millions of lines of code. The rest was generated from drag and drop designs produced by SME’s using an independent development cycle that delivered functionality 20 times faster than traditional code development [11]. This is one reason for the dramatic improvement in time, cost, and risk that is achievable with decision centric development.

The continuous versioning that is inherent in the IDIOM approach is designed to support nimble continuous perpetual change in a decision centric application.

Transaction Versions

A ‘transaction version’ is the net effect of all of the rules that implement the entirety of the business policy as it was defined at the point in time that is defined by the transaction’s effective date. Transaction versioning is therefore implicitly an effective dating regime. Each change in the rules or of any other aspect of the decision model within a business transaction implies a version update for that transaction, and by extension, for the product or service that it supports. All such, versions always remain valid and operational, subject to the effective date of the transaction at runtime.

Business transactions can also be versioned on criteria other than effective date through configuration or rules changes – for instance, the term for specific conditions might be negotiated on a customer by customer basis, so that each customer might see a different version of the same transaction rules on the same date. This is achieved by dynamically setting the effective date.

Effective Dating

Effective dating is fully supported by the IDIOM Decision Manager.

An effective date is always present during the execution of decision models, which may be set by default (i.e. today); or by the calling application; or by rules within the model itself (it is possible to change the effective date during the processing of a decision model using business rules).

The effective date that is set by/for the transaction as above is used to automatically select every aspect of the decision model at runtime. All aspects of the decision model, from the complete model itself down to single reference data values and/or rules, are start and end dated as required (usually automatically).

Effective dating is not supported by the W3C XML Schema standard, and since these are the basis of IDIOM forms, this also limits effective dating in forms. However, effective dating in the schemas and forms can be emulated by the rules that process them. For instance, effective dating the presence of an element in a form is achieved by effective dating the decisions that instantiate that element.

Finally, enumerations (i.e. pick-lists etc.) pose a specific versioning problem in forms. While the Schema standard does support enumerations, these cannot be used because of context and effective dating constraints which are not supported. Therefore, enumerations in an IDIOM form are typically generated by purpose built decision models, so that the enumerations used in the forms are selected at runtime according to both context and effective date.

Versioning with New Data

As we have noted, the host application described in the Taming the Beast Article [12] is generic. When implemented as described, this means that entirely new business entities and their related meta-data can be introduced at any time without changing the host application code. For the sake of clarity, new types of entities, including new product types, can be introduced and, unless new activities are required, can be fully functional within the existing host application without any change in that application’s code base.

Existing IDIOM design patterns also provide just-in-time auto-migration of existing data onto new schema designs, which allows versioning of the underlying entity data designs.

This means that multiple schemas, forms and associated decision models, and all versions thereof, need to be managed and available simultaneously within the host application. A database design that can contain and manage all of these artifacts is described in the ‘Simplified Data Model’ section of the Taming the Beast Article [13]. 

Following are samples of real decision centric projects that highlight the significant problem complexity, the operational scale, and the range of business domains that the approach is capable of servicing.

Examples of Decision Model Complexity

Following are two recent project examples that illustrate the complexity that can be addressed using decision models. Each of these examples was implemented by a single decision model.

Defined Benefit Scheme

Defined benefit schemes are contracts between a pension fund and its members. They are renowned for their complexity and long-life. We have recently codified the trust deed (described in the adjacent sidebar) for one large fund with a membership in the hundreds of thousands. The underlying trust deed is more than 30 years old, and the complexity of the entitlement calculation has grown considerably since it was first drafted.

A member’s entitlement under the scheme can change on a daily basis; as you can imagine, the current entitlement is keenly sought on a regular basis by the members managing their affairs approaching and post retirement.

Calculation of this entitlement was previously performed in batch by a legacy system that took >48 hours to run; furthermore, these were only top-up calculations for the current period.

The IDIOM version of the calculation re-calculates the full 30+ years so that it can address anomalies in historical data and calculations; it also runs a full order of magnitude faster than the legacy system ‘top-up only’ equivalent. In fact, this calculation speed, and the fact that each member is processed within a discrete transaction, means that the calculation can now be performed in real-time as well as in batch. The downstream benefits of this are significant.

For instance, the fund can now provide modelling of future changes in real-time so that the member can use the calculation for financial planning purposes; and/or, the fund can provide a service that provides advance information to members regarding future regulatory changes, scheme changes, etc.; and/or the fund can perform financial forecasting and affordability testing for the investment managers of the fund.

Payroll – Audit and Remediation

The aim of this project for a Government Agency was to: 

• Recalculate and remediate termination payments for all terminated employees
• Recalculate and remediate past payments for active employees
• Provide a transaction to correct the current payroll system on a go-forward basis, pending its replacement.

To achieve these aims it was necessary to build an employee level business transaction that:

• Uses the IDIOM Mapper to retrieve the required source data from the payroll system database (e.g. Timesheets, Employment Details, Allowances, Payments Made, Leave Taken, LWOP Days etc.)
• Constructs a temporary data structure suitable for the analysis
• Calculates what the payments should have been, and compares these with payments actually made to calculate remediation amounts
• Produces a report with references to the individual inputs and the intermediate calculated structures to provide a detailed audit trail to support the remediation and ongoing payments.

The complexities involved in this project included:

• ~10,000 members terminated, ~20,000 members current
• Total remediation pay outs of tens of $$million over a 10 year horizon
• Poor discipline and procedures in the underlying payroll systems resulted in varying data quality, including inconsistent and irregular data (e.g. leave taken following termination) that needed to be rectified according to new and agreed remediation policies
• Data structures and database keys changing over time as systems migrated
• Running different calendars for different parts of the country
• Data to be analysed reached as far back as 40 years
• All nuances in the legislation and awards over the full term including definitions of Base Rate, Ordinary Rate and Average Weekly Earnings
• Creating an intermediate data structure representing each day the person was employed, to be marked with the amount the employee was paid that day in aggregate, or if leave was taken
• Calculating various cash-ups: Annual Leave, Long Service Leave, Shift Leave and Statutory Holiday pay outs
• Comparing the Calculated Amount to the Amount Paid and calculating the pay-out amount
• The Amount Paid was not uniformly found and needed to be located from different places depending on time period

Examples of Operational Scale

The decision centric transaction approach allows applications to scale well as the following examples show:

1. A global leader in micro-financing is on-boarding up to 42 million new business applications per day on a standard PC. Many decision models, including channel and partner specific models, are used to process each application.
2. The hospital authority of a major economy is using IDIOM decision models to generate 1 million invoices/week across 47 hospitals, 121 out-patients clinics.
3. A pension fund is performing several hundred distinct audits that are described by dozens of ‘decision models’ for ~1million fund members daily. The scale of data is substantial, involving a total of ~1billion rows of data that are extracted from a relational database in ~30 minutes using the IDIOM Mapper. Current audit processing takes around 2 hours on a single i7 class processor using 24 processes; this time can be reduced to match the data delivery rate by increasing the number of processors, something that is easy to achieve in a decision centric application.

Examples of Projects across Various Domains

The following list is a snapshot of some of the projects that IDIOM has been working on over the last 2 years. This list is provided as a working example of the type and nature of development works being generated by ‘decision centric’ thinking.

• Major Regional and Global Insurer: Conversion of dozens of insurance products to IDIOM Forms and IDIOM Decision Manager, supported by IDIOM Decision Manager Workbench for product modelling and testing. These products are large and complex, with thousands of data fields involved per customer policy application. Some policies cover thousands of individual risks.
• Major Regional and Global Insurer: Centralised underwriting and rating service for core insurance products based on IDIOM Decision Manager. This project positions IDIOM as a core technology in one of the largest insurers in the region.
• Health Funding Authority: Codify hospital funding rules using IDIOM Decision Manager. All patient episodes state wide are evaluated and costed.
• Intercity Bus Operator: Full system build, using IDIOM Decision Manager for the just-in-time pricing that differentiates the operator.
• Superannuation Fund: Automation of month end processing – insurances, fees, member adjustments, and reporting using IDIOM Decision Manager. 120,000 members processed each run, with time to run reduced from 2 days to a few hours when compared with the prior legacy system on the same machine.
• Fund Manager: Full system build with significant use of IDIOM Decision Manager and IDIOM Mapper.
• Insurance Service Provider: A nationwide service linking all life insurers with GP’s for collection of medical history data, using IDIOM Forms that manage >1000 data items/form.
• National Health Service: Automation of a lifetime clinical pathway using IDIOM Decision Manager.
• National Health Service: Automation of organ donor/recipient matching to provide an immediate, fair, and transparent ‘next recipient’ selection, using IDIOM Decision Manager.
• Micro Insurance Provider: Full insurance systems build for multi-country use, using IDIOM Forms and IDIOM Decision Manager. The number of active policy holders is 10’s millions.
• Award Winning Airline: Frequent flyer loyalty calculations using IDIOM Decision Manager.
• Life Insurer: Underwriting management process for enterprise wide use by 100’s of underwriters using IDIOM Decision Manager and IDIOM Forms. This application is cloud based, and credited with helping the underwriter go from zero to market leader in 2 years.
• Life Insurer: Claims management process using IDIOM Decision Manager and IDIOM Forms. This application is cloud based.
• Border Protection Agency: Adjudication of entry pass/fail at border control points using IDIOM Decision Manager.
• Regional Local Body (NZ): Web application to calculate developer contributions using IDIOM Decision Manager.

As indicated above, the IDIOM tools provide a cost effective and risk averse means to realizing the benefits of decision centric approaches. Please contact us if you would like to discuss any aspect of this publication, to see a product demonstration, or to contemplate a proof of concept.

We would be happy to oblige.

Mark has more than 35 years history in software development, primarily with enterprise scale systems. During the 1980s Mark was actively involved in the development and use of data- and model-driven development approaches that later achieved widespread use. Application of these approaches to the development of an Insurance system for one of the world's largest insurers was formally reviewed in 1993 by the University of Auckland who concluded that "this level of productivity changes the economics of application development."

In 2001 Mark led a small group of private investors to establish Idiom Ltd. He has since guided the development program for the IDIOM products toward even more remarkable changes in "the economics of application development." He has had the opportunity to apply IDIOM's "decision oriented" tools and approaches to projects in Europe, Asia, North America, and Australasia for the benefit of customers in such diverse domains as superannuation, finance, insurance, health, government, telecoms, and logistics.

IDIOM Bio:

IDIOM develops and licenses decision-making software that automates business policy on a large scale, making systems more transparent, agile, and durable, while reducing development cost, risk, and time.

IDIOM’s innovative business oriented software is used by business users to graphically define, document, and verify corporate decision-making and related business rules; it then auto-generates these into small footprint, non-intrusive software components for use in systems of any type or scale. IDIOM is a pioneer in the development and use of decision automation concepts, and has applied these concepts to develop and automate business policy for customers around the world in local/state/central government, insurance/superannuation/finance, health admin/clinical health, telecoms, logistics, and utilities. 

IDIOM automated business policy and decision making extends far beyond mere business rules, so that larger and more complex decision making can be fully delegated to business experts. IDIOM enabled development and management of policy based decision making by policy owners creates a propitious ‘business policy life-cycle’ that significantly improves business agility and transparency.

IDIOM develops and licenses: IDIOM Decision Manager™, IDIOM Forms™, IDIOM Document Builder™ IDIOM Mapper™, IDIOM Tracker™, and IDIOM Decision Manager Workbench™.

References/Footnotes:

1. http://www.brcommunity.com/b326a.php Decisioning: A new approach to Systems Development
2. http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/1354/Requirements-and-the-Beast-of-Complexity.aspx
3. http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/2713/Decisioning-the-next-generation-of-Business-Rules.aspx
4. http://www.thefreedictionary.com/idiom [a. ‘A specialized vocabulary used by a group of people’]
5. See, ‘Taming the IT Beast with Decision Centric Processes’ 
6. http://www.modernanalyst.com/Resources/Articles/tabid/115/ID/3109/The-Role-of-SQL-in-Decision-Centric-Processes.aspx
7. See, ‘Taming the IT Beast with Decision Centric Processes’
8. See, ‘Taming the IT Beast with Decision Centric Processes’
9. See, ‘Taming the IT Beast with Decision Centric Processes’
10. See, ‘Taming the IT Beast with Decision Centric Processes’
11. See this paper for details: http://www.idiomsoftware.com/uploads/gallery/temp/1346423022IDIOM_Development_Performance.pdf
12. See, ‘Taming the IT Beast with Decision Centric Processes’
13. See, ‘Taming the IT Beast with Decision Centric Processes’