The Event Model (TEM) in Action

This article is Part 2 of "Introducing the Next Horizon: The Event Model"

NOTE: The diagrams and Tables in this article are drawn from materials copyrighted by Knowledge Partners International LLC and IBM Corporation

The Event model (TEM) is a novel way to model, develop, validate, maintain, and implement event-driven applications targeted at business people. It’s a result of a two year joint study between KPI[1] and IBM Haifa-Research to influence the world of events in the same way The Decision Model (TDM) influenced the world of business rules when it first appeared in 2009.

This is the second article about TEM. Our first article [2] answers two fundamentals questions: Why TEM? and WHY TEM now? In this article we address the question of How TEM? by providing a deeper look into TEM main constructs and how to do a high level design in TEM. For those who missed the first article we hope that you will still find this article useful and self-contained in such a way that it will be easy to follow and understand.

The Power of Events

According to Forrester[3], (complex) event processing, or CEP, combines data from multiple sources to infer events or patterns that suggest more complicated circumstances. The ultimate goal: to identify meaningful events, such as opportunities or threats, and respond to them as quickly as possible.

Luckham[4] (Dec 2013) points out that companies that implement active, continuously-running real-time intelligence systems will leverage CEP because it is the only way to extract insights from current data in an event-driven manner. Companies that understand CEP have more and better real-time intelligence than those that don’t understand it. The use of CEP will expand further as the pace of business accelerates, more data becomes available in real-time, and business people demand better situation awareness. However, more than 95% of complex event processing is still done on an ad-hoc “build” basis, without the use of any systematic event processing tool[5]. The question is: why is the adoption of event processing tools so low? We believe that to really leverage the power of events, the critical piece of the puzzle that must be solved is how to make event driven applications easier for non-technical people to understand, easier to develop, and easier to deploy. TEM is intended to exactly fill in this gap. Today, The Event Model is in its early stage just as The Decision Model was prior to its publication in 2009.

TEM Benefits

Let’s recall what the main benefits of TEM are from Part 1:

• A simple diagram designed specifically to derive event logic
• No program code required for understanding
• A business-oriented glossary
• Absence of technical terms or artifacts
• Independent of technology
• Supported by integrity principles, and
• Automatable.

Who is a TEM user?

TEM is primarily targeted at a non-technical audience who wants to gain control over the event logic in their organizations. As you will see later on, a non-technical user should be able to specify the event business logic of an application in TEM, while the technical details can come at a later phase, from which the model is generated into code.

Objectives

In this Part 2, our main objective is to walk through the essentials of TEM using an illustrative example that demonstrates the top-down design approach. Therefore, included in this top-down approach are the TEM pieces most useful to the non-technical audience. It does not contain all TEM pieces needed to build up a complete and correct TEM model for automatic code generation.

Our Illustrative Example

In Part 1 we introduced the situation called a Suspicious account. A situation in TEM is an ultimate event-based conclusion or derived event that is emitted to the outside world. In our first example, A suspicious account is derived whenever there are at least three large cash deposits every 10 days.

Now, in order to show the richness, and at the same time, the simplicity of TEM, let’s complicate our story of a suspicious account. A situation of a Suspicious account is derived when any of the following three derived events is detected (Figure 1): Frequent large cash deposits; Frequent cash deposits followed by transfers abroad; or Lack of account activity.

Figure 1: Suspicious account situation

Now for the specifics.

We derive a Frequent large cash deposit event whenever we have at least three Large cash deposits every 10 days for a certain account. A Large cash deposit event is derived whenever the amount of the cash deposit is larger than the amount allowable for the customer (customer threshold). The keyword “every” in the temporal context denotes a sliding time window of 10 days, that is, every day we open a new context that closes after 10 days.

A Frequent cash deposits followed by transfers abroad event is derived whenever we have at least 10 Cash deposit events followed by a transfer abroad event every 30 days. The latter is derived whenever we detect a Transfer abroad event that occurs after a Cash deposit event for the same account and that the cash amount is larger than the transfer amount, for the same account. The temporal window starts with a Cash deposit event for a period of 3 days.

A Lack of account activity event is derived whenever, for a specific account, there are neither Cash deposit events nor Transfer abroad events in a period of 20 days starting with the occurrence of either a Cash deposit or Transfer abroad event.

Finally we derive a Suspicious account situation whenever a Frequent large cash deposit, a Frequent cash deposits followed by transfer abroad, or a Lack of account activity events are derived.

Let’s see how we design this using TEM. But first let’s introduce TEM main constructs or building blocks.

TEM Main Constructs

In TEM we have five main constructs, detailed and illustrated in the following sub-sections:

TEM Glossary: The set of tables that stores all glossary concepts of a specific application.

TEM Diagrams: The set of diagrams that describes the events dependencies, and hence the event flow, in an event-driven application. For each situation in TEM, there is a corresponding TEM diagram.

TEM Logic: The set of tables that describes all logic concepts of a specific application.

TEM Principles: The set of assertions and integrity constraints that assure the correctness of a TEM model. The ultimate goal of a TEM model is to be translated into a running application with minimal IT intervention. TEM principles enable validation of the model so it is correct, complete, and consistent and can be automatically translated into an event processing running application.

Although a correct, complete, and valid model requires the completion and specification of all these building blocks, as we will see later on, the most relevant artifacts for a business user in TEM are the diagrams and the logic tables. These can be defined first by non-technical people, while the other artifacts can be specified in parallel or afterwards by business architects or more technical people. We will describe the diagrams and the logic tables in the following sections using our Suspicious account example, but first let’s look at some TEM basic terms required to understand these building blocks.

TEM Basic Terms

Let’s recall and expand on the vocabulary presented in Part 1.

In TEM, an event is an occurrence of something that has happened that is of interest to a business and is published so that we can detect it. We detect and publish it because it may require a reaction on the business’s part.

A raw event is simply the raw output of an event producer. An example of a raw event is a cash deposit into a bank account and the (event) producer is human if a person is doing the depositing or banking transaction systems in most businesses. 

A derived event is the output or conclusion of applying event logic on input events. An example is that a large cash deposit has been made into a bank account. The event logic, in this case, contains the criteria for qualifying a specific cash deposit as being a large one. A situation is a final conclusion from an entire event model and it has at least one consumer who is interested in it. This means that it is published to the consumer who can react to it. In our example, the consumer of the Suspicious situation is the Compliance officer of the bank.

A fact is an atomic piece of data contained in any event, producer, or consumer. For example: customer ID or amount in a Transaction event.

As seen in Part 1, context is one of the main characteristics and singularities in event processing. Context defines the way we partition or group the event occurrences so these partitions can be processed separately. When we derive an event in TEM (see Section on TEM logic below), we do so by applying conditions or expressions against input events in a specified context. TEM distinguishes between two types of context:

Partition by, that is, we group the event occurrences by one or more fact values. For example: customer ID, meaning we group and process together all transactions belonging to a specific customer.

When, that is, we group together the event occurrences based on time or temporal windows. For example, we are looking at event occurrences that happened in a time window of 10 days.

As pointed out, a business user defines or confirms the business logic of an event-driven application by specifying two main artifacts, the TEM diagrams and logic tables. Let’s discuss them in detail, starting with TEM diagrams.

TEM Diagrams

The Event Model diagram is a simple drawing that illustrates the structure of the logic by showing a situation along with the connections of derivations in a top-down manner. At the top of the diagram is a goal which is the situation to be derived. This goal is connected with the raw and derived events that are identified as participants in the situation derivation. This is done in a recursive way until raw events or facts are encountered as depicted in Figure 3 for our Suspicious account example. There is a set of nine icons to express all the relevant terms and relationships in a TEM diagram (see Figure 2)

Figure 2: TEM diagram icons

Each block in the diagram, separated by vertical lines, represents a single event derivation table (detailed in the next section). Each rectangle in the block, separated by a thick black line, corresponds to a row in the respective logic table. The red rectangles in the background represent the context for the specific row. The contexts can be collapsed or expanded. In the case of the Lack of account activity derived event, the context is expanded, and so you can see that the temporal context is initiated either by a Cash deposit or Transfer abroad event, while the events are partitioned by the Account ID of the customer. In our TEM diagram, there are six blocks that correspond to six different logic tables. These will be detailed later on. Dotted lines specify event flows to and from the event-driven system, that is, events flows from producers, the Bank transaction system in our case, or to consumers, the Compliance officer.

The TEM diagram is the major design tool that provides a top down view. All blocks that describe situations or derived events require the definition of the corresponding logic concepts.

Figure 3: TEM diagram for the Suspicious account example

TEM Logic

Logic concepts are the details behind event derivation. The main logic table is the Event Derivation Table (EDT). Each derived event in TEM is detailed in a specific EDT with a name which correlates it to the derived event. An EDT specifies the conditions for generation of occurrences of a specific event. Table 1 details the circumstances under which a new occurrence of a Lack of account activity event is derived.

Table 1: Lack of account activity Event Derivation Table

Event Derivation Tables Structure

The first row in an EDT is its name, which is comprised of the derived event name + “Logic”. So Table 1 contains the details for the derived event Lack of account activity. The table consists of two parts, context and conditions, separated by a vertical red line. The context part consists of two logical sections. The temporal context, represented by When expression, When start, and When end columns; and the segmentation context represented by the Partition by column. For example, Table 1 describes a temporal window initiated each time a Cash deposit or a Transfer abroad event occurs and ends after 20 days. The Partition by context groups the input events by the same Account ID, meaning all Cash deposit and Transfer abroad transactions of the same account are processed together looking for the pattern specified under the Condition part.

Each row in an EDT specifies a different set of conditions and context to derive a new occurrence of the derived event at hand. In order to derive a new occurrence of the derived event at least one of the rows must be satisfied, meaning all conditions in the row must be satisfied for the specific context in that row.

Event Derivation Tables Conditions

The conditions part of an event derivation table consists of three types of conditions. The conditions are logically applied in the following order:

Filter conditions are expressions evaluated against the content of a single event instance. The role of filter conditions is to determine whether an event instance satisfies the filtering condition and should participate in the derivation. For example, the Filter on event column in Table 2, describes a condition on a fact type cash amount of the event Cash deposit. The cash amount value must be equal or larger than the amount allowed for this customer denoted by the customer threshold fact. Input events with cash amount that are smaller than the customer threshold won’t be considered at all and discarded from this event derivation.

Table 2: Large cash deposit EDT

Pattern conditions are expressions on related event occurrences, such as, Detected, Absent, or Thresholds over Aggregations. The role of pattern conditions is to detect the specified relationships among event instances. For example, in Table 1, the Pattern condition describes an absence detection of both Cash deposit and Transfer abroad events, which means that no event occurrence from these two events is detected within the specified context. In Table 3, the Pattern condition is specified by the aggregate Count, and is satisfied only if the number of Large cash deposit events is larger than 3 for the specific context.

Table 3: Frequent large cash deposits EDT

Filter on pattern conditions are expressions on multiple event occurrences, including comparisons, memberships, and time-relationships that result after applying the Pattern condition. The role of the filter on patterns conditions is to filter the pattern result based on conditions among the different events that issue this pattern. For example, in Table 4, the pair of events that come in the sequence of first a Cash deposit and then a Transfer abroad, are then filter to test whether the cash amount is equal or less than the transfer amount. Only those pairs of transactions that satisfy this condition are included in this derivation.

Table 4: Cash deposit followed by transfer abroad EDT

The three types of conditions are optional, meaning they can either appear or not in an EDT, however an EDT is valid only if it contains at least one condition per row. We also don’t restrict the number of conditions per condition type. For example, Table 1 contains two conditions for the Pattern type of condition.

TEM connection is a dependency among EDTs when the conclusion, i.e. derived event, of one EDT is referenced in another EDT. Connections are shown in TEM tables with underlines or hyperlinks. For example, Large cash deposit in Table 3 is underlined since this event itself is the conclusion of the Large cash deposit EDT.

Naming Conventions

Although the names of concepts in TEM can be determined freely by the business user, we use some naming conventions in the logic tables for the sake of clarity. For example, specific fact names used to partition the events input occurrences as well as events (called Domain fact types) , start with a capital letter; other fact names start with a lowercase letter. We also, as you have seen, underline event types in condition columns that have an Event Derivation Table of their own (hyperlinks), to stress the fact that these events are themselves derived from another piece of logic, and enabling to follow paths of inference by pressing these links (as you saw for example Large cash deposit in Table 3).

How do we do a High Level Design in TEM

TEM is best practiced following a top-down methodology starting from a TEM diagram. The design of a new TEM model follows six steps as described next.

Step 1: Draw the Situation

For each situation in the model, identify one or more consumers. In our example, the situation is the Suspicious account derived event emitted to the Compliance officer as shown in Figure 4.

Figure 4: First step in the design of a new TEM

Step 2: Add the Situation’s Skeleton

In this step, we add the context and conditions that lead directly to the situation. In Figure 5 we add the three derived events that contribute directly to Suspicious account.

Figure 5: Second step in the design of a new TEM

Step 3: Go one step further

For each such situation, we construct a diagram that drills down to what is needed to be known or detected in order to derive this situation. Figure 6 shows that we need three branches, one for each derived event in the situation.

Figure 6: Third step in the design of a new TEM

Step 4: Complete the Diagram

Drill down in the diagram until raw events are reached. Figure 3 presents the final step in this example.

Step 5: Construct TEM Derivation Tables

For each block in the diagram, construct a corresponding event derivation table, in a bottom-up approach, starting from raw events all the way to the situation. Figure 7 to Figure 12 show the corresponding EDTs for the blocks in our example TEM diagram.

Figure 7: Relation between a block in the diagram with its corresponding EDT for Large cash deposit derived event

Figure 8: Relation between a block in the diagram with its corresponding EDT for Cash deposit followed by transfer abroad derived event

Figure 9: Relation between a block in the diagram with its corresponding EDT for Frequent cash deposits derived event

Figure 10: Relation between a block in the diagram with its corresponding EDT for Frequent cash deposits followed by transfer abroad derived event

Figure 11: Relation between a block in the diagram with its corresponding EDT for Lack of account activity derived event

Figure 12: Relation between a block in the diagram with its corresponding EDT for Suspicious account derived event

Step 6: Complete the Glossary

A complete TEM model requires all tables to be filled out and completed.

Steps 1-5 can be performed by business analysts or business specialists that possess moderate or no programming skills. Step 6 can be either done by business analysts or specialists, if the organization has accessible meta-data repositories and all the required items are available. Alternatively, it can serve as a starting point for the handshake between the business and IT, if there is a requirement to disambiguate terms, or to create instrumentation to detect events. At the end of Step 6, the model is complete and ready for validation. These steps are done iteratively until the model is complete. Once the model is complete, it is validated. If model violations are found, then these are corrected and the model is validated once more. In some cases, the completion and validation of the model can be done by non-technical people. This process continues iteratively until a valid model is reached. Only then, it can be generated into executable code.

Introducing modifications in TEM

One of the main advantages of TEM resides in the fact that any modification is done only in the model and not in the generated code. This simplifies the process since:

• The new logic is validated along with the entire application, thus avoiding potential inconsistencies that can result from including new logic
• Since there exists only one event derivation table per derived event , the modification is done solely in one place, that is, in the event derivation table it affects and in the relevant entrances in the glossary tables.

In our example, let’s assume that there is a new logical requirements by which an account is to be considered as suspicious. Specifically, we will derive a Suspicious account situation also in the case that the amount in consecutive deposit transactions is constantly increasing for 15 days. Figure 13 shows the new TEM diagram resulting from this addition (highlighted with yellow rectangles in the Figure). Table 5 shows the new EDT for the new derived situation Increasing cash deposits and Table 6 shows the new entrance for this derived event in the EDT for the Suspicious account situation (see row number 4 in the EDT). The new model is updated, validated, and the code is generated in a matter of minutes!

Figure 13: New diagram for the Suspicious account example

Table 5: New EDT for Increasing cash deposit

Table 6: New EDT for Suspicious account situation

Wrap-up

Today is the right time for TEM

Companies that understand CEP have more and better real-time intelligence than those that don’t understand it. The use of CEP will expand further as the pace of business accelerates, more data becomes available in real-time[6], and business people demand better situation awareness. CEP will inevitably be adopted in multiple places within virtually every company[7].

If you realize that you need complex event processing for real-time intelligence in your organization and you want to implement complex event processing, then TEM is the right way to do it.

If You are Interested

See O. Etzion and B. von Halle, Modelling the event driven world, ER2013 tutorial
http://www.slideshare.net/opher.etzion/er-2013-tutorial-modeling-the-event-driven-world.

Also available is a short high level overview movie:
https://www.youtube.com/watch?v=9zjy8wngy5Y&feature=youtu.be

Authors: Dr. Fabiana Fournier, Researcher at IBM Research & Barbara von Halle, Managing Partner of KPI LLC.

See more articles by Barbara von Halle. 

 References:

1. KPI stands for Knowledge Partners International LLC
2. Introducing the Next Horizon: The Event Model (TEM)
3. The Forrester Wave™: Complex Event Processing ( CEP) Platforms, Q3 2009
4. http://www.complexevents.com/2013/09/17/understanding-real-time-intelligence
5. http://www.complexevents.com/2012/07/25/does-anyone-care-about-event-processing/
6. We are using the term “real time” loosely, to include “near-real-time” or “business real time.”
7. Hype Cycle for Operational Technology, 2013 (report # G00246894)