BRE Pipeline Framework v1.5.4 now available on CodePlex – further caching functionality provided

I have now uploaded the BRE Pipeline Framework v1.5.4 installer to the CodePlex project page.  This version is as always fully backwards compatible with BRE Policies created with previous versions, and installing it over an existing version simply requires uninstalling the currently installed version, installing the new version, and importing the new vocabularies from the program files folder (default location is C:\Program Files (x86)\BRE Pipeline Framework\Vocabularies).

In a previous post I  discussed a pattern that allows for the temporary persistence of context properties on a request message in a request/response send port, and then reapplying those context properties on the response message before it gets sent to the BizTalk message box.  The beauty of this pattern is that it allows you to enrich response messages with state from the request message without the need for orchestration.  The temporary persistence store that I used for these context properties is based on the .Net MemoryCache class.

One of the features of the MemoryCache class is that it is really easy to set expiry times on cached items so you aren’t forced to explicitly remove items from the cache (though you still have the ability to), and setting expiry times and explicitly deleting cached items are features provided for in the BRE Pipeline Framework.

The MemoryCache class also has a feature whereby if the server is under heavy pressure for memory it will automatically drop items from the cache.  This is great when the items in the cache are truly temporary and you have a means of reloading the item in the cache, however doesn’t work very well for the pattern described in my previous post which depends on the cached items being available.  To get around this I have taken advantage of the Priority property in the CacheItemPolicy class, which allows you to override this default behavior, instructing .Net not to automatically remove your cached items.

This feature is introduced in the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary v1.1’s vocabulary definition called SetCachingPriority.  You have two priority values that you can apply as below.

• Default – this implies that .Net should remove cached items automatically if under memory pressure.  If you don’t use the SetCachingPriority vocabulary definition then this value will be automatically chosen.  Cached items can also be removed based on expiry time or based on explicit removal.
• NotRemovable – this instructs .Net not to ever remove cached items, even if under memory pressure.  Cached items can only be removed based on expiry time or based on explicit removal.

An example usage of the SetCachingPriority vocabulary definition is in the screenshot below.

Using the BRE Pipeline Framework for caching

One of the most compelling features that was made available in v1.5 of the BizTalk BRE Pipeline Framework was the ability to cache context properties or any custom strings and then fetch them back later on.  I purposely chose not to blog about this feature till now because I wanted to employ it in a real life scenario to prove it’s value first which I have now.

The caching features are contained within the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary that was shipped with v1.5 of the framework.  If you haven’t already published the vocabulary you can find the exported XML for this vocabulary in the Vocabularies subfolder in the BRE Pipeline Framework program files folder (typically “C:\Program Files (x86)\BRE Pipeline Framework”) and can use the Rules Engine Deployment Wizard to publish it.  After reading this blog post you will have an idea of what caching features are available in the BRE Pipeline Framework, how the framework achieves the caching, and you’ll also be exposed to a summary description of each vocabulary definition in the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

 

Caching Context Properties

The most common use for the caching feature in the BRE Pipeline Framework will be to cache context properties and then to reapply them.  Let me paint a picture to you, which will seem all too common to those experienced with BizTalk and who strive to create messaging only applications in situations where no orchestration is warranted.  You need to call a WCF service which returns either a valid response which doesn’t contain any of the identifiers from the request, or a SOAP fault (unless SOAP faults were generated in a custom fashion you typically only see an error message with no details that help you to identify which request the fault corresponds to), and you need to write the status of the response to a database along with the identifiers from the request for correlation purposes.

BizTalk Server messaging is stateless by nature so you only have access to the message currently being processed and not historical messages when you are performing your inbound mapping on the request/response send port.  Thus it wouldn’t typically be possible to satisfy the requirement of writing the response to the database with the identifiers from the request unless you used an orchestration that persisted the state of the original request and then executed a multi-source map that used the original request and the response message to transform to the destination message.  I consider orchestration to be overkill for such a scenario and prefer to go down a messaging only route.

The answer to the above scenario is to ensure that all the identifiers in the request message are promoted to context properties on the message that is directed to the WCF service send port, use the BRE Pipeline Framework to cache the context properties from the request, and to then use the BRE Pipeline Framework to reapply those context properties on the response message.  You could then use the Context Accessor pipeline component/functoids to insert the context properties into the target message body via a map, or use property demotion (see this TechNet article for more details) to achieve the same result.

Below is an example rule where the BRE Pipeline Framework is used to cache the RequestID and Requestor context properties on the request message on a send port.  In the below case the BRE Pipeline Framework will throw an exception if the RequestID context property doesn’t exist on the request message, but will just ignore and carry on if the Requestor context property doesn’t exist on the request message.  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the AddCustomContextPropertyToCache definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

Reapplying these context properties is easy as per the below screenshot in which the RequestID context property will be promoted (potentially for routing) while the Requestor context property is merely written to the context of the message (thus not available for routing).  In the below case the BRE Pipeline Framework will throw an exception if the RequestID context property doesn’t exist in the cache, but will just ignore and carry on if the Requestor context property doesn’t exist in the cache.  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the ReapplyCachedContextProperty definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

How does this work?  Under the hood the BRE Pipeline Framework makes use of the .NET MemoryCache class (specifically used because of it’s cached item auto-expiry features) which ships with .NET 4.0 onwards.  Each context property that you choose to cache on a given message gets added to a Dictionary in which the context property namespace#name is used as the key and the context property value is the value.  The Dictionary is then added to the Memory Cache with the key based on the BTS.TransmitWorkID context property of the request message.  The reason the BTS.TransmitWorkID context property is used as the key is because this context property is automatically generated on request-response send ports and response messages will always have the same value for this context property as the request message, thus this value can be used to write to and read from the MemoryCache.  Each item in the MemoryCache by default has an expiry time of 30 minutes (if not explicitly removed before), after which they will automatically be removed from the cache to ensure that RAM isn’t permanently earmarked for the caching.   The MemoryCache is isolated to a given AppDomain thus is unique per host instance (thus if you cache a value in one host instance you can’t reapply it in another, and is also not shared across machines even if host instances pertain to the same host) and if a single host instance is used for both receive/send ports and orchestration then the receive/send ports will have one MemoryCache while the orchestration will have another (see this fantastic article by Saravana Kumar if you are interested in reading more about BizTalk and AppDomains).

 

Caching custom strings

The caching functionality in the BRE Pipeline Framework can also be used to cache any string and to fetch it later.  A common use for this would be to cache a value that has been read in from a relatively expensive data source to read from such as SQL Server or the SSO.  Doing so means that you don’t have to hit the database each time your rule is executed.

Below is an example in which the value ReadInFromSQL (the hardcoded value could be replaced with a vocabulary definition that fetches the value from the relevant data source) is cached using the key Configuration Info, the cached entry being set to expire in 5 hours (note the time units is an Enum so you can choose whichever applicable unit you want).  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the AddCustomStringToCache definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

Fetching the cached value is easy as per below.  In the below case the BRE Pipeline Framework will return Null if the key isn’t found in the cache (other options are to throw an exception or return a blank string).  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the GetCustomStringFromCache definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

The primary difference between the way that caching custom strings works vs. caching context properties works is that with the custom strings each value is cached separately in the MemoryCache rather than within a single dictionary per message which contains all the context properties for that message, and it is up to you to explicitly state the expiry time and the key for each value you choose to cache.

 

Walkthrough of vocabulary definitions in the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary

• AddCustomContextPropertyToCache – Allows you to specify a context property against the current message that should be cached.
• AddCustomStringToCache – Allows you to specify any string that should be cached.  You will also need to supply a key and an expiry time for this cached entry.
• CacheAllContextProperties – Allows you to cache all the context properties against the current message.  Can help  you to future proof your rule since adding more context properties to the message will not require a change to the rule.
• ChangeKeyContextProperty (advanced feature)- By default when you cache context properties they are all collapsed into a single Dictionary which is added to the MemoryCache with a key based on the BTS.TransmitWorkID context property which is great for synchronous request/response type scenarios.  If you want to cater for asynchronous scenarios whereby you might be trying to reapply the context property on a message processed on a totally separate port then you have the ability to nominate another context property as the key in the MemoryCache.  Note that to reapply the context properties on the target message it must already have the same context property value for the key context property as the original message.  Also note that this vocabulary definition would need to be contained within the action for the rule that adds to the cache and the rule that gets from the cache.
• DeleteContextFromCacheIfItStillExists – Allows you to explicitly delete context properties from the cache for the current message.  You might want to do this after you have reapplied the context properties from the cache rather than wait for the expiry time to pass to reduce strain on RAM.
• DeleteCustomStringFromCache – Allows you to explicitly delete a custom string from the cache based on the specified key.  You might want to do this when you are sure you no longer need the cached string rather than wait for the expiry time to pass to reduce strain on RAM.
• GetCustomContextPropertyFromCache – Allows you to fetch a cached context property.  Note that this definition doesn’t reapply the cached context property to the current message but rather is just used to fetch the value, potentially for inspection in conditions or to use as a parameter in another definition within a rule’s action.
• GetCustomStringFromCache – Allows you to fetch a cached string which can be used as a string parameter in other definitions in a rule’s condition or action.
• ReapplyAllCachedContextProperties – Allows you to reapply all the cached context properties against the current message.  Be wary of using this in tandem with the CacheAllContextProperties definition, as this might result in an attempt to override out of the box context properties such as BTS.MessageType or BTS.MessageID etc…
• ReapplyCachedContextProperty – Allows you to reapply a specific cached context property against the current message.
• UpdateCacheExpiryTime – Allows you to override the default expiry time for cached context properties, the default being 30 minutes.

Using the BRE Pipeline Framework to execute maps dynamically

A couple of months ago I released the BRE Pipeline Framework v1.5 (since superseded by v1.5.1) to CodePlex.  One of the new features in this version of the framework is support for dynamic transformation.  In this blog post I’ll explain some scenarios in which this feature might be useful to you and show you how you can use the BRE Pipeline Framework to execute your maps dynamically.

 

Why you’d want to use the BRE Pipeline Framework for Dynamic Transformation

The first reason to take advantage of the BRE Pipeline Framework for Dynamic Transformation is to build messaging only applications that support hot deployments for maps.  A hot deployment from a BizTalk perspective is effectively one in which a BizTalk application doesn’t need to be stopped during a redeployment of your map assemblies, the only requirement being that the relevant host instances are restarted after the deployment is complete.  In a messaging application with maps on your receive/send port you might find that BizTalk complains if you try to import an MSI with maps that are used on receive/send ports if the application is in a running state, in fact you might even need to delete the receive/send ports altogether or at the very least remove the maps from the ports before your redeployment takes.  A hot deployment would not require more than a momentary outage of your application while host instances restart which fits in well with an application that is not tolerant to outages.

The second reason to take advantage of the BRE Pipeline Framework for Dynamic Transformation is for transformation selectivity.  The way BizTalk works if you apply inbound/outbound maps on your receive/send ports is that it chooses the first map for which the source schema matches the message type of the message in question.  If you apply two maps with the same source schema on a receive/send port there is no way for you to specify any additional conditions that determine which map executes, the second map will always be ignored.  With the BRE Pipeline Framework you can apply complex conditions which include combinations of checking message body content through XPath statements or regexes, checking values in the message context, checking values from SSO configuration stores or EDI trading partner agreements, or even based on the current time of the day…and the best part is that if the selectivity functionality already exists in the BRE Pipeline Framework (which it does in all the mentioned scenarios and many more) then you can achieve all of this with 0 lines of code, and if the functionality doesn’t exist then the framework provides extensibility points to cater for this.  Another great advantage of using the BRE to choose which maps get executed is that you can always change the selectivity rules at runtime if required without any code changes, which can offer you much flexibility for applications that are not tolerant to outages.

The third reason to take advantage of the BRE Pipeline Framework for Dynamic Transformation is so that you can chain maps.  As mentioned above, on a BizTalk receive/send port only the first matching map will execute.  It is not possible to execute multiple maps sequentially within a port within a single direction (you can of course specify one inbound and one outbound map thus two if your port is a two-way port).  In a messaging only solution if you receive a message and send it out on a send port then the maximum number of maps you can execute is 2, one on the receive port in the inbound direction, and one on the send port in the outbound direction.  For the most part this is adequate but there might be some scenarios where this isn’t.  With the BRE Pipeline Framework you can specify as many maps to execute sequentially within a single rule or across multiple rules (make sure you set priorities across the different rules to guarantee the required map execution order), or you can execute a map in your BRE Policy and then apply one on your port as well (keep in mind that for an inbound message the pipeline executes before the port maps and the reverse is true for outbound messages).  Beyond just chaining maps, you can also chain other Instructions contained within the BRE Pipeline Framework together with your maps, so you could for example execute a map dynamically then perform a string find/replace against the message body.

Another possible benefit of the dynamic transformation is avoiding inter-application references, which is possibly one of the most painful aspects of BizTalk Server. When you want to pass a message from one application to another you can have almost no choice but to have one application reference the other (the one being referenced containing the schema). Inter-application referenced make deployments a lot more difficult since you can’t update the referenced application without deleting all the referencing applications first (forget about hot deployments altogether, this is the other extreme). To get around this problem you could potentially have separate schemas in each application, create a map in an assembly that only gets deployed to the GAC rather than to the BizTalk Management Database, and then use the BRE Pipeline Framework to transform the message in a pipeline (in either application, it shouldn’t matter). This could allow you to create more decoupled applications with much easier deployment models.

The aforementioned benefits can also be achieved through the use of the ESB Toolkit.  The main difference between the BRE Pipeline Framework and the ESB Toolkit is that the former is intended to provide utility whereas the latter is more about implementing the routing slip pattern with added utility as well.  The ESB Toolkit comes with a fair learning curve as there are a whole lot of new concepts to wrap your head around such as itineraries, resolves, messaging extenders, orchestration extenders etc… and you’ll find that at least the out of the box utility can be quite limited (there are of course community projects that have improved on these lackings).  I definitely see valid scenarios in which either framework should be used, and wouldn’t consider them to be competing frameworks…they could even be used in tandem.

One final reason that I can think of off the top of my head is traceability.  Given that the BRE Pipeline Framework caters for tracing based on the CAT Instrumentation framework and also provides rules execution logs (see this post for more info) you can always tell why a map was chosen for a given message.  This can be especially handy when you are debugging a BizTalk application.

Just like with the ESB Toolkit executing a map dynamically within your pipeline goes against one of the best practice principles of building streaming pipeline components, so please carefully evaluate the pros and cons of using this feature before implementing it.

 

Implementing Dynamic Transformation with the BRE Pipeline Framework

To illustrate how to use the BRE Pipeline Framework to execute maps dynamically I will provide you with and walk you through a simple example solution.  The solution contains three schemas, one called PurchaseOrder, one called PurchaseOrderEDI, and one called PurchaseOrderXML.  All the schemas contain an element (with varying names) which hold the purchase order number.  The PurchaseOrder schema also contains an additional node called OrderType which is also linked to a promoted property of the same name.  The rule we want to put in place for PurchaseOrder messages is that if the value of the OrderType context property is XML then we want to execute a map that converts the message to a PurchaseOrderXML.  If the result of an XPath query against the OrderType node is EDI then we want to execute a map that converts the message to a PurchaseOrderEDI.

On to the implementation…   You will of course need to download and install at least v1.5.1 of the framework from the CodePlex site and import the required vocabularies from the program files folder.  Once done you will need to create a receive pipeline (receive or send are both supported) and drag the BREPipelineFrameworkComponent from the toolbox (you’ll need to add it to your toolbox by right clicking within the toolbox choosing “Choose items” and selecting the component from within the pipeline components tab if it isn’t already in your toolbox) to the validate pipeline stage (you can choose any stage except disassemble/assemble).  The only parameter you have to set on the pipeline component is ExecutionPolicy which specifies which BRE Policy will be called to resolve the maps to execute (you could optionally specify the ApplicationContext parameter if you plan on calling the BRE Policy from multiple pipelines and you want some rules to only apply for certain pipelines).   For the purpose of this example we will use an XML Disassembler component prior to the BREPipelineFrameworkComponent and ensure the StreamsToReadBeforeExecution parameter on the BREPipelineFrameworkComponent is left at its default value of Microsoft.BizTalk.Component.XmlDasmStreamWrapper so that we will be able to inspect context property values promoted by the XML Disassembler (see this post for more info).

Once all the components are deployed to BizTalk we’ll create a receive location that picks up a file and makes use of the aforementioned receive pipeline, as well as a file send port that subscribes to messages from this receive port.  Finally we’ll create the BRE Policy which contains two rules.

The first rule is used to transform messages to the PurchaseOrderXML message format as below.  The rule is made of a single condition which uses the GetCustomContextProperty vocabulary definition from the BREPipelineFramework.SampleInstructions.ContextInstructions vocabulary to evaluate the value of a custom context property.

The second rule is used to transform messages to the PurchaseOrderEDI message format as below.  The rule is made of a single condition which uses the GetXPathResult vocabulary definition from the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary to evaluate the value of a node from within the message body with the use of an XPath statement.

Both of the aforementioned rules make use of the TransformMessage vocabulary definition from the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary in their actions to apply a map against the message.  The input format of the vocabulary definition is as follows – Execute the map {0} in fully qualified assembly {1} against the current message – {2}.  The first parameter in this vocabulary definition is the fully qualified map name (.NET namespace + .NET type), and the second parameter is the fully qualified assembly name including the assembly version and the PublicKeyToken (you can execute gacutil -l with the assembly name to get the fully qualified assembly name from a Visual Studio Command Prompt).  The third parameter is used to specify what sort of validation is performed against the input message before executing the map and is an enumeration with the below values.

• ValidateSourceSchema – This option will validate the current messages BTS.MessageType context property against the source schema of the specified map.  If they don’t match up then an exception is thrown rather than the map being executed.  If a message type is not available then an exception will be thrown.
• ValidateSourceSchemaIfKnown – This option will validate the current messages BTS.MessageType context property against the source schema of the specified map.  If they don’t match up then an exception is thrown rather than the map being executed.  If a message type is not available then no exception will be thrown and the map will execute.
• DoNotValidateSourceSchema – This option will not result in any validation of the BTS.MessageType context property on the current message and will result in the map being executed regardless, possibly resulting in a runtime error during execution of your map or an empty output message.  If a message type is not available then no exception will be thrown and the map will execute.  I haven’t experimented with this myself but this might allow you to create generic maps which apply generic XSLT against varying input messages to create a given output message format.  If anyone decides to experiment with this then please do let me know your results.

That’s all that is required to stitch together a solution making use of the BRE Pipeline Framework which involves dynamic transformation.  If you push through a PurchaseOrder message with an OrderType of XML then it will get converted to a PurchaseOrderXML message, if the OrderType is EDI then it will get converted to a PurchaseOrderEDI message, and if the OrderType is anything else then the message will remain a PurchaseOrder as expected.

As previously mentioned the BRE Pipeline Framework comes along with a  lot of traceability features (also documented here).  If you set the CAT Instrumentation Framework Controller to capture pipeline component trace output you will get information such as the below which tells you which map is getting executed, what the source message type is, and what the destination message type is.

EventTrace
[3]1FF4.2690::08/16/2014-22:13:01.245 [Event]:TRACEIN: BREPipelineFramework.PipelineComponents.BREPipelineFrameworkComponent.TraceIn() => [102f63bb-2c86-4213-a892-2a5175569469]
[3]1FF4.2690::08/16/2014-22:13:01.245 [Event]:START -> 102f63bb-2c86-4213-a892-2a5175569469
[3]1FF4.2690::08/16/2014-22:13:01.245 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - BRE Pipeline Framework pipeline component has started executing with an application context of , an Instruction Execution Order of RulesExecution and an XML Facts Application Stage of BeforeInstructionExecution.
[3]1FF4.2690::08/16/2014-22:13:01.245 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - BRE Pipeline Framework pipeline component has an optional Execution policy paramater value set to BREMaps.
[3]1FF4.2690::08/16/2014-22:13:01.245 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - BRE Pipeline Framework pipeline component has an optional tracking folder paramater value set to c:\temp.
[3]1FF4.2690::08/16/2014-22:13:01.245 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Inbound message body had a stream type of Microsoft.BizTalk.Component.XmlDasmStreamWrapper
[3]1FF4.2690::08/16/2014-22:13:01.245 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Inbound message body stream was not seekable so wrapping it with a ReadOnlySeekableStream
[3]1FF4.2690::08/16/2014-22:13:01.246 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Reading stream to ensure it's read logic get's executed prior to pipeline component execution
[1]1FF4.2690::08/16/2014-22:13:01.255 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.CachingMetaInstructions to Execution Policy facts.
[1]1FF4.2690::08/16/2014-22:13:01.255 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.ContextMetaInstructions to Execution Policy facts.
[1]1FF4.2690::08/16/2014-22:13:01.255 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.HelperMetaInstructions to Execution Policy facts.
[1]1FF4.2690::08/16/2014-22:13:01.255 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.MessagePartMetaInstructions to Execution Policy facts.
[1]1FF4.2690::08/16/2014-22:13:01.255 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.XMLTranslatorMetaInstructions to Execution Policy facts.
[1]1FF4.2690::08/16/2014-22:13:01.265 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Executing Policy BREMaps 1.0
[0]1FF4.2690::08/16/2014-22:13:01.277 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Adding Instruction BREPipelineFramework.SampleInstructions.Instructions.TransformationInstruction to the Instruction collection with a key of 0.
[0]1FF4.2690::08/16/2014-22:13:01.277 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Starting to execute all MetaInstructions.
[0]1FF4.2690::08/16/2014-22:13:01.277 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Executing instruction BREPipelineFramework.SampleInstructions.Instructions.TransformationInstruction.
[0]1FF4.2690::08/16/2014-22:13:01.277 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Applying transformation BREMaps.PurchaseOrder_To_PurchaseOrderXML,   BREMaps, Version=1.0.0.0, Culture=neutral, PublicKeyToken=21bb7669ee013ee3 to the message
[0]1FF4.2690::08/16/2014-22:13:01.277 [Event]:102f63bb-2c86-4213-a892-2a5175569469 - Message is being transformed from message type http://BREMaps.PurchaseOrder#PurchaseOrder to message type http://BREMaps.PurchaseOrderXML#PurchaseOrderXML
[0]1FF4.2690::08/16/2014-22:13:01.278 [Event]:END <- 102f63bb-2c86-4213-a892-2a5175569469: 32ms
[0]1FF4.2690::08/16/2014-22:13:01.278 [Event]:TRACEOUT: BREPipelineFramework.PipelineComponents.BREPipelineFrameworkComponent.Execute(...) = "102f63bb-2c86-4213-a892-2a5175569469"

If you set the TrackingFolder parameter on the BREPipelineFrameworkComponent pipeline component to a valid folder then you will get output like the below (note this is just an excerpt) which provides valuable information telling you which BRE rules get fired and why.

CONDITION EVALUATION TEST (MATCH) 16/08/2014 10:13:01 p.m.
Rule Engine Instance Identifier: f2c966cf-b248-4e94-a96a-99d110d59a9b
Ruleset Name: BREMaps
Test Expression: BREPipelineFramework.SampleInstructions.MetaInstructions.ContextMetaInstructions.GetContextProperty == XML
Left Operand Value: XML
Right Operand Value: XML
Test Result: True

CONDITION EVALUATION TEST (MATCH) 16/08/2014 10:13:01 p.m.
Rule Engine Instance Identifier: f2c966cf-b248-4e94-a96a-99d110d59a9b
Ruleset Name: BREMaps
Test Expression: BREPipelineFramework.SampleInstructions.MetaInstructions.HelperMetaInstructions.GetXPathResult == EDI
Left Operand Value: XML
Right Operand Value: EDI
Test Result: False

RULE FIRED 16/08/2014 10:13:01 p.m.
Rule Engine Instance Identifier: f2c966cf-b248-4e94-a96a-99d110d59a9b
Ruleset Name: BREMaps
Rule Name: Map To XML Format
Conflict Resolution Criteria: 0

One more thing worth mentioning is that once the BRE Pipeline Framework executes a map, it will promote the output message type to the BTS.MessageType context property just as a map on a port would.  This means that you can reliably create routing filters based on the BTS.MessageType context property if you make use of the dynamic transformation feature in the BRE Pipeline Framework.

The aforementioned solution is available for download here.  I’ve included the Visual Studio solution with the source code as well as an export of the BRE Policy, as well as an MSI installer which will create the example application for you (you might need to reconfigure the folders in the file receive and send port based on where you unzip the solution and you might have to provide full control permissions on these folders to your host instance user).  I’ve also included some example XML messages for your convenience.

Happy transforming.

BizTalk Business Rules and complex XML structures

The BizTalk business rules engine is a very powerful asset in an integration specialist’s toolbox, but it can be tricky when you are trying to operate with complex schemas as you try to get your head around how it works.  The problem I will try to illustrate here is that the default schema vocabulary definitions will not allow you to spread your conditions and actions across different records in an XML schema if they are both contained within a single instance of a repeating reccord.

For this example, lets say that we want to evaluate a batch of customers, and for those that are Infinity years old (ok, not the best example) we are going to set their AwardsLevel to Gold.  We are going to deal with the below XML schema (note that RegularCustomer is unbounded and that Age is optional) and XML input file.

Of course Chuck Norris is Infinity!

Let’s add some vocabulary definitions to get a customer’s age and to set his awards level.  To start with let’s just select the elements we’re interested in from the schema view when creating the definitions.

It’s now time to define our policy as below.

We need to test out the policy we’ve just created so lets save it first.  Now right click on the policy version and choose test policy.  You will need to click on the schema listed under XML Documents and click the add instance button, choosing to point towards our test XML file.

When you execute the test you’ll see that the results are somewhat unexpected, with every customer being awarded gold status, even the ones younger than Infinity.

It’s time to do some investigation.  I’ve found the quickest way to understand what makes a policy tick is to export it using the Business Rules Engine Deployment Wizard (you’ll find the shortcut under your start menu in the BizTalk Server folder) and to inspect the XML file.

The first thing of note is that in the bindings section there are two XML document instances, each with their own XPath selectors defined.  The first selector is to the CustomerDetails record and the second to the AwardsDetails record.  When you dig into the GoldAwards rule you’ll notice that the xmldocumentmember element in the <if> record has a xmldocumentref attribute of xml_31 (the CustomerDetails selector) and the xmldocumentmember element in the <then> record has a xmldocumentref attribute of xml_32 (the AwardsDetails record).  It’s pretty obvious that our if and then statement are not scoped to the same RegularCustomer record, and the path to enforcing the scope is to ensure that the xpath statements executed in the <if> and <then> records of our rule use the same selector.

So this time lets revisit our vocabulary definitions.  By default the XML selector and XML field values for GetAge and SetAwardsLevel look like the below.

GetAge Selector – /*[local-name()=’RegularCustomers’ and namespace-uri()=’http://BusinessRulesTest.BusinessRuleSchema’%5D/*%5Blocal-name()=’RegularCustomer&#8217; and namespace-uri()=”]/*[local-name()=’CustomerDetails’ and namespace-uri()=”]

GetAge Field – *[local-name()=’Age’ and namespace-uri()=”]

SetAwardsLevel Selector – /*[local-name()=’RegularCustomers’ and namespace-uri()=’http://BusinessRulesTest.BusinessRuleSchema’%5D/*%5Blocal-name()=’RegularCustomer&#8217; and namespace-uri()=”]/*[local-name()=’AwardsDetails’ and namespace-uri()=”]

SetAwardsLevel Field – *[local-name()=’AwardsLevel’ and namespace-uri()=”]

Let’s edit the definitions (you’ll need to make a new vocabulary version) so that the  XML selector and XML field values for GetAge and SetAwardsLevel look like the below (note that the selectors are now the same being scoped to the RegularCustomer record while the field digs into the required child records).

GetAge Selector – /*[local-name()=’RegularCustomers’ and namespace-uri()=’http://BusinessRulesTest.BusinessRuleSchema’%5D/*%5Blocal-name()=’RegularCustomer&#8217; and namespace-uri()=”]

GetAge Field – *[local-name()=’CustomerDetails’ and namespace-uri()=”]/*[local-name()=’Age’ and namespace-uri()=”]

SetAwardsLevel Selector – /*[local-name()=’RegularCustomers’ and namespace-uri()=’http://BusinessRulesTest.BusinessRuleSchema’%5D/*%5Blocal-name()=’RegularCustomer&#8217; and namespace-uri()=”]

SetAwardsLevel Field – *[local-name()=’AwardsDetails’ and namespace-uri()=”]/*[local-name()=’AwardsLevel’ and namespace-uri()=”]

Let’s change the rule such that it uses the new version of our vocabulary (note that a rule refers to a specific version of a vocabulary definition, it will not automatically choose the latest version of the vocabulary).  If we give it a test now we’ll see that only the RegularCustomer records with an Age of Infinity have Gold status :).  You’ll also see if you export the policy to XML that there is now only one xmldocument record in the bindings section and that both the <if> and the <then> use this instance.

Chuck Norris = Gold

There is one last important thing to note.  What if we passed in a RegularCustomer record which for whatever reason did not contain an Age element at all (remember that it is defined as an optional element)?  You’ll see that the rule fails and throws an exception, and thus the entire policy fails to execute with the below error.

RuleEngineRuntimeFieldNotDefinedException –  Field “*[local-name()=’CustomerDetails’ and namespace-uri()=”]/*[local-name()=’Age’ and namespace-uri()=”]” does not exist in XML document “BusinessRulesTest.BusinessRuleSchema”, selector “/*[local-name()=’RegularCustomers’ and namespace-uri()=’http://BusinessRulesTest.BusinessRuleSchema’%5D/*%5Blocal-name()=’RegularCustomer&#8217; and namespace-uri()=”]”.

This here blog post (I suggest that everyone that intends to use the BRE read this as you will undoubtedly hit this problem sooner or later) suggests that we can get around this problem by editing the selector to ensure that the relevant fields exist.  In this case we need to ensure that the Age and AwardsLevel (might as well cater for this too) fields exist.  We can do this by editing the xpath selectors to look like below.

/*[local-name()=’RegularCustomers’ and namespace-uri()=’http://BusinessRulesTest.BusinessRuleSchema’%5D/*%5Blocal-name()=’RegularCustomer&#8217; and namespace-uri()=”][CustomerDetails/Age and AwardsDetails/AwardsLevel]

The catch here is that if you don’t update the xpath selectors for both the GetAge and SetAwardsLevel definitions then you’ll find that you are back to square one with all the RegularCustomer records getting a Gold AwardsLevel.  Thus one can conclude that the xpath selectors must match exactly when attempting to define a scope between rule actions and conditions.