Tag Archive: BRE Pipeline Framework


One of the core tenets of RESTFul architecture is that APIs should support content negotiation.  That is to say that the service consumer should be able to tell the service through the use of the Accept HTTP header what format the response message should be returned in.  On the flip side, it is commonly accepted that RESTFul APIs should support a range of request formats; typically XML and JSON at the very least, and that the Content-Type HTTP header is used to convey the format.

BizTalk Server 2013 R2 delivered the new JSON Decoder and Encoder pipeline components that at last treat JSON as a first class format in BizTalk Server.  Internally, BizTalk maps will still only work with XML as will a range of other BizTalk functions, however at the very least JSON is now accepted on the outskirts.  It is possible to get JSON support in older versions of BizTalk however you will have to create custom pipeline components to cater for this, since Microsoft has only added out of the box functionality in BizTalk Server 2013 R2.

Unfortunately, the traditional means of employing these pipeline components comes with some rigidity.  If you want to expose a RESTFul API using a WCF-WebHttp receive location then you will need to choose whether or not to include a JSON Decoder pipeline component in your receive pipeline.  On the flip side, you will need to choose whether to use a JSON Encoder on your send pipeline.  There is no way to dynamically decide whether to employ the JSON Decoder and Encoder based on the Content-Type and Accept header on the request message without employing an orchestration.  And it gets even worse…what if you wanted to build an API that also supported an HTML response type?

I’m not against the use of orchestration when they are used to provide some form of workflow or process orchestration, however I do not like being forced to use orchestration due to technical constraints.

I have addressed this problem for the most part using the BRE Pipeline Framework.  Because JSON decoding and encoding is only supported by BizTalk Server 2013 R2, I have addressed this problem in a custom MetaInstruction, or an extension to the framework, rather than making a change to the core framework.  Note that this extension requires that you have the BRE Pipeline Framework v1.6.0 or above installed on your machine.  This new functionality is catered for via the new BREPipelineFramework.JSONInstructions vocabulary.

Vocab

To get started download the installer for the extension and install it on your BizTalk 2013 R2 machine.  Install the vocabulary using the Business Rules Engine Deployment Wizard, the vocabulary file being found in the program files folder which by default is “C:\Program Files (x86)\BREPipelineFramework.JSONInstructions”.

Here’s a quick rundown on what these vocabulary definitions do.

  • DecodeJSON – This vocabulary definition executes the JSON Decoder against the message.  You will need to specify a root node name and namespace with which the decoded XML will be wrapped in.
  • EncodeJSON – This vocabulary definition executes the JSON Encoder against the message.  You will need to specify whether the root node will be stripped off prior to encoding or not.
  • ExecuteXSLTTransform – This vocabulary definition will execute an XSLT transformation based on an XSLT file which you must provide a file location for.  This is handy if you want to transform to/from HTML.
  • AssessContentType – This vocabulary definition lets you assess what the content type of your request message is so you can decide how to process it.  It will first attempt to derive the content type by studying the Content-Type HTTP header on the request.  If no Content-Type header is available then it will probe the first character of the message to determine if it is XML or JSON.
  • CacheAcceptHeader – Since we need to assess the Accept header value when choosing how to encode the response message, we must cache the Accept header value while processing the request.  If we did not do this then we would not have access to the value later.
  • AssessCachedAcceptHeader – This vocabulary definition lets you assess what the desired content type is based on the cached Accept header.  This can be executed on the send pipeline that is used to encode the response message.
  • GetCachedAcceptHeader – This vocabulary definition will return the raw value of the cached accept header.  This can be executed on the send pipeline that is used to encode the response message.  Only use this if you aren’t happy with the logic used by the AssessCachedAcceptHeader vocabulary definition.

The following screenshot demonstrates how we can dynamically decode incoming request messages based on the derived content type of the request message.  Note that if the message was XML instead of JSON then this rule would not first.  Note as well that we are caching the Accept header value for later use…theoretically this should be done in a separate rule which will execute regardless of the request content type but I have collapsed the two actions into one rule for brevity.

Rcv JSON

The following screenshot demonstrates the assessment of the cached Accept header value to dynamically encode the response message to JSON.

Snd JSON

And the next screenshot similarly dynamically transforms the response to HTML by executing custom XSLT.

Snd HTML

So how do you make use of this new vocabulary?  The easiest way to do this is to put the fully qualified class/assembly name of the custom MetaInstruction in the CustomMetaInstructionSeparatedList parameter (which takes in a ; delimited list) of the BREPipelineFrameworkComponent pipeline component as below.  Once you’ve done this then the vocabulary will be available for use in the ExecutionPolicy.

PC Config

The fully qualified class/assembly name value for the JSON extension is as below.

BREPipelineFramework.JSON.JSONMetaInstruction, BREPipelineFramework.JSON, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc

So what are the shortcomings of the current solution?

  • The response content type derivation based on the cached Accept header doesn’t currently go as far as is normally expected by RESTFul standards.  These standards dictate that multiple content types can be specified and each can be given priority weightings.  The solution I’ve put in place is just a v1, and it will always prioritize as follows regardless of explicit priorities – JSON > XML > URLEncodedForm > CSV > HTML > Text > Other.  If you don’t like this logic then you can always make your own choice by assessing the raw Accept header value with the GetCachedAcceptHeader vocabulary definition instead of the AssessCachedAcceptHeader vocabulary definition.
  • I have not yet found a context driven way of overriding the Content-Type HTTP header on the response message.  This will always be set to Application/XML, unless you have set an alternate value on the Messages tab of your receive location’s configuration in which case it will always be that alternative value.  For now the only way I can think of overriding this is via a WCF behavior.
  • It only works with BizTalk Server 2013 R2, but you could easily get it to work with BizTalk 2013 by creating your own custom MetaInstruction and vocabulary, and for the most part you can just reuse my own code except in your MetaInstruction you reference your custom JSON encoding/decoding pipeline components.  You can find the source code for the extension at “$/BREPipelineFramework/Main/BREPipelineFramework.BizTalk2013.R2/BREPipelineFramework.JSON” in the BRE Pipeline Framework source code repository.
Advertisements

I have just released v1.6.0 of the BRE Pipeline Framework to the CodePlex project page.  The framework is now mature enough that it’s core has not changed much this time around, but there are still tons of new features and optimizations to be found in this version. For a recap on what the framework is all about you can read the primer or the provided documentation.

Before I dive into the new features, here are some steps to get started. If you’re installing the framework for the first time or upgrading from a version above v1.4.0 then all you need to do is download the installer, uninstall the previous version if applicable, run the new installer to completion, and then import the BREPipelineFramework.LatestVocabs.xml vocabulary definition file from the vocabulary subfolder in the program files folder (default location is C:\Program Files (x86)\BRE Pipeline Framework\Vocabularies).  If you are upgrading from a version older than v1.4.0 then you might be forced to stop your BizTalk host instances since older versions of the pipeline component were not signed with a strong named key and installed in the GAC.

Onto the new features.

HTTP Header Manipulation
The BREPipelineFramework.SampleInstructions.ContextInstructions vocabulary already has vocabulary definitions that allow you to read in the WCF.InboundHttpHeaders context property to read in inbound headers, or to set the WCF.HttpHeaders context property to set outbound headers. However what you’ll find is that reading or setting HTTP headers from these context properties is not exactly easy as each context property actually contains a delimited list of all the inbound or outbound HTTP headers as per the following example. Getting to an individual HTTP header can be tricky if you try to use these vocabulary definitions.

32d6f253-de53-4cf3-b559-1ec34ac1ad28
Important_User: True
Content-Length: 328
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue
Host: servicebusrelaytest.servicebus.windows.net
Authorized_Token: 32d6f253-de53-4cf3-b559-1ec34ac1ad28" Namespace="http://schemas.microsoft.com/BizTalk/2006/01/Adapters/WCF-properties" Promoted="false" />

So now there is a new vocabulary called BREPipelineFramework.SampleInstructions.HttpHeaderInstructions which gives you the below vocabularies that allow you easier access to get or set inbound or outbound HTTP headers.
HTTPHeaderInstructions

The following screenshot demonstrates how these vocabulary definitions can be used. As you’ll see, you can read in inbound HTTP headers, explicitly set outbound HTTP headers, or copy inbound HTTP headers to outbound HTTP headers. You can set as many outbound HTTP headers as you want, and the framework will roll them all up into the single context property for you.
ExampleHttpHeader

Dynamic Pipeline Component Execution
This version of the framework now allows you to dynamically execute some of the out of the box Microsoft pipeline components. This feature was provided as a direct result of an issue posted on the framework’s page that asked for the ability to have context properties promoted on the target message when performing dynamic transformation using the framework. This would only really be possible by executing the XML Disassembler pipeline component, so the framework now has the ability to execute this component and some others.  You will be able to execute these pipeline components from within the new vocabulary BREPipelineFramework.SampleInstructions.PipelineInstructions.

PipelineInstructions

As you’ve seen in the previous screenshot, you can now execute the XML Assembler, XML DIsassembler, Flat File Assembler, Flat File Disassembler, and XML Validator pipeline components dynamically.  This means that you can assess conditions that allow you to decide whether/which pipeline components you should execute, and moreover you can dynamically configure parameters for these pipeline components as well.  The following screenshot demonstrates how you can execute the Flat File Disassembler while specifying the trailer schema parameter value.

FFDisassemble

One thing to note is that the BRE Pipeline Framework pipeline component is not a disassembler, so if you choose to execute a disassembler pipeline component and it ends up debatching an envelope message, the pipeline component will only pass through the first disassembled message.  In the case of the XML Disassembler, if you don’t want the message to be debatched you can make use of the DisassembleXMLMessagePropertyPromotionOnly vocabulary definition, which will result in context properties being promoted against the message without any debatching taking place.

Easier assertion of custom MetaInstructions

In the past, if you wanted to use your own custom MetaInstructions and corresponding vocabularies (see the project document page for more info on creating these) you would have to use an Instruction Loader Policy to assert the MetaInstruction fact so that it could be used in your Execution Policy (the BRE Policy in which you assess and manipulate your message’s context and content). In the following screenshot you’ll see how you a rule in an Instruction Loader Policy is used to assert a custom MetaInstruction fact into an Execution Policy by specifying the fully qualified class name and fully qualified assembly name of the MetaInstruction class.
InstructionLoaderPolicy
With v1.6.0 of the framework there is now a new pipeline component parameter called CustomMetaInstructionSeparatedList within which you can provide a semicolon separated list of fully qualified class/assembly names in the below format.

BREPipelineFramework.TestSampleInstructions.MetaInstruction, BREPipelineFramework.TestSampleInstructions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc;BREPipelineFramework.TestSampleInstructions.MetaInstructionCopy, BREPipelineFramework.TestSampleInstructions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc

This makes usage of custom MetaInstructions a whole lot easier to manage. You might still want to use an Instruction Loader Policy if any of the below scenarios applies to you.

  • You only want to conditionally assert the custom MetaInstruction in certain scenarios
  • You want to assert XML or SQL based facts
  • You want to conditionally override the ExecutionPolicy or the version

Assorted changes
There’s a couple more interesting new features across a variety of vocabularies.
In the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary, you’ll now find a vocabulary definition called GetCachedValueFromSSOConfigStore. The first time this vocabulary definition is executed it will read in a value from an SSO configuration store and it will also cache it. On further use of the vocabulary definition it will read the cached value rather than the SSO configuration store, keeping the number of hits to the SSO database to a minimum. By default the values will be cached for 30 minutes, but you can override this through the use of the UpdateCacheExpiryTime vocabulary definition in the same vocabulary.
CachedSSO
There is also a new vocabulary definition GetValueFromSSOConfigStore in the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary which will read in a value from an SSO configuration store without caching the result. This is probably useful in scenarios where you expect that you will be updating the configuration values regularly and always want to be reading in the latest value.
There are two new vocabulary definitions in the BREPipelineFramework.SampleInstructions.XMLTranslatorInstructions vocabulary that allow you to remove an element and all child elements from the XML message being processed, in a fully streaming manner. These are the RemoveElementAndChildrenByName and RemoveElementAndChildrenByNameAndNamespace vocabulary definitions. An example of their usage is below. No exception will be thrown in an element matching the criteria you set is not found in the message, so this vocabulary can be used liberally.
RemoveElementAndChildren
There are also two new vocabulary definitions ReturnFirstRegexMatchInString and ReturnRegexMatchInStringByIndex that allow you to run regular expressions against strings (these can be any strings, you could potentially chain together vocabulary definitions to run the regular expression against a context property for example).
V1.6.0 of the framework also has more tracing all over the place which will make it easier to understand what is going on within your BRE rules.

Optimizations

Finally, there have been some optimizations made to the framework. All reflected types for custom MetaInstructions and maps being executed dynamically will now be cached rather than having their types reflected every single time. This results in improved runtime performance of the framework.

I have just released a new version v1.5.5 of the BRE Pipeline Framework to the CodePlex project page.  This is a relatively minor update in that it only adds some missing WCF context properties to the enumerations used to get and set WCF context properties.

What makes this minor update important is that most of these context properties are REST related, and will enable you to build flexible REST based integration solutions where you can dynamically set HTTP methods, URLs, or headers.  The list of context properties added to the enumeration is below.

  • HttpHeaders
  • HttpMethodAndUrl
  • InboundHttpHeaders
  • InboundHttpMethod
  • InboundHttpStatusCode
  • InboundHttpStatusDescription
  • IssuerName
  • IssuerSecret
  • OutboundHttpStatusCode
  • OutboundHttpStatusDescription
  • StsUri
  • SuppressMessageBodyForHttpVerbs
  • VariablePropertyMapping

This new version is fully backwards compatible, and installing it will only require uninstalling the previous version from the add/remove programs control panel and then running the new installer.  Please post any issues/feedback/comments to the CodePlex project discussions page.

I have now uploaded the BRE Pipeline Framework v1.5.4 installer to the CodePlex project page.  This version is as always fully backwards compatible with BRE Policies created with previous versions, and installing it over an existing version simply requires uninstalling the currently installed version, installing the new version, and importing the new vocabularies from the program files folder (default location is C:\Program Files (x86)\BRE Pipeline Framework\Vocabularies).

In a previous post I  discussed a pattern that allows for the temporary persistence of context properties on a request message in a request/response send port, and then reapplying those context properties on the response message before it gets sent to the BizTalk message box.  The beauty of this pattern is that it allows you to enrich response messages with state from the request message without the need for orchestration.  The temporary persistence store that I used for these context properties is based on the .Net MemoryCache class.

One of the features of the MemoryCache class is that it is really easy to set expiry times on cached items so you aren’t forced to explicitly remove items from the cache (though you still have the ability to), and setting expiry times and explicitly deleting cached items are features provided for in the BRE Pipeline Framework.

The MemoryCache class also has a feature whereby if the server is under heavy pressure for memory it will automatically drop items from the cache.  This is great when the items in the cache are truly temporary and you have a means of reloading the item in the cache, however doesn’t work very well for the pattern described in my previous post which depends on the cached items being available.  To get around this I have taken advantage of the Priority property in the CacheItemPolicy class, which allows you to override this default behavior, instructing .Net not to automatically remove your cached items.

This feature is introduced in the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary v1.1’s vocabulary definition called SetCachingPriority.  You have two priority values that you can apply as below.

  • Default – this implies that .Net should remove cached items automatically if under memory pressure.  If you don’t use the SetCachingPriority vocabulary definition then this value will be automatically chosen.  Cached items can also be removed based on expiry time or based on explicit removal.
  • NotRemovable – this instructs .Net not to ever remove cached items, even if under memory pressure.  Cached items can only be removed based on expiry time or based on explicit removal.

An example usage of the SetCachingPriority vocabulary definition is in the screenshot below.

VocabDef

One of the most compelling features that was made available in v1.5 of the BizTalk BRE Pipeline Framework was the ability to cache context properties or any custom strings and then fetch them back later on.  I purposely chose not to blog about this feature till now because I wanted to employ it in a real life scenario to prove it’s value first which I have now.

The caching features are contained within the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary that was shipped with v1.5 of the framework.  If you haven’t already published the vocabulary you can find the exported XML for this vocabulary in the Vocabularies subfolder in the BRE Pipeline Framework program files folder (typically “C:\Program Files (x86)\BRE Pipeline Framework”) and can use the Rules Engine Deployment Wizard to publish it.  After reading this blog post you will have an idea of what caching features are available in the BRE Pipeline Framework, how the framework achieves the caching, and you’ll also be exposed to a summary description of each vocabulary definition in the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

 

Caching Context Properties

The most common use for the caching feature in the BRE Pipeline Framework will be to cache context properties and then to reapply them.  Let me paint a picture to you, which will seem all too common to those experienced with BizTalk and who strive to create messaging only applications in situations where no orchestration is warranted.  You need to call a WCF service which returns either a valid response which doesn’t contain any of the identifiers from the request, or a SOAP fault (unless SOAP faults were generated in a custom fashion you typically only see an error message with no details that help you to identify which request the fault corresponds to), and you need to write the status of the response to a database along with the identifiers from the request for correlation purposes.

BizTalk Server messaging is stateless by nature so you only have access to the message currently being processed and not historical messages when you are performing your inbound mapping on the request/response send port.  Thus it wouldn’t typically be possible to satisfy the requirement of writing the response to the database with the identifiers from the request unless you used an orchestration that persisted the state of the original request and then executed a multi-source map that used the original request and the response message to transform to the destination message.  I consider orchestration to be overkill for such a scenario and prefer to go down a messaging only route.

The answer to the above scenario is to ensure that all the identifiers in the request message are promoted to context properties on the message that is directed to the WCF service send port, use the BRE Pipeline Framework to cache the context properties from the request, and to then use the BRE Pipeline Framework to reapply those context properties on the response message.  You could then use the Context Accessor pipeline component/functoids to insert the context properties into the target message body via a map, or use property demotion (see this TechNet article for more details) to achieve the same result.

Below is an example rule where the BRE Pipeline Framework is used to cache the RequestID and Requestor context properties on the request message on a send port.  In the below case the BRE Pipeline Framework will throw an exception if the RequestID context property doesn’t exist on the request message, but will just ignore and carry on if the Requestor context property doesn’t exist on the request message.  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the AddCustomContextPropertyToCache definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

CacheRule

Reapplying these context properties is easy as per the below screenshot in which the RequestID context property will be promoted (potentially for routing) while the Requestor context property is merely written to the context of the message (thus not available for routing).  In the below case the BRE Pipeline Framework will throw an exception if the RequestID context property doesn’t exist in the cache, but will just ignore and carry on if the Requestor context property doesn’t exist in the cache.  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the ReapplyCachedContextProperty definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

ReapplyRule

How does this work?  Under the hood the BRE Pipeline Framework makes use of the .NET MemoryCache class (specifically used because of it’s cached item auto-expiry features) which ships with .NET 4.0 onwards.  Each context property that you choose to cache on a given message gets added to a Dictionary in which the context property namespace#name is used as the key and the context property value is the value.  The Dictionary is then added to the Memory Cache with the key based on the BTS.TransmitWorkID context property of the request message.  The reason the BTS.TransmitWorkID context property is used as the key is because this context property is automatically generated on request-response send ports and response messages will always have the same value for this context property as the request message, thus this value can be used to write to and read from the MemoryCache.  Each item in the MemoryCache by default has an expiry time of 30 minutes (if not explicitly removed before), after which they will automatically be removed from the cache to ensure that RAM isn’t permanently earmarked for the caching.   The MemoryCache is isolated to a given AppDomain thus is unique per host instance (thus if you cache a value in one host instance you can’t reapply it in another, and is also not shared across machines even if host instances pertain to the same host) and if a single host instance is used for both receive/send ports and orchestration then the receive/send ports will have one MemoryCache while the orchestration will have another (see this fantastic article by Saravana Kumar if you are interested in reading more about BizTalk and AppDomains).

 

Caching custom strings

The caching functionality in the BRE Pipeline Framework can also be used to cache any string and to fetch it later.  A common use for this would be to cache a value that has been read in from a relatively expensive data source to read from such as SQL Server or the SSO.  Doing so means that you don’t have to hit the database each time your rule is executed.

Below is an example in which the value ReadInFromSQL (the hardcoded value could be replaced with a vocabulary definition that fetches the value from the relevant data source) is cached using the key Configuration Info, the cached entry being set to expire in 5 hours (note the time units is an Enum so you can choose whichever applicable unit you want).  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the AddCustomStringToCache definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

CacheCustomStringRule

Fetching the cached value is easy as per below.  In the below case the BRE Pipeline Framework will return Null if the key isn’t found in the cache (other options are to throw an exception or return a blank string).  The vocabulary definition used in the Condition in the below rule is the ApplicationContext definition from the BREPipelineFramework vocabulary, while the definitions used in the Action is the GetCustomStringFromCache definition from the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary.

FetchCustomStringRule

The primary difference between the way that caching custom strings works vs. caching context properties works is that with the custom strings each value is cached separately in the MemoryCache rather than within a single dictionary per message which contains all the context properties for that message, and it is up to you to explicitly state the expiry time and the key for each value you choose to cache.

 

Walkthrough of vocabulary definitions in the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary

  • AddCustomContextPropertyToCache – Allows you to specify a context property against the current message that should be cached.
  • AddCustomStringToCache – Allows you to specify any string that should be cached.  You will also need to supply a key and an expiry time for this cached entry.
  • CacheAllContextProperties – Allows you to cache all the context properties against the current message.  Can help  you to future proof your rule since adding more context properties to the message will not require a change to the rule.
  • ChangeKeyContextProperty (advanced feature)- By default when you cache context properties they are all collapsed into a single Dictionary which is added to the MemoryCache with a key based on the BTS.TransmitWorkID context property which is great for synchronous request/response type scenarios.  If you want to cater for asynchronous scenarios whereby you might be trying to reapply the context property on a message processed on a totally separate port then you have the ability to nominate another context property as the key in the MemoryCache.  Note that to reapply the context properties on the target message it must already have the same context property value for the key context property as the original message.  Also note that this vocabulary definition would need to be contained within the action for the rule that adds to the cache and the rule that gets from the cache.
  • DeleteContextFromCacheIfItStillExists – Allows you to explicitly delete context properties from the cache for the current message.  You might want to do this after you have reapplied the context properties from the cache rather than wait for the expiry time to pass to reduce strain on RAM.
  • DeleteCustomStringFromCache – Allows you to explicitly delete a custom string from the cache based on the specified key.  You might want to do this when you are sure you no longer need the cached string rather than wait for the expiry time to pass to reduce strain on RAM.
  • GetCustomContextPropertyFromCache – Allows you to fetch a cached context property.  Note that this definition doesn’t reapply the cached context property to the current message but rather is just used to fetch the value, potentially for inspection in conditions or to use as a parameter in another definition within a rule’s action.
  • GetCustomStringFromCache – Allows you to fetch a cached string which can be used as a string parameter in other definitions in a rule’s condition or action.
  • ReapplyAllCachedContextProperties – Allows you to reapply all the cached context properties against the current message.  Be wary of using this in tandem with the CacheAllContextProperties definition, as this might result in an attempt to override out of the box context properties such as BTS.MessageType or BTS.MessageID etc…
  • ReapplyCachedContextProperty – Allows you to reapply a specific cached context property against the current message.
  • UpdateCacheExpiryTime – Allows you to override the default expiry time for cached context properties, the default being 30 minutes.
%d bloggers like this: