SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition

The book is now available, well done co-authors, reviewers and Packt Publishing team.

Connected Pawns

I am pleased to announce “SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition” by Mark Brimble, Johann Cooper, Coen Dijgraaf and Mahindra Morar can now be purchased on the Amazon or from Packt. This is based on the first edition written by by Richard Seroter. Johann has given a nice preview of the book here.

This is the first book I have co-authored and I can now tick that off on my bucket list. It was an experience to work with such talented co-authors and I was continually amazed how they rose to the occasion. I agree with Johann’s statement about being privileged to write the second edition of what was the first BizTalk book I ever owned.

As I updated chapters I was continually struck by how little had changed. We added new chapters, Chapter 4, Chapter 5, and Chapter 6…

View original post 125 more words

Sharing context between BizTalk and WCF behavior extensions

For those who extend their BizTalk solutions using WCF behaviors, a common challenge is working out how to flow details from your BizTalk message into your WCF behavior extension so that you can access them in the extension.

Luckily for you, BizTalk and WCF are like a match made in heaven and work together really seamlessly.  BizTalk context properties flow through to WCF behavior extensions and are easily accessible, and it is quite possible to flow details back from a WCF behavior to the response message’s BizTalk context as well.

For the rest of this blog post I will assume that you have already setup a WCF Message Inspector extension and are making use of it on a WCF send port in BizTalk (a lot of what I demonstrate should be applicable for other types of behavior extensions and for receive locations as well).  I will demonstrate how you can access BizTalk context from within your Message Inspector, and also how you can set context property values on the response message as well.

For your request messages this is a no brainer.  Every single context property that was attached to the message in BizTalk will automatically flow through to your WCF Message Inspector in the form of a property bag attached to the request message. Context property values can be fetched out of the property bag by using the context property namespace#name as in the following example which fetches the BTS.Operation context property value (you would normally of course want to check whether you are getting a null value back before converting it to a string to avoid a NullReferenceException in case the context property didn’t exist in the property bag).

        public object BeforeSendRequest(ref System.ServiceModel.Channels.Message request, System.ServiceModel.IClientChannel channel)
        {
            string _BTSOperation = request.Properties["http://schemas.microsoft.com/BizTalk/2003/system-properties#Operation"].ToString();
            ///Do some stuff with the BTS Operation
        }

Leveraging your context properties in WCF behaviors opens the door towards much more cohesive solutions in which you use the right types of components for the right kinds of problems. A good example of this would be scenarios in which you need to calculate an OAUTH signature, which is probably best done in a WCF behavior, however you might have otherwise been tempted to do it in a pipeline component instead since you needed access to BizTalk context properties.  Knowing that you can flow your context properties through to your behavior extensions keeps your options open.

How about the flip side when you want to flow some data back from your WCF behavior to BizTalk context properties. Here your options diverge slightly depending on whether you are dealing with a SOAP or RESTFul service based on the WCF-WebHttpBinding binding, and you’ll also find that all options include a bit more difficulty.  You can choose to pass the context in the form of inbound SOAP headers or inbound HTTP headers as demonstrated in the following code snippet.

        public void AfterReceiveReply(ref System.ServiceModel.Channels.Message reply, object correlationState)
        {
            //Writing to WCF.InboundHttpHeaders context property
            System.ServiceModel.Channels.HttpResponseMessageProperty httpHeaders = (System.ServiceModel.Channels.HttpResponseMessageProperty)reply.Properties["httpResponse"];
            httpHeaders.Headers.Add("ContextHTTPHeader", "Content property value");

            //Writing to WCF.InboundHeaders as well as to http://contextproperty#ContextSOAPHeader context property
            var header = new MessageHeader<string>("Content property value");
            var untypedMessageHeader = header.GetUntypedHeader("ContextSOAPHeader", "http://contextproperty");
            reply.Headers.Add(untypedMessageHeader);
        }

So if I actually use this behavior on a WCF SOAP based send port what will the result look like?

We can see that populating an inbound HTTP header in the WCF behavior has resulted in an HTTP header called ContextHTTPHeader being appended to the end of the WCF.InboundHttpHeaders context property. You can use the BRE Pipeline Framework to extract individual HTTP headers easily. The value of the context property looks like the below.

X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Connection: Keep-Alive
Content-Length: 535
Cache-Control: private, max-age=0
Content-Type: application/soap+xml; charset=utf-8
Date: Wed, 01 Jul 2015 08:59:10 GMT
Server: WWW Server/1.1
ContextHTTPHeader: Content property value

Populating the inbound SOAP header in the WCF behavior actually results in it showing up in two context properties. The first is in the WCF.InboundHeaders context property, and the value looks like the below.

<headers><ContextSOAPHeader xmlns=”http://contextproperty”>Content property value</ContextSOAPHeader></headers>

The second context property which you’ll notice is a custom context property with the same name/namespace as the SOAP header which we created – http://contextproperty#ContextSOAPHeader.

<ContextSOAPHeader xmlns=”http://contextproperty”>Content property value</ContextSOAPHeader>

In both of the above scenarios you’ll notice that it isn’t too simple to actually extract the value that you set for your inbound SOAP header. Once again, the BRE Pipeline Framework could help you extract the value using regex.

Note that if you try to set an inbound SOAP header in a WCF behavior on a WCF-WebHttp send port you will be met with the following exception.

System.InvalidOperationException: Envelope Version ‘EnvelopeNone (http://schemas.microsoft.com/ws/2005/05/envelope/none)’ does not support adding Message Headers.

Dynamic request content and response content negotiation for RESTFul APIs exposed by BizTalk Server

One of the core tenets of RESTFul architecture is that APIs should support content negotiation.  That is to say that the service consumer should be able to tell the service through the use of the Accept HTTP header what format the response message should be returned in.  On the flip side, it is commonly accepted that RESTFul APIs should support a range of request formats; typically XML and JSON at the very least, and that the Content-Type HTTP header is used to convey the format.

BizTalk Server 2013 R2 delivered the new JSON Decoder and Encoder pipeline components that at last treat JSON as a first class format in BizTalk Server.  Internally, BizTalk maps will still only work with XML as will a range of other BizTalk functions, however at the very least JSON is now accepted on the outskirts.  It is possible to get JSON support in older versions of BizTalk however you will have to create custom pipeline components to cater for this, since Microsoft has only added out of the box functionality in BizTalk Server 2013 R2.

Unfortunately, the traditional means of employing these pipeline components comes with some rigidity.  If you want to expose a RESTFul API using a WCF-WebHttp receive location then you will need to choose whether or not to include a JSON Decoder pipeline component in your receive pipeline.  On the flip side, you will need to choose whether to use a JSON Encoder on your send pipeline.  There is no way to dynamically decide whether to employ the JSON Decoder and Encoder based on the Content-Type and Accept header on the request message without employing an orchestration.  And it gets even worse…what if you wanted to build an API that also supported an HTML response type?

I’m not against the use of orchestration when they are used to provide some form of workflow or process orchestration, however I do not like being forced to use orchestration due to technical constraints.

I have addressed this problem for the most part using the BRE Pipeline Framework.  Because JSON decoding and encoding is only supported by BizTalk Server 2013 R2, I have addressed this problem in a custom MetaInstruction, or an extension to the framework, rather than making a change to the core framework.  Note that this extension requires that you have the BRE Pipeline Framework v1.6.0 or above installed on your machine.  This new functionality is catered for via the new BREPipelineFramework.JSONInstructions vocabulary.

To get started download the installer for the extension and install it on your BizTalk 2013 R2 machine.  Install the vocabulary using the Business Rules Engine Deployment Wizard, the vocabulary file being found in the program files folder which by default is “C:\Program Files (x86)\BREPipelineFramework.JSONInstructions”.

Here’s a quick rundown on what these vocabulary definitions do.

• DecodeJSON – This vocabulary definition executes the JSON Decoder against the message.  You will need to specify a root node name and namespace with which the decoded XML will be wrapped in.
• EncodeJSON – This vocabulary definition executes the JSON Encoder against the message.  You will need to specify whether the root node will be stripped off prior to encoding or not.
• ExecuteXSLTTransform – This vocabulary definition will execute an XSLT transformation based on an XSLT file which you must provide a file location for.  This is handy if you want to transform to/from HTML.
• AssessContentType – This vocabulary definition lets you assess what the content type of your request message is so you can decide how to process it.  It will first attempt to derive the content type by studying the Content-Type HTTP header on the request.  If no Content-Type header is available then it will probe the first character of the message to determine if it is XML or JSON.
• CacheAcceptHeader – Since we need to assess the Accept header value when choosing how to encode the response message, we must cache the Accept header value while processing the request.  If we did not do this then we would not have access to the value later.
• AssessCachedAcceptHeader – This vocabulary definition lets you assess what the desired content type is based on the cached Accept header.  This can be executed on the send pipeline that is used to encode the response message.
• GetCachedAcceptHeader – This vocabulary definition will return the raw value of the cached accept header.  This can be executed on the send pipeline that is used to encode the response message.  Only use this if you aren’t happy with the logic used by the AssessCachedAcceptHeader vocabulary definition.

The following screenshot demonstrates how we can dynamically decode incoming request messages based on the derived content type of the request message.  Note that if the message was XML instead of JSON then this rule would not first.  Note as well that we are caching the Accept header value for later use…theoretically this should be done in a separate rule which will execute regardless of the request content type but I have collapsed the two actions into one rule for brevity.

The following screenshot demonstrates the assessment of the cached Accept header value to dynamically encode the response message to JSON.

And the next screenshot similarly dynamically transforms the response to HTML by executing custom XSLT.

So how do you make use of this new vocabulary?  The easiest way to do this is to put the fully qualified class/assembly name of the custom MetaInstruction in the CustomMetaInstructionSeparatedList parameter (which takes in a ; delimited list) of the BREPipelineFrameworkComponent pipeline component as below.  Once you’ve done this then the vocabulary will be available for use in the ExecutionPolicy.

The fully qualified class/assembly name value for the JSON extension is as below.

BREPipelineFramework.JSON.JSONMetaInstruction, BREPipelineFramework.JSON, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc

So what are the shortcomings of the current solution?

• The response content type derivation based on the cached Accept header doesn’t currently go as far as is normally expected by RESTFul standards.  These standards dictate that multiple content types can be specified and each can be given priority weightings.  The solution I’ve put in place is just a v1, and it will always prioritize as follows regardless of explicit priorities – JSON > XML > URLEncodedForm > CSV > HTML > Text > Other.  If you don’t like this logic then you can always make your own choice by assessing the raw Accept header value with the GetCachedAcceptHeader vocabulary definition instead of the AssessCachedAcceptHeader vocabulary definition.
• I have not yet found a context driven way of overriding the Content-Type HTTP header on the response message.  This will always be set to Application/XML, unless you have set an alternate value on the Messages tab of your receive location’s configuration in which case it will always be that alternative value.  For now the only way I can think of overriding this is via a WCF behavior.
• It only works with BizTalk Server 2013 R2, but you could easily get it to work with BizTalk 2013 by creating your own custom MetaInstruction and vocabulary, and for the most part you can just reuse my own code except in your MetaInstruction you reference your custom JSON encoding/decoding pipeline components.  You can find the source code for the extension at “$/BREPipelineFramework/Main/BREPipelineFramework.BizTalk2013.R2/BREPipelineFramework.JSON” in the BRE Pipeline Framework source code repository.

“SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition” available for pre-order from Packt Publishing

Disclaimer – Shameless (but hopefully informational) plug follows 🙂

Close to a year ago myself and my colleagues Mark, Mahindra, and Colin were approached by Packt Publishing to update Richard Seroter’s “SOA Patterns with BizTalk Server 2009” book for BizTalk Server 2013.  We considered this a great honour seeing as the base material in the first edition was so good, and decided to take on the project.

Nine months of working through the authoring process later, I am pleased to announce that “SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition” is due to be published imminently and is available for pre-order.

So what will you find in this edition of the book?

For me personally, the first edition of this book made me realize that there if more to BizTalk than tightly bound orchestrations, and it opened my eyes to how the platform could be used to maximum effectiveness.  It is my personal favourite BizTalk book, and it is one of only two BizTalk books that I personally own and have read cover to cover many times (the other is Professional BizTalk Server 2006).  I’ve of course read loads more BizTalk books, but these are ones I want in my personal library at home.

Given this high degree of respect for the first edition, you’ll find that much of it has been preserved in its original form.  What you will find is that some of the original material has been tweaked to make it more relevant, and that it has been interspersed with many more warnings, tips, and tricks.

Beyond the original source material, we now have major portions of the book dedicated to Azure platforms such as Service Bus, and MABS (Microsoft Azure BizTalk Server).  We also explore the usage of Azure App Services’ API Apps and Logic Apps, as well as Azure API Management however this is more of a peek under the covers to whet your whistle rather than an extreme deep dive.

We also dive deep into the use of REST and how to implement RESTFul APIs using BizTalk Server.  Expect to get into the theory behind REST as well as read about some practical examples.  Given how relevant RESTFul APIs are in the integration space these days this is something which all BizTalkers should be exposed to.

We have an entire chapter that is dedicated to introducing you to tools that we believe that you should be aware of, and should be carefully assessing for usage in your own environment.  These include the ESB Toolkit, the BRE Pipeline Framework, Sentinet, BizTalk 360, AIMS for BizTalk, and the BizTalk Documenter.  While there are way more frameworks we’d have loved to discuss, there was simply limited space so we had to prioritize.  We have provided quick summaries of those that we feel need awareness but weren’t in our top pick (due to relevance to the book’s title rather than due to how functional and generally useful they are).

As BizTalk Server 2013 R2 was released after we started working on the book and the number of new features that are applicable to this book are relatively limited, we decided to truck on and focus on BizTalk Server 2013.  However, we would be negligent not to mention those features in BizTalk Server 2013 R2 that are relevant to this book, so you will find a section on BizTalk Server 2013 R2 in the book as well.

Finally, you will find that we have updated the existing accompanying source code for BizTalk Server 2013 and expanded the existing samples with further examples (for example we demonstrate in brief how the async features in .NET 4.5 can be used in WCF clients, and we have also provided samples for untyped orchestrations).  And of course we have added brand new source code to accompany new chapters as well.

It has been a great honour writing this book, and it will be an even greater honour if you decide to read it as well.  I sincerely hope that we have done Richard’s original book justice, and that this new edition helps to add to your knowledge, expand your horizons, and further your careers.

BRE Pipeline Framework v1.6.0 now published on CodePlex – New feature breakdown

I have just released v1.6.0 of the BRE Pipeline Framework to the CodePlex project page.  The framework is now mature enough that it’s core has not changed much this time around, but there are still tons of new features and optimizations to be found in this version. For a recap on what the framework is all about you can read the primer or the provided documentation.

Before I dive into the new features, here are some steps to get started. If you’re installing the framework for the first time or upgrading from a version above v1.4.0 then all you need to do is download the installer, uninstall the previous version if applicable, run the new installer to completion, and then import the BREPipelineFramework.LatestVocabs.xml vocabulary definition file from the vocabulary subfolder in the program files folder (default location is C:\Program Files (x86)\BRE Pipeline Framework\Vocabularies).  If you are upgrading from a version older than v1.4.0 then you might be forced to stop your BizTalk host instances since older versions of the pipeline component were not signed with a strong named key and installed in the GAC.

Onto the new features.

HTTP Header Manipulation
The BREPipelineFramework.SampleInstructions.ContextInstructions vocabulary already has vocabulary definitions that allow you to read in the WCF.InboundHttpHeaders context property to read in inbound headers, or to set the WCF.HttpHeaders context property to set outbound headers. However what you’ll find is that reading or setting HTTP headers from these context properties is not exactly easy as each context property actually contains a delimited list of all the inbound or outbound HTTP headers as per the following example. Getting to an individual HTTP header can be tricky if you try to use these vocabulary definitions.

32d6f253-de53-4cf3-b559-1ec34ac1ad28
Important_User: True
Content-Length: 328
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue
Host: servicebusrelaytest.servicebus.windows.net
Authorized_Token: 32d6f253-de53-4cf3-b559-1ec34ac1ad28" Namespace="http://schemas.microsoft.com/BizTalk/2006/01/Adapters/WCF-properties" Promoted="false" />

So now there is a new vocabulary called BREPipelineFramework.SampleInstructions.HttpHeaderInstructions which gives you the below vocabularies that allow you easier access to get or set inbound or outbound HTTP headers.

The following screenshot demonstrates how these vocabulary definitions can be used. As you’ll see, you can read in inbound HTTP headers, explicitly set outbound HTTP headers, or copy inbound HTTP headers to outbound HTTP headers. You can set as many outbound HTTP headers as you want, and the framework will roll them all up into the single context property for you.

Dynamic Pipeline Component Execution
This version of the framework now allows you to dynamically execute some of the out of the box Microsoft pipeline components. This feature was provided as a direct result of an issue posted on the framework’s page that asked for the ability to have context properties promoted on the target message when performing dynamic transformation using the framework. This would only really be possible by executing the XML Disassembler pipeline component, so the framework now has the ability to execute this component and some others.  You will be able to execute these pipeline components from within the new vocabulary BREPipelineFramework.SampleInstructions.PipelineInstructions.

As you’ve seen in the previous screenshot, you can now execute the XML Assembler, XML DIsassembler, Flat File Assembler, Flat File Disassembler, and XML Validator pipeline components dynamically.  This means that you can assess conditions that allow you to decide whether/which pipeline components you should execute, and moreover you can dynamically configure parameters for these pipeline components as well.  The following screenshot demonstrates how you can execute the Flat File Disassembler while specifying the trailer schema parameter value.

One thing to note is that the BRE Pipeline Framework pipeline component is not a disassembler, so if you choose to execute a disassembler pipeline component and it ends up debatching an envelope message, the pipeline component will only pass through the first disassembled message.  In the case of the XML Disassembler, if you don’t want the message to be debatched you can make use of the DisassembleXMLMessagePropertyPromotionOnly vocabulary definition, which will result in context properties being promoted against the message without any debatching taking place.

Easier assertion of custom MetaInstructions

In the past, if you wanted to use your own custom MetaInstructions and corresponding vocabularies (see the project document page for more info on creating these) you would have to use an Instruction Loader Policy to assert the MetaInstruction fact so that it could be used in your Execution Policy (the BRE Policy in which you assess and manipulate your message’s context and content). In the following screenshot you’ll see how you a rule in an Instruction Loader Policy is used to assert a custom MetaInstruction fact into an Execution Policy by specifying the fully qualified class name and fully qualified assembly name of the MetaInstruction class.

With v1.6.0 of the framework there is now a new pipeline component parameter called CustomMetaInstructionSeparatedList within which you can provide a semicolon separated list of fully qualified class/assembly names in the below format.

BREPipelineFramework.TestSampleInstructions.MetaInstruction, BREPipelineFramework.TestSampleInstructions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc;BREPipelineFramework.TestSampleInstructions.MetaInstructionCopy, BREPipelineFramework.TestSampleInstructions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc

This makes usage of custom MetaInstructions a whole lot easier to manage. You might still want to use an Instruction Loader Policy if any of the below scenarios applies to you.

• You only want to conditionally assert the custom MetaInstruction in certain scenarios
• You want to assert XML or SQL based facts
• You want to conditionally override the ExecutionPolicy or the version

Assorted changes
There’s a couple more interesting new features across a variety of vocabularies.
In the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary, you’ll now find a vocabulary definition called GetCachedValueFromSSOConfigStore. The first time this vocabulary definition is executed it will read in a value from an SSO configuration store and it will also cache it. On further use of the vocabulary definition it will read the cached value rather than the SSO configuration store, keeping the number of hits to the SSO database to a minimum. By default the values will be cached for 30 minutes, but you can override this through the use of the UpdateCacheExpiryTime vocabulary definition in the same vocabulary.

There is also a new vocabulary definition GetValueFromSSOConfigStore in the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary which will read in a value from an SSO configuration store without caching the result. This is probably useful in scenarios where you expect that you will be updating the configuration values regularly and always want to be reading in the latest value.
There are two new vocabulary definitions in the BREPipelineFramework.SampleInstructions.XMLTranslatorInstructions vocabulary that allow you to remove an element and all child elements from the XML message being processed, in a fully streaming manner. These are the RemoveElementAndChildrenByName and RemoveElementAndChildrenByNameAndNamespace vocabulary definitions. An example of their usage is below. No exception will be thrown in an element matching the criteria you set is not found in the message, so this vocabulary can be used liberally.

There are also two new vocabulary definitions ReturnFirstRegexMatchInString and ReturnRegexMatchInStringByIndex that allow you to run regular expressions against strings (these can be any strings, you could potentially chain together vocabulary definitions to run the regular expression against a context property for example).
V1.6.0 of the framework also has more tracing all over the place which will make it easier to understand what is going on within your BRE rules.

Optimizations

Finally, there have been some optimizations made to the framework. All reflected types for custom MetaInstructions and maps being executed dynamically will now be cached rather than having their types reflected every single time. This results in improved runtime performance of the framework.