SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition

The book is now available, well done co-authors, reviewers and Packt Publishing team.

Connected Pawns

I am pleased to announce “SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition” by Mark Brimble, Johann Cooper, Coen Dijgraaf and Mahindra Morar can now be purchased on the Amazon or from Packt. This is based on the first edition written by by Richard Seroter. Johann has given a nice preview of the book here.

This is the first book I have co-authored and I can now tick that off on my bucket list. It was an experience to work with such talented co-authors and I was continually amazed how they rose to the occasion. I agree with Johann’s statement about being privileged to write the second edition of what was the first BizTalk book I ever owned.

As I updated chapters I was continually struck by how little had changed. We added new chapters, Chapter 4, Chapter 5, and Chapter 6…

View original post 125 more words

Keeping Service Bus Relays alive for BizTalk based WCF services – Application Initialization and keepalives

One of the key areas to focus on when using Service Bus Relays to expose on premise BizTalk hosted WCF services externally without making any firewall changes is availability. Service Bus Relay endpoints in Azure will only be enlisted upon initialization of your on premise service. NOT when your application pool starts, NOT when you create your Service Bus Namespace, but only when a request is made to the local endpoint of the service. Browsing (effectively an HTTP GET) to the local .svc file on the BizTalk VM serves just as well to enlist the Azure Service Bus Relay endpoint.

A common solution to this problem is to use the IIS 8.0 Application Initialization module which has been really well documented here. What this effectively results in is that your .svc file is activated any time IIS resets, your application recycles, or your app domain reloads. The Application Initialization module is also available for IIS 7.5 and can be downloaded here.  This results in your relay endpoint being enlisted in Azure.

Application Initialization is an absolutely core part to any Service Bus Relay service, regardless whether the backend service is a vanilla WCF service or based on a BizTalk receive location. No WCF Service Bus Relay solution should be built without this in mind.

However, I found that when dealing with BizTalk there is an additional consideration to keep in mind. What I observed is that if your BizTalk environment encounters outages due to an inability to connect to the BizTalk Management database (due to network or database issues) the services (not the application pool, not IIS, the services themselves!) will get shut down by BizTalk. When BizTalk recovers from the outage the services will not get spun up again until someone calls on the local service endpoint or browses to the local .svc file. Because the IIS application pool has not restarted, Application Initialization will not kick in and thus your endpoint will not be enlisted in Azure.

The solution I have put in place is to generate keepalive requests to the local endpoint every minute. This makes me feel a bit dirty, but I haven’t found a better solution yet, so I will detail it for you. If you can think of a better option, please do share it and I will update this post.

What I have done is setup a receive location based on the Scheduled Task adapter that generates a keepalive message every one minute (I’ve chosen quite a regular interval because availability is really key to my services, choose your interval appropriately) that gets routed to a solicit-response WCF-WebHttp send port (actually in my case 4 send ports, one corresponding to each service being exposed by Service Bus Relays that I want to keep alive). The endpoint address on this send port is pointed towards the .svc file for the service I want to keep alive, and the “HTTP Method and URL Mapping” section of the send port’s configuration is set to “GET” since we want to perform an equivalent action to browsing to the .svc file.

One more thing of note is that since this is an HTTP GET, the target endpoint is not expecting a request message body, so we must use the WCF-WebHttp adapter’s functionality to supress it, as per the following screenshot.

Because I’m not really interested in the response messages being returned to the send port, I route these messages to a file send port which uses the BRE Pipeline Framework to discard the message (utilizing the NullifyMessage vocabulary definition in the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary).  The address on the send port can be set to any folder since no file will actually get written out by the adapter.

Using the WCF-WebHttp adapter and pointing it directly to the .svc file helps minimize the requirement for additional development since you aren’t forced down the path of exposing additional operations in your service to cater for the keepalives.

Service Bus Relay based services slow to startup and slow runtime performance

While working on a project using the WCF webHttpRelayBinding binding with SAS based authentication over transport security, I found that my services were taking a very long time to spin up (30-60 seconds) and that my runtime performance was a bit less than optimal in terms of latency (and I had proven that the latency was not a result of my backend service).  To give you an idea what I was working with, my web.config file had contents similar to the below in the system.serviceModel element.

      <endpointBehaviors>
        <behavior name="sharedSecretClientCredentials">
          <transportClientEndpointBehavior>
            <tokenProvider type="SharedAccessSignature">
              <sharedAccessSignature keyName="keyname" key="key" />
            </tokenProvider>
          </transportClientEndpointBehavior>
        </behavior>
      </endpointBehaviors>
    </behaviors>
    <services>
      <service name="Microsoft.BizTalk.Adapter.Wcf.Runtime.BizTalkServiceInstance" behaviorConfiguration="ServiceBehaviorConfiguration">
        <endpoint name="RelayEndpoint" address="https://sbnamespace.servicebus.windows.net/service" binding="webHttpRelayBinding" bindingNamespace="http://tempuri.org/" bindingConfiguration="RelayEndpointConfig" behaviorConfiguration="sharedSecretClientCredentials" contract="Microsoft.BizTalk.Adapter.Wcf.Runtime.ITwoWayAsync" />
      </service>
    </services>
    <bindings>
      <webHttpRelayBinding>
        <binding name="RelayEndpointConfig">
          <security relayClientAuthenticationType="RelayAccessToken" mode="Transport" />
        </binding>
      </webHttpRelayBinding>     
    </bindings>

I didn’t observe such problems on my development VM (which I was running with pretty much no firewalls behind it), but did observe this on my client’s UAT environment. This was in spite of following Microsoft’s guidelines that suggest that you should have outbound TCP ports 9350-9354 open on your firewall to enable Service Bus connectivity.  I went through an exercise using the PortQuiz website to prove that these ports were indeed accessible from the UAT server so the performance issues were puzzling.

Next up, I spun up a fiddler capture.  To start with I applied the below filter into Fiddler to get rid of the extra noise.

tcp.port eq 9350 || tcp.port eq 9351 || tcp.port eq 9352 || tcp.port eq 9353 || tcp.port eq 9354

I then initialized some of my services (I shut them down forcibly and then spun them up again) to observe which ports were in use.  I saw that the conversation with Service Bus was being initialized on port 9350 as expected, however that appeared to be the end of the story.  I wasn’t seeing any comms on ports 9351-9354.  I then right clicked one of the displayed records in WireShark and chose “Conversation Filter -> IP” which updates the filter such that it displays anything with a source or destination IP address matching those on the selected record.

This suddenly resulted in a whole lot more records being displayed and helped me get to the root of the issue.  What I was observing was that after Service Bus made the initial connection on port 9350, it attempted to continue the conversation on port 5671 (AMQPS or AMQP over SSL) which hadn’t been opened on the firewall.  This connection attempt was obviously failing, and the observed behavior was that some retries were attempted with fairly large gaps in between until Service Bus finally decided to fall back to port 443 (HTTPS) instead.  Pay particular attention to the lines in the following screenshot with the numbers 1681, 2105, and 2905 in the first column.

This explained why my service was taking a long time to start up (because Service Bus was failing to connect via AMQPS and was going through retry cycles before falling back to HTTPS) and also explained why my runtime performance was lower than my expectation (because HTTPS is known to be slower than TCP).  However this didn’t explain why Service Bus was attempting to use port 5671 rather than 9351-9354 as per documentation.

Repeating the same test on my own VM showed that Service Bus was continuing the connection on ports 9351-9354 as expected… So why the difference? On the suggestion of my colleague Mahindra, I compared the assembly versions of the Microsoft.ServiceBus assembly across the two machines. You can do this by running “gacutil -l Microsoft.ServiceBus” in a Visual Studio command prompt, or by manually checking the GAC directory which is typically “C:\Windows\Microsoft.NET\assembly\GAC_MSIL” for .NET 4+ assemblies.

Voila. I found that I was running version 2.1.0.0 on the machine that was behaving correctly, and version 2.6.0.0 on the machine that was misbehaving. It appears that the protocol choosing behavior for Service Bus relays has changed sometime in between these two assembly versions. I have not pinpointed exactly which version this change occurred in, and I don’t yet know whether this change was by design or accidental. Either way, Microsoft have not yet updated their documentation, which means that others will be as confused as I was.

So what are your choices?

1. You can downgrade to an older version of the assembly.  2.1.0.0 will definitely work for you, but you might be able to get away with a slightly higher version which is less than 2.6.0.0, but it will be up to you to establish which versions are acceptable since I haven’t managed to do this.  You’ll need to update the webHttpRelayBinding binding registration in your machine.config files (or wherever you’ve chosen to register it if you’ve gone for a more granular approach) to point to the older assembly as well.
2. You can choose to stick with the latest version of the assembly and open up outbound TCP connections on port 5671 on your firewall.

I chose to stick with option #1 because I’m not sure at this stage whether this change in behavior is intentional or incidental, and also because my impression is that raw TCP over ports 9351-9354 would be faster than the AMQPS protocol.  You will find that option #2 is also functional.

With the older version of the assembly in play I could not see traffic on ports 9351-9354 as expected, my services were spinning up in less than a second, and latency was much more in line with my expectations.

Getting Azure Service Bus Relays and the BizTalk WCF-WebHttp adapter to play nice with each other

This blog details problems that you will definitely encounter if you use the BizTalk WCF-WebHttp Receive Adapter in tandem with Azure Service Bus Relays (at least in BizTalk 2013 I haven’t confirmed if the problem has been fixed since but I doubt it). The problem is that if the service consumer tries to append URL parameters to the Service Bus Relay URL then they will be met with the dreaded AddressFilterMismatch exception. The traditional means of working around AddressFilterMismatch exceptions also leads to more problems which I will describe in this post. I have come up with a generic solution that ensures that you will not encounter such problems and that using the WCF-WebHttp BizTalk adapter in Receive Locations in tandem with Service Bus Relays is viable.

In this blog post I’ve detailed the problem, explained why it occurs, and explained how the solution works as well. If all you’re interested in is using the solution then you can download the installer here (or here if your machine is 32-bit), or the source code here.

The installer will GAC the relevant dll, and add a Service Behavior called BTWebHTTPSBRelayCompatibility to your machine.config files (if you prefer not to have your machine.config file updated then please don’t use the installer, download the source code and use the behavior as you wish). You will need to use this Service Behavior on your BizTalk WCF-WebHttp receive location to fix the problem detailed in this post. Please ensure you test the installer and the behavior before you attempt to use it in a real life environment, and let me know if you have any feedback.

 

Preamble about WCF-WebHttp Adapter and Service Bus Relays

Adding a relay endpoint to your WCF-WebHttp receive location is a piece of cake.  When running through the WCF Service Publishing Wizard, you just need to ensure you tick the checkbox in the below screenshot and you will be presented with additional screens at the final stages of the wizard that allow you to choose a service bus namespace and specify the ACS credentials that BizTalk uses to connect to the relay (I’ve found a way to get this to work with SAS, but more on this in another blog post).

The wizard doesn’t actually create the relay for you.  This is done at the time the service is started up based on entries that get placed in the service’s web.config file as a result of the wizard (you could even manually add the web.config sections if you missed the relevant wizard options).  Note in the below screenshot that an additional endpoint that uses the webHttpRelayBinding binding has been added to the web.config file.  The beauty of this is that you can change the relay endpoint URL anytime by just updating the web.config.

Now when the service spins up you’ll find that the relay has been enlisted automatically for you.  If you tear the service down, for example by taking down the application pool or stopping IIS, you’ll notice that the relay is no longer listed in the Azure Portal.

 

The Problem

What I found was that when I tried to send requests (regardless of the HTTP method) to the base URL of the relay (which in the above example is https://jctest.servicebus.windows.net/JCTestService/Service1.svc) there were no problems.  However the second I tried to add URL parameters to the end of the URL I would get an AddressFilterMismatch error.  I also encountered similar problems if I tried to change my relay address URL such that it differs from the URL format of my receive location (for example if I specify the relay URL in my web.config to be a value of https://jctest.servicebus.windows.net/API/V1).

 

The obvious workaround and why it isn’t good enough

The aforementioned error message shouldn’t be new to anyone who has used the WCF-WSHttp receive location, as similar errors are encountered in scenarios where the URL that service consumers post to doesn’t match the server name (due to a DNS alias/load balancer etc…). The most common solution I’ve seen touted around to get around this problem is to use a WCF Service Behavior that changes the Dispatcher AddressFilter to a MatchAllMessageFilter which basically means that URL matching will not occur. This would be done in the ApplyDispatchBehavior method of the Service Behavior as below.

The nice thing about Service Behaviors is that they apply to all endpoints within our service, including our relay endpoint. Adding this behavior to the behaviors tab of our receive location will actually get around the two aforementioned problems which resulted in AddressFilterMismatch errors, however they introduce a new, very BizTalk specific problem.
You’ll now find that regardless of your URL/Method/Operation mapping on your BizTalk Receive location, the BTS.Operation context property will never be promoted on your inbound message. This is very limiting, because the BTS.Operation context property is a fantastic way to route your message to relevant orchestrations/send ports without having to have their filters based on the URL of the message. Any URL parameters that you tokenized in the mapping will also not be written to the context.

 

Why the Problem Occurs
Getting to the bottom of the problem required a fair amount of usage of RedGate’s fantastic Reflector tool, I don’t think anyone could possibly figure out the problem without a reflection tool or having access to the source code.
I reflected the assemblies that are used within the BizTalk receive location and found that the way Microsoft implement the URL/Method to context property mapping is via a private Endpoint Behavior which is aptly named InternalEndpointBehavior. Microsoft actually adds this behavior to the default endpoint of the receive location programmatically within the adapter code, and the behavior does not get added to the relay endpoint which is why the relay endpoint behaves differently from the default endpoint.
The InternalEndpointBehavior sets the AddressFilter on the default endpoint to an instance of the WebHttpReceiveLocationEndpointAddressFilter class. This AddressFilter is what does all the magic of promoting the BTS.Operation and other context properties based on the inbound URL/Method. Unfortunately this class is marked as internal as well, so we can’t instantiate an instance of it from a custom behavior.
Since our first solution attempt was to use the MatchAllMessageFilter on our endpoint we would not see this context property promotion occur.
 
The Solution

The solution I’ve put in place is to create a custom service behavior that performs the below logic in it’s ApplyDispatchBehavior method.  You can download all the source code here.

First the behavior needs to figure out which ChannelDispatcher is related to the relay (since we’ll have the relay as well as the default ChannelDispatcher to contend with).

Next the behavior needs to loop through all endpoints, saving the base endpoint addresses for the relay endpoint as well as the default endpoint to local variables (the reason for this will be apparent soon). It also identifies the default endpoint and copies a reference to the contained InternalEndpointBehavior (which as I’ve mentioned applies the WebHttpReceiveLocationEndpointAddressFilter AddressFilter to the endpoint).

Next we’re going to apply the InternalEndpointBehavior to the relay endpoint which will apply the WebHttpReceiveLocationEndpointAddressFilter to the relay endpoint when it is executed.

This actually isn’t the end. This solution is good enough if the URL of the Service Bus relay mirrors the on-premise URL (except for the host name of course), however the second we want our relay URL to differ in structure (perhaps because we want to remove the .SVC extension, or provide a more friendly URL format) we will start to get an AddressFilterMismatch exception. The reason for this is that the URL format that the filter is searching for is based on the default endpoint’s format.  And because the filter is an internal class we have no ability to override the expected URL format.
So if your relay endpoint URL is https://jctest.servicebus.windows.net/test/service1.svc and the default endpoint URL is https://servername.onpremises.com/test/service1.svc then you won’t have a problem. But if the relay endpoint URL was https://jctest.servicebus.windows.net/API/V1 then you’d get the dreaded AddressFilterMismatch exception. For the default non-relay endpoint, the typical solution to this would be to use IIS’ URL Rewrite feature as described here, but I have found that this didn’t work for me with the Relay Endpoint.

So to get around this, we need to replace the Service Bus URL with the default endpoint URL in the To WCF Addressing header before the message gets to the AddressFilter.  I haven’t found an appropriate component that can change the To SOAP header value before the AdressFilter stage, so the solution I’ve implemented is to delay the execution of the AddressFilter such that it runs in a custom Message Inspector which will first override the To WCF Addressing header value such that it matches the AddressFilter’s accepted format, execute the AddressFilter, and then change the To WCF Addressing header back to it’s original value.

I’ll show you how I’ve achieved the aforementioned solution.  We now have the WebHttpReceiveLocationEndpointAddressFilter filter applied to our relay endpoint but this won’t work for us because we can’t run the Message Inspector prior to the filter.  So what we’ll do instead is copy a reference to the WebHttpReceiveLocationEndpointAddressFilter filter to a local variable, and change the filter on our relay endpoint to be a MatchAllMessageFilter, which will let pretty much any URL through (don’t worry, we’ll run our WebHttpReceiveLocationEndpointAddressFilter later).

The final step in our behavior is to now instantiate our custom Message Inspector, passing in the base address of the default endpoint, the base address of the relay endpoint and the reference to the WebHttpReceiveLocationEndpointAddressFilter filter object.  We then add this Message Inspector to our relay endpoint dispatcher.

The custom Message Inspector is very simple. All it needs to do is override the To WCF Addressing Header from the Relay Endpoint URL format to the default URL format, execute the AddressFilter, and then set the To WCF Addressing Header back to the Relay Endpoint URL format.

The beauty of this solution is that no change is introduced to the default endpoint at all, so you are only changing the behavior of the relay endpoint such that it matches the behavior of the default endpoint. Also, the code in the Service Behavior class will only execute when your service is first spun up. From then on the only code that will execute as each request is received will be the custom message inspector which is pretty lightweight and does very little additional work that isn’t encountered on the default endpoint anyways.  You can now set your Service Bus Relay endpoint URL to be whatever you want, and messages will get successfully relayed to your Receive Location with the BTS.Operation and the context properties based on tokenized URL parameters promoted/written to context as expected.

Of course, what would be even better would be if this behavior wasn’t necessary at all and that the BizTalk WCF-WebHttp receive location automatically set the appropriate filter on the relay endpoint for you…maybe Microsoft will adress this in BizTalk Server vNext? 🙂  Until then hopefully this will do the trick for you. If you have any better solutions then please do share them in the comments section.

Concatenating string constants and function outputs in Azure Logic Apps

Azure App Services and Logic Apps have now hit public preview, and they have generated a lot of excitement (including with this writer).  It’s a bit like Microsoft has hit a reset button and now all the Microsoft integration specialists have to start from scratch (at least in the cloud, BizTalk Server isn’t going anywhere for awhile, thankfully since I would miss it), at least when it comes to tooling, concepts don’t really change…

One of the first proof of concepts I tried to work with (actually this was a team effort with a bunch of integration specialists excitedly fighting over the keyboard) was creating a Logic App that would pick up emails from a POP3 mailbox, and write the contents to a folder in my DropBox account, the file being named based on the subject of the email + .txt, and the contents of the file being based on the body of the email.  This is easy enough to do with the POP3 and the DropBox connectors as below.

We had quite a lot of difficulty trying to figure out how to set the filename in DropBox since it required a combination of hardcoded strings and the email subject which flowed out of the POP3 connector.  We tried a kazillion different combinations, and nothing seemed to work.

I reached out to the BizTalk Advisors Yammer forums hosted by Microsoft and got a helpful response from Stephen Siciliano at Microsoft in a snap (that’s Stephen!).  His suggestion is that if you’re mixing string constants and functions then you should not surround your function with quotes, and you should not use the @ escape character before the function.  Also, if you want to concatenate strings you must use the @concat function.  My final function looked like the below.

@concat(‘jc\’, triggers().outputs.body.Subject, ‘.txt’ )

This resulted in the email body being written to a file in a folder in the root path called jc, with the filename being based on the email subject + .txt.  Interestingly, if you view the code view you’ll notice that the slash after the folder name gets automatically escaped (so you don’t have to escape it, in fact if you do then you will end up with two escaped slashes).

I hope this helps others who are also struggling to learn the new syntax.  Just like with any new language, once you understand the rules the learning curve is a lot less steep.  Don’t take it for granted that this syntax will remain the same.  I imagine that by the end of the public preview Azure App Services might be a very different beast.

Happy AAS-ing.