Latest Entries »

This week a BizTalk solution I had designed encountered an exception I’ve never seen before, as below.

System.ServiceModel.Security.MessageSecurityException: Security processor was unable to find a security header in the message. This might be because the message is an unsecured fault or because there is a binding mismatch between the communicating parties. This can occur if the service is configured for security and the client is not using security.

The issue was intermittent, and encountered while communicating with a third party web service.  Said web service was based on the Java stack rather than .NET and was being hosted by an ESB unknown to me.  This web service was based on the WS-I Basic Profile 1.1 so I was using a WCF-Custom adapter (mainly because I employed some custom behaviors to work around some funny behavior with this web service) employing the basicHttpBinding.  The security mode was set to transportWithMessageCredential while the clientCredentialType was set to Username, so I was effectively using SSL to encrypt my message with the username and password contained within the SOAP Headers in my request.

Seeing as the target system was a Java system and the error raised by the send port was clearly a .NET type error, I immediately suspected that the WCF stack on the BizTalk server must be raising the exception rather than the web service being called upon.  To prove my theory I decided to enable WCF tracing by adding the below within the system.serviceModel section of my BTSNTSVC64.exe.config file (the send host instance was 64-bit) and restarted my send host instance.

  <diagnostics>
    <messageLogging
         logEntireMessage="true"
         logMalformedMessages="true"
         logMessagesAtServiceLevel="true"
         logMessagesAtTransportLevel="true"
         maxMessagesToLog="300000"
         maxSizeOfMessageToLog="200000"
  />
  </diagnostics>
</system.serviceModel>
<system.diagnostics>
  <sources>
    <source name ="System.ServiceModel" switchValue="Verbose">
      <listeners>
        <add name="xml" />
      </listeners>
    </source>
    <source name ="System.ServiceModel.MessageLogging"
            switchValue="Verbose, ActivityTracing">        
      <listeners>
        <add name="xml" />
      </listeners>
    </source>
    <source name ="System.Runtime.Serialization" switchValue="Verbose">
      <listeners>
        <add name="xml" />
      </listeners>
    </source>
 </sources>
 <sharedListeners>
    <add name="xml" type="System.Diagnostics.XmlWriterTraceListener"             
         traceOutputOptions="LogicalOperationStack"
         initializeData="C:\log\WCFTrace.svclog" />
 </sharedListeners>
 <trace autoflush="true" />
</system.diagnostics>

With tracing enabled I played another few messages through BizTalk until I had a few success and failure scenarios. Studying the trace outputs using the Microsoft Service Trace Viewer tool I found that I was always getting a response from the service in question, but in the successful response messages I had SOAP security headers included, whereas in the failure messages (which were otherwise normal response messages from the service) I didn’t.

Below is a snippet of the SOAP headers for a success message.

<SOAP-ENV:Header>
  <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" SOAP-ENV:mustUnderstand="1">
    <wsu:Timestamp xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="Timestamp-96477">
      <wsu:Created>2015-02-25T02:22:38.760Z</wsu:Created>
      <wsu:Expires>2015-02-25T02:27:38.760Z</wsu:Expires>
    </wsu:Timestamp>
  </wsse:Security>
</SOAP-ENV:Header>

Whereas I found that the SOAP headers were completely empty in the case of the failure messages. The problem was clearly on the service side, and we told the third party in charge of the service that they needed to resolve this. However I decided that I wanted to dive a bit deeper and find out whether there was any way to get WCF/BizTalk to support the missing SOAP security headers.
The first port of call was to attempt to replicate the problem. I managed to create a stub for the service with the exact same binding using .NET on my development VM using all the same bindings and authentication modes. I struggled to find a way to programmatically remove the headers from my message. I suspected that I could write a Message Inspector WCF behavior to do so, but I was too lazy :)
Fiddler came to my rescue. I enabled the logging of HTTPS traffic, and toggled Fiddler to automatically apply breakpoints on responses.  The beauty of fiddler is that once it applied a breakpoint on the response, I was able to decrypt it (since it was encrypted using HTTPS), tamper with it (to remove the security SOAP headers) and then submit my tampered response back to BizTalk.  Voila, I had now found a way to reproduce my problem.

The next step was to find a way to work around the problem.  I found that there was no way to do so with the basicHttpBinding that I was using (and I’d bet that this wouldn’t be available with the other out of the box bindings either).  I changed my send port to use a customBinding instead, replicating my basicHttpBinding configuration by employing textMessageEncoding (very importantly setting the messageVersion attribute to Soap11 since I had to abide by the WS-I Basic Profile 1.1), httpsTransport, and the security binding elements.  I configured the security element such that the authenticationMode was set to UserNameOverTransport to mirror the UserName clientCredentialType in my original configuration.  Now the magic was to set the enableUnsecuredResponse attribute to true in the security binding element.

Port

Doing this resulted in BizTalk being able to consume the response messages, regardless whether the security headers were included in the response or not.  If for whatever reason the third party in question was unable to fix the problem on their end I now had a fallback plan.

The final piece of the puzzle was trying to see if I could get my .NET based WCF service (not BizTalk) to return a response without the security SOAP headers, so that I wouldn’t have to use Fiddler to replicate this problem anymore.  Armed with the knowledge about the enableUnsecuredResponse property on the security binding element I proceeded to switch my WCF Service from using basicHttpBinding to also employ a custom binding.

My original binding looked like the below.

<basicHttpBinding>
  <binding name="BasicHttpEndpointBinding">
    <security mode="TransportWithMessageCredential">
      <message clientCredentialType="UserName" />
    </security>
  </binding>
</basicHttpBinding>

While my updated binding looked like the below.

<customBinding>
  <binding name="customBasicHttpBinding">
    <security authenticationMode="UserNameOverTransport"  enableUnsecuredResponse="true"/>
    <textMessageEncoding messageVersion="Soap11" />
    <httpsTransport />
  </binding>
</customBinding>

Since I had included the enableUnsecuredResponse attribute in my security binding element, my WCF service was now returning responses without the security SOAP headers. The same would apply for a BizTalk receive location using a custom binding as well. Would you ever want to employ this trick? I’d say the answer is a resounding no since this enables non-standard behavior, unless for whatever reason you are backed into a corner on a project where you’re integrating with a fiddly system that will not accept the security SOAP headers (I can’t imagine this being the case but then again I wouldn’t be overly surprised).

I recently worked on a POC which had me spinning up an EDIFACT/AS2 exchange based on both BizTalk Server and MABS (Microsoft Azure BizTalk Server) and comparing the experiences.  Seeing as this was my first time dealing with AS2 (EDIFACT was already an old friend) the very first thing I worked on was finding a means of sending messages to the AS2 onramps.

I found a few open source programs that would let me submit AS2 messages but I decided I wanted to do this programmatically instead via BizUnit 4.0 so I had easily repeatable scenarios, which I could easily re-test across BizTalk Server and MABS.  Matt Frear has written a good post on how to send messages via AS2 using .NET, and I decided I was going to convert this into a BizUnit test step (with his blessing).

You can find the source code for the AS2SendTestStep here, or if you just want to download the assembly then you can do so here.

The test step support the below features.

  • Supports BizTalk Server and MABS (and theoretically other AS2 server products, but I haven’t tested any others).
  • Supports BizUnit data loaders which means that if you have many tests which require only slight variations to input messages then you don’t need to keep multiple copies of input files. You can just have a single base file and use a relevant DataLoader class to manipulate the stream as it is being read.
  • Supports signing and encrypting outbound messages.
  • Supports optional synchronous MDNs and allows you to validate them.
  • Logs decrypted MDN messages to the test execution report.

Usage tips are below.

  • You will need to reference the BizUnit.AS2TestSteps.dll assembly and the BizUnit.dll assembly provided by the BizUnit 4.0 framework.
  • Select an input file directly by supplying the file name into the InputFileLocation property or supply a BizUnit DataLoader (which allows you to manipulate the file as you read it in) in the InputFileLoader property.
  • Encrypt the outbound message – you will need a copy of the public key certificate and will need to supply the file path to it in the EncryptionCertificateFileLocation property.
  • Sign the outbound message – you will need a copy of the private key certificate and will need to know the password.  You will need to supply the path to the certificate to the SigningCertificateFileLocation property and the password to the SigningCertificatePassword property.
  • Supports setting AS2From and AS2To headers via the As2From and As2To properties.
  • Supports the use of a proxy server by setting appropriate values to the Ps property which allows you to supply the proxy server URL, and if required credentials as well.
  • Allows you to set the subject HTTP header by setting the As2FileName property.
  • Allows you to set the URL to post the request to by setting the Url property.
  • Allows you to override the default timeout of 20 seconds by setting the TimeoutMilliseconds property.
  • Allows you to run BizUnit substeps against the decrypted response message in case you want to validate it by suppying substeps into the SubSteps property.

An example test which sends an EDIFACT message to BizTalk Server (using a proxy server) and runs some regular expressions against the synchronous MDN response is below.  Note that the RegexValidationStep in use here is not part of the BizUnit framework and is proprietary so sorry I can’t share that.


var testCase = new BizUnit.Xaml.TestCase();

var as2TestStep = new AS2SendTestStep();

as2TestStep.As2From = "FABRIKAM";

as2TestStep.As2To = "CONTOSO";

as2TestStep.EncryptionCertificateFileLocation = @"c:\host.cer";

as2TestStep.As2FileName = "EFACT_D95B_CODECO_output.txt";

as2TestStep.InputFileLocation = @"c:\temp\EFACT_D95B_CODECO_output.txt";

as2TestStep.SigningCertificateFileLocation = @"c:\fab.pfx";

as2TestStep.SigningCertificatePassword = "test";

as2TestStep.TimeoutMilliseconds = 20000;

as2TestStep.Url = "http://localhost/AS2Receive/BTSHTTPReceive.dll";

WebTestPlugins.AS2Helpers.ProxySettings ps = new WebTestPlugins.AS2Helpers.ProxySettings();

ps.Name = "http://proxyserver.test.co.nz";

as2TestStep.Ps = ps;

var regexValidationStep = new RegexValidationStep();

regexValidationStep._RegexDefintion.Add(new RegexDefinition("Content-Type: message/disposition-notification"));

regexValidationStep._RegexDefintion.Add(new RegexDefinition("Disposition: automatic-action/MDN-sent-automatically; processed"));

regexValidationStep._RegexDefintion.Add(new RegexDefinition("Final-Recipient: rfc822; CONTOSO"));

as2TestStep.SubSteps.Add(regexValidationStep);

testCase.ExecutionSteps.Add(as2TestStep);

var bizUnitTest = new BizUnit.BizUnit(testCase);

bizUnitTest.RunTest();

And below is a screenshot of the test execution results.  Note that the MDN text is logged here, and you can see the regular expressions being evaluated against the MDN as well.

Capture

You can of course take your own test steps much further by validating that BizTalk/MABS has consumed the file and written the message to its target destination, be it a SQL database or a file system folder etc… using the out of the box BizUnit test steps or your own custom ones.

The test step doesn’t currently support compression.  If anyone can point me towards any good documentation on compressing AS2 messages in C# then I might try to include that.

Any other suggestions are welcome.

I thought I would share a trick which might seem obvious to those who know it already, but should seem like an eureka moment to those who don’t.

How many of you have tried to use the Performance Monitor tool to try to track resource usage of a BizTalk application and have been stumped as to which Host Instances you are actually tracking?  The problem with Performance Monitor is that for many of the counters (especially the generic non-BizTalk specific counters such as Process or Memory) the Host Instance names are not listed, but instead all you see are the service names (BTSNTSVC or BTSNTSVC64) with a counter based on how many instances of that specific service are currently running, as in the below screenshot.

Random counters

You could be creative (you won’t believe the number of stories I’ve heard of how people work around this) and start your Host Instances one at a time and try to figure out which one relates to which Performance Monitor instance but there is a better way.  The key is to add the performance counter called “ID Process” under Process for each of the different BizTalk Services as in the below screenshot.

ID Process List

Now that you’ve added the ID Process performance counters, you can click on any of the specific instances in Performance Monitor and if you take a look at the values in the Last/Average/Minimum/Maximum columns (they’ll all have the same value) you’ll find that they contain the value of the PID (Process ID) for a specific host instance which you can find under the Services tab of the Windows Task Manager as per the below screenshot.  You have now built an association between the Performance Monitor Instance and the BizTalk Host Instance, and those associations will carry over to other Performance Monitor counters.

Link

Something to keep in mind is that when you restart Host Instances they will get new PIDs, and if you restart all the Host Instances at the same time there is a possibility that the order of Instances in Performance Monitor might get swapped around.  My recommendation is that if you are currently profiling your BizTalk environment using Performance Monitor and you want consistent results, you should either track your Data Collector Sets to separate output files each time you restart your Host Instances, or you should look into restarting your Host Instances one at a time to maximize (I’m not sure if this is a guarantee) the chance that the order will stay the same.  Regardless which path you choose, it is important to take note of which Host Instance relates to which Performance Monitor instance, especially if you plan on studying the results in the future at which point it might no longer be possible to link the associations.

I have now uploaded the BRE Pipeline Framework v1.5.4 installer to the CodePlex project page.  This version is as always fully backwards compatible with BRE Policies created with previous versions, and installing it over an existing version simply requires uninstalling the currently installed version, installing the new version, and importing the new vocabularies from the program files folder (default location is C:\Program Files (x86)\BRE Pipeline Framework\Vocabularies).

In a previous post I  discussed a pattern that allows for the temporary persistence of context properties on a request message in a request/response send port, and then reapplying those context properties on the response message before it gets sent to the BizTalk message box.  The beauty of this pattern is that it allows you to enrich response messages with state from the request message without the need for orchestration.  The temporary persistence store that I used for these context properties is based on the .Net MemoryCache class.

One of the features of the MemoryCache class is that it is really easy to set expiry times on cached items so you aren’t forced to explicitly remove items from the cache (though you still have the ability to), and setting expiry times and explicitly deleting cached items are features provided for in the BRE Pipeline Framework.

The MemoryCache class also has a feature whereby if the server is under heavy pressure for memory it will automatically drop items from the cache.  This is great when the items in the cache are truly temporary and you have a means of reloading the item in the cache, however doesn’t work very well for the pattern described in my previous post which depends on the cached items being available.  To get around this I have taken advantage of the Priority property in the CacheItemPolicy class, which allows you to override this default behavior, instructing .Net not to automatically remove your cached items.

This feature is introduced in the BREPipelineFramework.SampleInstructions.CachingInstructions vocabulary v1.1’s vocabulary definition called SetCachingPriority.  You have two priority values that you can apply as below.

  • Default – this implies that .Net should remove cached items automatically if under memory pressure.  If you don’t use the SetCachingPriority vocabulary definition then this value will be automatically chosen.  Cached items can also be removed based on expiry time or based on explicit removal.
  • NotRemovable – this instructs .Net not to ever remove cached items, even if under memory pressure.  Cached items can only be removed based on expiry time or based on explicit removal.

An example usage of the SetCachingPriority vocabulary definition is in the screenshot below.

VocabDef

A few years ago I successfully implemented a BizTalk solution making use of a WCF-NetMsmq receive location which was used to receive XML messages into BizTalk.  The solution worked really well for a good couple of years until a few months ago when the environment encountered a major unrelated outage during which a pretty nasty side effect, poisoned messages, was encountered.  The outcome of this experience is that I will only use the WCF-Custom adapter with a netMSMQBinding rather than using the WCF-NetMsmq adapter which has no capability to handle poisoned messages.

The outage was caused due to the file server on which the BizTalk database transaction logs were stored running out of disk space.  As a result all transactions that BizTalk attempted to make failed and were rolled back, including transactions trying to commit the received messages from the transactional MSMQ queue to the BizTalk message box.  The problem with this is that after retries are exhausted (the default MSMQ retry settings dictate that there will be 5 retries with 30 minute intervals between them) the message will be considered a poisoned message, and the receive location will be stuck in a faulted state until the message has been removed from the queue.  The WCF-NetMsmq adapter doesn’t allow the poison message handling settings to be overridden.

Thankfully, if you use the WCF-Custom adapter with the netMsmqBinding binding instead you will find that you have full control over the poison message handling settings (these settings are detailed in this MSDN article).  You’ll of course be able to override the number of retries and retry interval, but the setting which is most important to us is the one called ReceiveErrorHandling.  When using the WCF-NetMsmq adapter this setting is set to “Fault”, which means that poisoned messages remain in the queue and no messages can be consumed till the message has been removed.  We can instead set this to “Drop” if we want to get rid of the message automatically, “Reject” if we want to drop the message and send a negative acknowledgement back to the sending queue, or “Move” if we want to move the message to a queue (actually a sub-queue of the main queue) called poison.  Note that the aforementioned MSDN article states that the “Reject” and “Move” options are only available on Windows Vista, however I have successfully tested them on Windows Server 2008R2, and Windows Server 2012, and would be extremely surprised if they don’t also work on Windows 7 and Windows 8/8.1.

PoisonMessageSettings

We chose to go down the “Move” path ourselves because that results in the continued processing of our queue and allows us to deal with the poisoned message in our own time.  You’d need to find a smart way to deal with the poisoned messages, building some sort of notification process to ensure that messages don’t get left in the poison queue indefinitely.  One option is to use a vanilla MSMQ (rather than WCF based, in case the reason the message is poisoned is due to a problem with the SOAP envelope or malformed XML) receive location to receive messages off the poison queue (the URL for the poison queue is in the following format – net.msmq://<machine-name>/applicationQueue;poison) thus kicking off your notification process.

One takeaway from this post is to avoid using the WCF-NetMsmq adapter and to use it’s more flexible cousin, the WCF-Custom adapter with the netMsmqBinding binding.  I would extend this advice to the majority of the WCF adapters, since using the WCF-Custom adapter generally affords you a lot more flexibility in the way of being able to make use of WCF behaviors and binding settings that the WCF adapters don’t always expose.  The one definite exception (at least that I can think of) to this advice is for the webHttpBinding (i.e. REST) in which case it is best to use the WCF-WebHttp adapter, since it gives you access to URL variable mapping which you will not find on the WCF-Custom adapter.  Luckily Microsoft allow you to use WCF behaviors on the WCF-WebHttp adapter so you don’t lose anything by avoiding the WCF-Custom adapter in this case.

Another takeaway is that you should always consider how you will deal with poisoned messages when consuming messages off an MSMQ queue with BizTalk.  Even if you have full control over the WCF clients that send messages to the queue, you might still encounter poisoned messages out of no fault of the message sender, like in the outage scenario I described above.  It is best to familiarize yourself with the concept of poisoned messages and plan how you will handle them, rather than find yourself in the position of having to figure it out during a production outage (as was the case for me).

Follow

Get every new post delivered to your Inbox.

Join 88 other followers

%d bloggers like this: