Category: EDI


I recently worked on a POC which had me spinning up an EDIFACT/AS2 exchange based on both BizTalk Server and MABS (Microsoft Azure BizTalk Server) and comparing the experiences.  Seeing as this was my first time dealing with AS2 (EDIFACT was already an old friend) the very first thing I worked on was finding a means of sending messages to the AS2 onramps.

I found a few open source programs that would let me submit AS2 messages but I decided I wanted to do this programmatically instead via BizUnit 4.0 so I had easily repeatable scenarios, which I could easily re-test across BizTalk Server and MABS.  Matt Frear has written a good post on how to send messages via AS2 using .NET, and I decided I was going to convert this into a BizUnit test step (with his blessing).

You can find the source code for the AS2SendTestStep here, or if you just want to download the assembly then you can do so here.

The test step support the below features.

  • Supports BizTalk Server and MABS (and theoretically other AS2 server products, but I haven’t tested any others).
  • Supports BizUnit data loaders which means that if you have many tests which require only slight variations to input messages then you don’t need to keep multiple copies of input files. You can just have a single base file and use a relevant DataLoader class to manipulate the stream as it is being read.
  • Supports signing and encrypting outbound messages.
  • Supports optional synchronous MDNs and allows you to validate them.
  • Logs decrypted MDN messages to the test execution report.

Usage tips are below.

  • You will need to reference the BizUnit.AS2TestSteps.dll assembly and the BizUnit.dll assembly provided by the BizUnit 4.0 framework.
  • Select an input file directly by supplying the file name into the InputFileLocation property or supply a BizUnit DataLoader (which allows you to manipulate the file as you read it in) in the InputFileLoader property.
  • Encrypt the outbound message – you will need a copy of the public key certificate and will need to supply the file path to it in the EncryptionCertificateFileLocation property.
  • Sign the outbound message – you will need a copy of the private key certificate and will need to know the password.  You will need to supply the path to the certificate to the SigningCertificateFileLocation property and the password to the SigningCertificatePassword property.
  • Supports setting AS2From and AS2To headers via the As2From and As2To properties.
  • Supports the use of a proxy server by setting appropriate values to the Ps property which allows you to supply the proxy server URL, and if required credentials as well.
  • Allows you to set the subject HTTP header by setting the As2FileName property.
  • Allows you to set the URL to post the request to by setting the Url property.
  • Allows you to override the default timeout of 20 seconds by setting the TimeoutMilliseconds property.
  • Allows you to run BizUnit substeps against the decrypted response message in case you want to validate it by suppying substeps into the SubSteps property.

An example test which sends an EDIFACT message to BizTalk Server (using a proxy server) and runs some regular expressions against the synchronous MDN response is below.  Note that the RegexValidationStep in use here is not part of the BizUnit framework and is proprietary so sorry I can’t share that.


var testCase = new BizUnit.Xaml.TestCase();

var as2TestStep = new AS2SendTestStep();

as2TestStep.As2From = "FABRIKAM";

as2TestStep.As2To = "CONTOSO";

as2TestStep.EncryptionCertificateFileLocation = @"c:\host.cer";

as2TestStep.As2FileName = "EFACT_D95B_CODECO_output.txt";

as2TestStep.InputFileLocation = @"c:\temp\EFACT_D95B_CODECO_output.txt";

as2TestStep.SigningCertificateFileLocation = @"c:\fab.pfx";

as2TestStep.SigningCertificatePassword = "test";

as2TestStep.TimeoutMilliseconds = 20000;

as2TestStep.Url = "http://localhost/AS2Receive/BTSHTTPReceive.dll";

WebTestPlugins.AS2Helpers.ProxySettings ps = new WebTestPlugins.AS2Helpers.ProxySettings();

ps.Name = "http://proxyserver.test.co.nz";

as2TestStep.Ps = ps;

var regexValidationStep = new RegexValidationStep();

regexValidationStep._RegexDefintion.Add(new RegexDefinition("Content-Type: message/disposition-notification"));

regexValidationStep._RegexDefintion.Add(new RegexDefinition("Disposition: automatic-action/MDN-sent-automatically; processed"));

regexValidationStep._RegexDefintion.Add(new RegexDefinition("Final-Recipient: rfc822; CONTOSO"));

as2TestStep.SubSteps.Add(regexValidationStep);

testCase.ExecutionSteps.Add(as2TestStep);

var bizUnitTest = new BizUnit.BizUnit(testCase);

bizUnitTest.RunTest();

And below is a screenshot of the test execution results.  Note that the MDN text is logged here, and you can see the regular expressions being evaluated against the MDN as well.

Capture

You can of course take your own test steps much further by validating that BizTalk/MABS has consumed the file and written the message to its target destination, be it a SQL database or a file system folder etc… using the out of the box BizUnit test steps or your own custom ones.

The test step doesn’t currently support compression.  If anyone can point me towards any good documentation on compressing AS2 messages in C# then I might try to include that.

Any other suggestions are welcome.

While implementing an EDI exchange using BizTalk Server 2010 I ran into a rather perplexing problem.  There was a requirement that the system should be able to support any identification qualifiers (in the UNB2.2 / UNB3.2 segments) that our trading partners prescribed, even if they were non-standard values.  The grief-inducing scenario was that some of the trading partners wanted to use the qualifier ZZ which is normally used for mutually defined identifications in X12 messages, however the problem in this case was that we were dealing with EDIFact messages for which ZZZ is normally the qualifier for mutually defined identifications and ZZ is not supported.

I followed the steps documented in this blog post by Miguel Herrera which describe how to support custom values in the UNB2.2 and UNB3.2 segments and while I now get the ZZ option in drop downs in parties I still run into trouble.

The first thing I tried doing was create a Party/Profile and manually add an identity to the profile with ZZ as the Qualifier and the party identifier as the Value, providing an arbitrary value for the Name since it isn’t actually used by the BizTalk runtime.

Manual profile

I made an agreement using that profile and the first thing I noticed was that the identifiers for the Recipient party were not pre-populated and the value Recipient did not show up in the Identification drop down box.  I then tried to manually enter these values in the identifiers section however when I tried to save the agreement I got the error – “An identity QualifierIdentity:ZZ:Recipient already exists in the profile”.

PartyError

I then deleted my parties/agreements and started again from scratch.  This time I created the parties/profiles but did not add any identities to them.  I created an agreement and manually added the identification and qualifier (using ZZ which is in the drop down box since we followed the instructions in Miguel’s blog post) values as desired.  This works at runtime however it does cause other problems.  If you open the relevant profile you will notice that an identification record has automatically been created using the values that were manually entered in the agreement identifiers tab.  However there is no name listed for this record.  While one might think that this isn’t a big deal since the Name isn’t actually used at all at runtime, it will actually cause you major grief the next time you decide to update the profile.

One of the issues is if you try to manually add an identity to the profile you will get the error “An identity cannot have an empty name” since the identity that was automatically created when you saved the agreement has no name.  You will most likely get this error if you try to make any changes to the profile and apply them.

Unable to add identity

If you try to add a name to the problematic identity you will run into the error “An identity QualifierIdentity:ZZ:Recipient already exists in the profile” since apparently the MMC doesn’t end up updating the identity but tries to add a new one.

Profile Error

You will also find that having an identity with no name will result in the BizTalk Documenter not working against your BizTalk group anymore, raising an error “System.InvalidCastException: Unable to cast object of type ‘System.DBNull’ to type ‘System.String'” since it always expects to find an identity Name.

I consider this to be a bug in the MMC and do intend to log it in the not too distant future.  Until then I have found a quick workaround as described below (disclaimer : approach with caution and do this on dev environments only).  Export your party setup to a bindings file (to do this choose to export bindings in the BizTalk Administration Console and ensure you choose to include global party bindings).  Open the binding file using a text editor and search for the qualifier in question that is causing the problem until you find the appropriate business profile as below.

BusinessProfile Before

Now manually add a description node just above the qualifier node and give it an appropriate value (which will be fed into the identity Name).

BusinessProfile After

Now you will need to delete the party (obviously this is an exercise you are undertaking on dev environments only, deleting parties/agreements on live environments is not at all recommended as you will lose any persisted information such as interchange control number counters etc…) altogether and then import the edited binding file.  If you inspect the re-created profile you should see that the value in the Description node has been placed into the Name for your identity.  You will now be able to use the BizTalk Documenter and add other identities to your profile.

Fixed

I have only encountered this problem using BizTalk 2010 and am not sure if it can be encountered in other versions.  I will update this blog post when this bug gets logged or if I find any more info.  If anyone else has any alternative suggestions to achieve what I am trying to do then please do let me know.

Torben Chrona Christiansen wrote a very handy blog post last year on how to bulk stop and start EDI batches in BizTalk Server 2009 using SQL queries. These queries can be used to aid in creating proof of concepts, to demonstrate batching in presentations, as a setup step in integration tests, or during deployments.

Seeing as the Trading Partner Management system has been overhauled to a large degree in BizTalk Server 2010 onwards with the introduction of partner agreements, these SQL queries unfortunately no longer work.

My most recent challenge has been to implement an EDI batching platform and I thought it would be very beneficial to rewrite these queries for BizTalk Server 2010. I have found them extremely useful during the POC stage to quickly stop, start and override (force a batch to release queued up messages even though the batch’s release criteria has not been met yet) batches, and I have also found that these queries are an absolute fundamental in building EDI batching integration tests as you need to ensure that all your batches are running and starting from a clean state (overriding helps tons here) in order to ensure a deterministic outcome. I have written BizUnit test steps built around these queries and have used them to a great degree.

All the below queries should be run against the BizTalkMgmtDb in order to take action against your batches.  Please note that this is not supported by Microsoft, has only been tested by myself to a certain extent, and you will need to judge for yourself how you will use this.

The below query can be used to start all batches that aren’t currently running.

Activate

The below query can be used to stop all running batches (note that this is a clean stop, any queued up messages will be batched up and sent out first).

Terminate

And lastly, the below query can be used to override all running batches, thus forcing them to send out any queued up messages rather than waiting for their regular release criteria.

Override

If anyone can come up with a cleaner way of performing the above then please do let me know. If you would like to download the SQL scripts they are located on my google drive.

Last week I ran into a really interesting issue. When I was running Visual Studio unit tests to validate instances of XML formatted EDIFact messages (see my previous blog post on BizTalk testing for more details on schema testing) I found that my tests just seemed to take forever and timed out with no reason provided.

Timeout

I decided to apply a break point in my unit test and debugged it.  Much to my surprise when I stepped into the line of code that was actually validating the instance file I saw the below.

EDIFact Properties

Well, that’s really painful.  The unit test was timing out because it was waiting for user input on the above screen, however unless the test was being run in debug mode the screen was not being displayed at all.

So far I haven’t found any way to get the out of the box ValidateInstance method to work on EDIFact schemas because I can’t supress the above screen.  Interestingly enough, if the instance file actually fails validation then the test will fail rather than time out.  Another alternative I explored was to use some non standard test methods to test instances of EDIFact messages against my schemas.  In my previous blog post discussing BizTalk unit testing I mentioned that I regularly use non standard methods to extend my schema unit tests anyways since the out of the box ValidateInstance method doesn’t return an error message upon failure, and it also doesn’t work when your schemas import/include from other schemas.  I chose to use only these extension methods (as described here), bypassing the out of the box testing methods altogether and the tests now complete succesfully.  Note that in the below screenshot the TestExtensions.ValidateSchema method contains an extension method as described here.

Success

Some of my colleagues asked me to help them out with a really hairy problem today.  Their project involved the receipt of EDIFact files by email.  The emails were being saved on the file system as .eml files which were then picked up and processed by BizTalk.  The email would contain the message body which was of content type text/plain and the EDI message which was an attachment of content type application/octet-stream.

Everything worked fine on development when they were working with eml files that were created by the developers, but when they tried to test the solution against an instance of a real eml file sent from one of the trading partners the receive pipeline failed with the following error – “A body part or a part with the same name has already been added to this message. A message can have only one body part and part names must be unique.”

Inspecting their pipeline showed that they were making use of a MIME/SMIME decoder component (they were selecting the body part by type application/octet-stream rather than index), an EDI disassembler component, an EDI Party Resolution component, and a whole lot more custom components. 

My initial gut feel was that the problem must lie in the disassemble stage so I got an example message off them and whipped up a receive pipeline containing the MIME/SMIME decoder component and the EDI Disassembler component, both with exactly the same settings as they had.  As soon as I ran the file through my pipeline I saw the same error.  To isolate the issue I tried to remove the EDI Disassembler from the pipeline and the error was no long encountered.

I opened the eml file in notepad and noticed that the content-description for the body of the message had a value of body (see below for an excerpt).  I changed the value to body1 and tried playing the message through my pipeline and this time it worked!   

I didn’t have a copy of reflector handy so I couldn’t quite inspect what the MIME/SMIME decoder or the EDI Disassembler component were doing, but my educated guess was that the MIME/SMIME decoder component would name each message part according to the content-description in the eml file, and that when the EDI Disassembler disassembled the raw EDIFact into XML and was trying to add it to the message it was setting the part name to body which resulted in a clash because one of the non-body parts was already name body.

My colleagues asked me to whip up a quick workaround for them which would ensure that if any of the non-body parts had a name of body that it would rename them to something else.  Due to time constraints I largely based my code off the ArbitraryXPathPropertyHandler pipeline component which is a Microsoft sample that can be found in C:\Program Files (x86)\Microsoft BizTalk Server 2010\SDK\Samples\Pipelines\ArbitraryXPathPropertyHandler on any BizTalk 2010 dev PC.

The execute method of my pipeline component looks like the below.  Note that it creates a new instance of a message and copies the context over (it does so by reference here, you can always do a manual copy of the context properties if you want to but since we are not manipulating the message whatsoever it isn’t entirely necessary), creates a new message part which points at a stream derived from the original message’s body part (again you could clone the stream rather than use the original stream if you wanted to), and then calls the CopyMessageParts method passing in the original message as the source, the copied message as the target, and the copied body part (the extraction of the body part in the execute message is somewhat unnecessary but since this was a quick I didn’t bother to vary from the Microsoft sample too much).  It then returns the copied message to the remaining components in the pipeline.

The CopyMessageParts method looks like the below.  It literates through each of the message parts in the source message and copies them over to the target (again, you could clone the message part data instead of copying it over by reference if you want to).  If however the name of a non-body part message is body then it will replace the part name with oldBody plus a GUID.

Placing this pipeline component in the decode stage after the MIME/SMIME decoder component now allows the EDI Disassembler to process successfully and work around the problem.  This pipeline component might not be 100% as I did this as a quick and dirty exercise, and my colleagues will most likely clean it up but I hope this helps others who bump into the same issue to identify their problem and come up with an appropriate workaround as well.

%d bloggers like this: