I have recently been working on an agile BizTalk project which has required me to flesh out testing frameworks for all the different aspects of BizTalk.  This has been a somewhat uncharted territory for myself in the past, though not for the lack of desire to explore this area, and was a great exercise for me.  In this post I will aim to discuss some of the tools I have used, some of the reasoning behind why I chose to use those tools, and some notes about BizTalk testing in general.  In later posts I will aim to do more of a deep dive into the tools mentioned.

The very first thing to mention is that in order to test your schemas, maps, and pipelines you will need to make a change to your project file.  You will need to open the project properties, navigate to the deployment tab.  On here, you will want to set the “enable unit testing” option to true.

What this does is change the code-behind for all your schemas, maps, and pipeline classes.  By default all schemas inherit from the base class SchemaBase (if you want to prove this, just click the expand button on the left of any of your schema files in a BizTalk project in the Visual Studio solution explorer and view the .cs file), however when you change the enable unit testing property for the project, all the contained schemas will instead derive from the base class TestableSchemaBase.  Before you start panicking about the fact that your schema has just been unnecessarily changed, the TestableSchemaBase class also derives from the SchemaBase class, it just implements a few extra properties/methods.  Similar behavior can be observed for the code-behind for maps and pipeline artifacts as well.  Using the Testable base class for your artifacts enables you to unit test them.

However, doing this also presents a problem.  You’ll notice that once you enabled unit testing on a project that a reference to BizTalk.TestTools has been added to the project as well.  On a typical non-dev BizTalk environment where you don’t have Visual Studio installed, this dll would not be in the GAC.  This means that if you deploy your project with unit testing enabled to such a server, that you would encounter massive failures at runtime.  One option is to of course make the missing dll’s available on those servers (you might find that you have good reasons to do this), but another somewhat cleaner approach is to only enable the unit testing property on your debug builds which will only be deployed to your development box, and ensure that it is turned off on the release builds which can then be deployed to your non-dev environments.  Purists would argue that you are now releasing assemblies which are actually different from those that you unit tested, but the way I see it there has to be some degree of trust that we allocate to Microsoft to have gotten this right🙂

On to the nitty gritty.  The first type of test I tried to implement was unit testing for schemas.  The idea of unit testing schemas is to validate instances of actual messages against a given schema, similar to right clicking on a schema file in the solution explorer and choosing validate instance.  I tried to follow the textbook examples for the out of the box unit testing for schemas (see http://msdn.microsoft.com/en-us/library/dd224279(v=bts.10) for an example) however I very quickly ran into some major roadblocks.

  • When tests fail they don’t report back to you the reason for the failure.  I consider this to be quite a hinderance as it then requires manual steps to be taken to find out the reason for failure.
  • If your schema imports types from another schema file then the test will always fail.

Clearly this is not quite acceptable so I started searching the internet for alternatives, and ended up implementing a static helper class that looks quite similar to those described here – http://stackoverflow.com/questions/751511/validating-an-xml-against-referenced-xsd-in-c-sharp.  This allows me to very quickly whip up a test with just a few lines of code to assert whether a given xml file validates against a specified schema.  Note that in the below example, you are able to specify whether the instance file is in native (flat file) or XML format.

Next up was to write unit tests for maps.  In this case I found that the out of the box map testing worked quite well (see here for some documentation – http://msdn.microsoft.com/en-us/library/dd224279(v=bts.10).aspx).  However once again I found a few things lacking with it.

  • For every variation in my input file that I wanted to test, I had to create and maintain an extra XML file.  You’ll find your repository of XML files building up very quickly.
  • The out of the box implementation of map unit tests helps you to execute the map, allowing for validation of the input and the output files, but does very little in the way of allowing you to actually check the data in the output file.  That is totally left upto you.

Once again, I decided to find out what those who came before me had decided to do, and it appeared that best of breed that suited my requirements was the BizTalk Map Testing Framework – http://mtf.codeplex.com/.  I must say that I am a huge fan of this framework because of the great flexibility it allows.  The way this framework works is that for each map you want to test, you supply a template input file and an output file.  You then supply xpath statements for all the nodes in the input file you want to vary, and the all the nodes in the output file that would be changed as a result of said variations.  You then create test scenarios, each one supplying values for the input nodes and expected values for the output nodes.  This allows you to very quickly create a lot of tests for complex maps, and if you find that you are dealing with a very simple map which only requires a simple test then just skip the xpaths and make use of the auto generated base test which ensures that when the map is executed against the template input file, that the output file matches the template output file.  The framework of course allows for actions to be taken before and after the map has actually been executed in order to cater for dynamic scenarios such as the output file containing the current time etc… and even provides some handy helper methods to read from and manipulate the output xml files before the test scenarios are executed.  I will definitely post in more detail about this in the future.

I had a glance at the out of the box unit testing for pipelines, and it didn’t look too bad at all.  However based on suggestions found around the internet I decided to make use of BizUnit 4.0 and the Winterdom pipeline testing framework which is very well documented here – http://blogdoc.biztalk247.com/article.aspx?page=a45b5fcd-a1fe-4219-844b-5e7e5660a4ec.  Once more, I can’t express how much of a convert I am to this framework and how easy it is to generate tests with this.  When setting up your test you can specify an input context file which contains all the context properties that should be applied to the message before the pipeline is executed (obviously only applies for send pipelines), the input file (or possibly files in the case of send pipelines which can be used for batching), and an instance config file which allows you to override the default parameters for the pipeline.  Executing the test will result in the generation of the resulting output message and optionally a resulting context file which contains all the context properties against the message after the execution of the pipeline.  Thanks to the robustness of the BizUnit 4.0 framework, it is very easy to implement validators against both of these files types.

Next up came integration testing.  By its very nature BizTalk is very much a black box and most people think of it as a system which messages go into, and messages eventually come out of.  In a large way this is what the focus of integration testing is, to evaluate the outcomes of a given scenario in a black box fashion.  Once again I decided to use BizUnit 4.0 as it is a very robust and flexible framework, with a very full featured set of test steps to carry out common functions required in integration testing.  I will have to write a whole other post regarding BizUnit 4.0 but I encourage all those interested in integration testing BizTalk or other types of integration projects to download and play around with the framework – http://bizunit.codeplex.com/

While we managed to create some pretty complex integration test scenarios, due to BizTalk’s black box nature we really had little idea whether we had actually managed to exercise all the different paths of our orchestration.  In comes the BizTalk Orchestration Profiler tool (http://biztalkorcprofiler.codeplex.com/) which allows you to view which parts of your orchestrations have been exercised, and which areas you need to focus more testing on.  A few tweaks are required to get the tool to work with BizTalk Server 2010, Colin Meade has made life easy for us by documenting the required steps on his blog – http://midheach.wordpress.com/2011/10/29/biztalk-orchestration-profiler-2010/.

Next on the list was performance testing.  The options really boiled down to using the load testing framework that is part of Visual Studio Ultimate or using LoadGen 2007 (earlier versions were not an option for me as I was testing against WCF transports which are only implemented from the 2007 version onwards).  In my case I had to go with LoadGen as the client did not have access to Visual Studio Ultimate, so that was an easy choice.  Since I had already used BizUnit 4.0 to implement my integration tests, I decided that it would be nice to use BizUnit 4.0 as a harness for LoadGen as well.  This also means that I could leverage of BizUnit test steps to actually validate the outputs of my performance test as well to ensure that all the records that I was expecting BizTalk to process were indeed being processed.

I hit a showstopper very quickly here.  LoadGen 2007 was unfortunately released in a state whereby its containing dll’s are not signed with a strong named key and thus can’t be added to the GAC as required by BizUnit 4.0.  This wasn’t an issue with earlier versions of LoadGen however as mentioned these weren’t an option to me as I needed to test against WCF transports.  Back to google once again I was guided to the following article – http://geekswithblogs.net/jorgennilsson/archive/2009/01/03/loadgen-2007-with-bizunit.aspx.  What this article describes is the process to disassemble the dll’s back to MSIL code, associate the IL files with a strong name key, and then update references in the dlls to also take note of the new strong name key.  Doing this and adding the resultant dlls to the GAC means that BizUnit 4.0 and LoadGen 2007 now play very nicely with each other, and allows for complex performance testing scenarios to be catered for.

While LoadGen 2007 allows for the generation of load on a server and BizUnit 4.0 test steps can be used to validate the outputs, an important part of performance testing is to study the impact that the load has on the BizTalk and SQL environments.  In order to do this I made use of the Performance Analysis tools in Windows Server 2008 and PAL (Performance Analysis of Logs – http://pal.codeplex.com/).  The idea is to generate data collection sets from Performance Monitor while the server is processing the load and to then have PAL process the resulting files.  It will then generate an HTML report for you which provides warnings when thresholds have been passed.  While this helps you analyze and search for problems, there are many important counters which need to be spot checked manually rather than solely relying on the PAL output, which is merely a guide and a starting point (albeit a fantastic one).

A further requirement which was explored, however was since dropped was to look at using Specflow (which is a BDD or Behavioral Driven Development testing framework – http://www.specflow.org/specflownew/) as part of our BizTalk integration testing.  Specflow allows for test analysts to specify tests scenarios using verbose statements which are based on the verbs Given (specifying prerequisites), When (specifying the actions to be taken in the test scenario), and Then (specifying the expected outcome of the test scenario).  Each sentence in the test scenario will end up being bound to a specific method which will then implement the actual test logic.  Each sentence in the test scenario can also specify variable inputs thus making the test methods quite reusable.  I chose to use BizUnit 4.0 to actually implement my test methods for all the reasons I mentioned previously.  Specflow also allows for your resulting trx test results file (assuming you are using MSTest to execute the tests) to be converted to a nice html format.  There are of course many free tools on the internet that allow you to do similar things (and one of my colleagues has even written a BizTalk project to do exactly this, now why didn’t I think of that first :)), but it is a nice little extra feature that comes with the toolset.

While there are many benefits to such a testing approach, especially for agile projects, it should be noted that there would be some up front overheads, especially with the first set of test scenarios being catered to.  My personal take on this is that the overhead mostly comes into play because each sentence on its own represents a test case, rather than the entire test scenario representing a test case.  Thus you have to structure your test cases quite carefully, in a reusable fashion (the reusability which might not necessarily be realized in some cases) and you have to think how you are going to flow test context between your different sentences.  The framework obviously caters for such things and is in no way limiting of what you are able to achieve, however it is up to each project to decide whether Specflow is a good fit for it or not.  There is some fantastic documentation about Specflow and BizTalk here – http://social.technet.microsoft.com/wiki/contents/articles/12322.behaviour-driven-development-with-biztalk.aspx (and do be sure to check out the very thorough videos by Michael Stephenson).

One final thing which I never managed to play around with myself, however did have on my wish list to investigate was to have my integration tests snoop on ETW tracing.  I have started using the CAT Instrumentation Framework in a large way (see http://blogs.msdn.com/b/appfabriccat/archive/2010/05/11/best-practices-for-instrumenting-high-performance-biztalk-solutions.aspx), especially in orchestrations to try to break down the black box barriers and get a peek into the internal state of the orchestration’s flow.  The “Testing inside BizTalk by using ETW Tracing” project (http://btsloggingeventsinbi.codeplex.com/) leverages off the ETW tracing enabled by the CAT Instrumentation Framework, allowing for your BizUnit tests to be aware of what the orchestration are currently doing.  This allows you to take testing in a whole new direction and tear down the black box walls.

Please let me know if you want to discuss any of these topics in further detail, I will definitely aim to post more on testing tools in the near future.