Latest Entries »

A lesson I learnt (the hard way) while working on the BRE Pipeline Framework was that if you use one of the out of box disassembler pipeline components such as the XML/EDI/Flat File disassembler and you rely on them to promote context properties from the body of your message, you will find that those context properties are not promoted until the message stream has been read at least once.

What this actually means is that if you build a custom pipeline component which you intend to use in your receive pipeline after a disassembler, and you try to read a context property that you expected to be promoted by the disassembler, you will find that it has a null value if you don’t read the stream.

However, if you perform a read of the inbound stream using a StreamReader (remember to add the reader as a resource to the pipeline context like below so that it won’t get disposed till the pipeline completes processing, to ensure that your stream is seekable and to rewind the stream after reading it) you will find that all your context properties are promoted.

StreamReader reader = new StreamReader(inmsg.BodyPart.Data);
reader.Read();
pc.ResourceTracker.AddResource(reader);
copiedBodyPart.Data.Position = 0;

One other interesting fact is that even if you call the StreamReader.Read() method which should only read one character from the stream, you will find that all your context properties are promoted, and in fact the current position in the stream (before you reset it of course) is the end of the stream!

So why does this behaviour happen? It turns out that when the out of the box disassemblers execute they don’t actually promote the context properties themselves. They actually wrap the inbound stream with a Microsoft.BizTalk.Component.XmlDasmStreamWrapper wrapper stream, which is responsible for promoting properties amongst other tasks. The property promotion does not happen until the stream has been read. This also hold true for other context processing type functions that are performed by disassemblers, as detailed so well by Charles Young in this blog post about BizTalk Server 2004 (from which I’ve borrowed part of this blog’s title).

This behaviour is in line with best practice pipeline component development guidance, which suggests that pipeline components should be built in an efficient stateless manner in which content processing is only executed once when the stream is read as the message is committed to the BizTalk message box (rather than within the pipeline components themselves) with the use of wrapper streams. This model suggests that all content processing should be carried out either when a pipeline completes execution or in orchestrations.

This guidance works well for solutions which include orchestrations which provide workflow functionality, but obviously doesn’t hold water in messaging only solutions (of course this could spurn a debate as to whether a solution that mandates content processing must contain orchestration, however that is not a topic for today) in which you might want to perform some evaluation of the message content (potentially via context properties) in a pipeline. In this case you might have no choice but to read the stream prior to attempting your own processing if you depend on having available the outputs of a preceding disassembler.

You will also find if you are creating a new message in your pipeline and copying over the content and context from the original message to your new message, that if you don’t read the stream and attempt to perform a shallow copy of the context from the original message to the target message, that your target message will lose any context properties that the disassembler was meant to promote.  You should instead copy the context over by reference.

So how this affect the BRE Pipeline Framework?  Prior to v1.5 I was naively making a copy of the original stream from the source message to the target output message, which of course means that the stream was being read.  This is certainly not the most efficient way to create the output message, especially if I don’t intend to manipulate the message body in any way since passing the original stream by reference should be good enough.  I decided that this was the path I would go down with the BRE Pipeline Framework v1.5.  However, of course, without reading the stream the context properties would no longer be available for evaluation in rules that are executed by the BRE Pipeline Framework, as was the case with prior versions of the framework.

To get around this issue the pipeline component included in the BRE Pipeline Framework has a new parameter called StreamsToReadBeforeExecution, which is a comma separated list of stream types that should be read prior to calling any BRE Policies, and it is pre-populated with the value Microsoft.BizTalk.Component.XmlDasmStreamWrapper.  If you are building a solution based on the BRE Pipeline Framework and do not need access to context properties that are promoted by disassemblers then I would urge you to remove the value from this parameter on the pipeline component so that your pipeline component behaves in a streaming manner.  Rest assured that regardless whether the parameter is populated or not, the promoted context properties will be on the output message once the pipeline has completed processing.

You will find that if you run a trace using the CAT Instrumentation Framework Controller, specifically for pipeline component tracing, that the stream type intercepted by the BRE Pipeline Framework component will be displayed and if the stream is being read that will be displayed as well as below.

[1]135C.09E0::07/10/2014-22:10:29.877 [Event]:9d59b3fb-5a39-43a3-8b90-7d33a5b2ec17 - Inbound message body had a stream type of Microsoft.BizTalk.Component.XmlDasmStreamWrapper
[1]135C.09E0::07/10/2014-22:10:29.877 [Event]:9d59b3fb-5a39-43a3-8b90-7d33a5b2ec17 - Inbound message body stream was not seekable so wrapping it with a ReadOnlySeekableStream
[2]135C.09E0::07/10/2014-22:10:29.890 [Event]:9d59b3fb-5a39-43a3-8b90-7d33a5b2ec17 - Reading stream to ensure it's read logic get's executed prior to pipeline component execution

The takeaway from this blog post is that you must not assume that context properties will be available for evaluation in your pipeline components following a disassembler unless you read the stream first (which might or might not be acceptable, based on your specific requirements).

I have just uploaded the BRE Pipeline Framework v1.5.1 installer to the CodePlex project page. If you have previously downloaded v1.5 then please uninstall it, download v1.5.1 and install that as it fixes a pretty major bug.

The bug (issue #1767) results in context properties promoted by XML/FF/EDI disassemblers prior to BRE Pipeline Framework components not being available for evaluation in execution policies.

I had actually found this bug during development on v1.5, fixed it, created unit tests, and then broke the code. Unfortunately due to a specific combination of rules in the test policy I was using I was getting a false positive in my tests. Be assured that I have updated my unit tests so that this bug is specifically tested for now.

The cause of this bug warrants an entire new blog post which I will write up in the next few days, and will highlight the difficulties in accessing context properties promoted by out of the box disassembler components in further stages of a pipeline.

While implementing dynamic transformation in the BRE Pipeline Framework I ran into an interesting problem.  In BizTalk 2013 Microsoft changed the way transformations are executed to be based on XSLCompiledTransform, rather than on the long deprecated XSLTransform, which delivers performance benefits in the mapping engine.  This however is a breaking change for all those that chose to implement dynamic transformation via custom .Net code in prior versions of BizTalk.  My specific problem was that I wanted to implement dynamic transformation in the BRE Pipeline Framework without forking the code to provide BizTalk 2010 and 2013+ support.

The code for BizTalk 2010 dynamic transformations in the BRE Pipeline Framework looks (note that it has been truncated to make it easier to view, visit the CodePlex page if you’d like to see the full source code) like the below.

TransformMetaData transformMetaData = TransformMetaData.For(mapType);
SchemaMetadata sourceSchemaMetadata = transformMetaData.SourceSchemas[0];
string schemaName = sourceSchemaMetadata.SchemaName;
SchemaMetadata targetSchemaMetadata = transformMetaData.TargetSchemas[0];

XPathDocument input = new XPathDocument(inmsg.BodyPart.GetOriginalDataStream());
XslTransform transform = transformMetaData.Transform;
Stream output = new VirtualStream();
transform.Transform(input, transformMetaData.ArgumentList, output, new XmlUrlResolver());
output.Position = 0;
inmsg.BodyPart.Data = output;

The above wouldn’t build on a BizTalk 2013 development machine since an ITransform object was expected instead of an XSLTransform object.  The working code looks like the below.

TransformMetaData transformMetaData = TransformMetaData.For(mapType);
SchemaMetadata sourceSchemaMetadata = transformMetaData.SourceSchemas[0];
string schemaName = sourceSchemaMetadata.SchemaName;
SchemaMetadata targetSchemaMetadata = transformMetaData.TargetSchemas[0];

XPathDocument input = new XPathDocument(inmsg.BodyPart.GetOriginalDataStream());
ITransform transform = transformMetaData.Transform;
Stream output = new VirtualStream();
transform.Transform(input, transformMetaData.ArgumentList, output, new XmlUrlResolver());
output.Position = 0;
inmsg.BodyPart.Data = output;

Note that the major point of difference in the above two code snippets is the type of the transform variable.  In order to cater for both scenarios I decided to take advantage of .Net 4’s dynamic type feature whereby instead of specifying a class name (XSLTransform or ITransform) I use the dynamic keyword instead as below.

dynamic transformMetaData = TransformMetaData.For(mapType);
SchemaMetadata sourceSchemaMetadata = transformMetaData.SourceSchemas[0];
string schemaName = sourceSchemaMetadata.SchemaName;
SchemaMetadata targetSchemaMetadata = transformMetaData.TargetSchemas[0];

XPathDocument input = new XPathDocument(inmsg.BodyPart.GetOriginalDataStream());
dynamic transform = transformMetaData.Transform;
Stream output = new VirtualStream();
transform.Transform(input, transformMetaData.ArgumentList, output, new XmlUrlResolver());
output.Position = 0;
inmsg.BodyPart.Data = output;

Note that in the above I also had to use the dynamic keyword in place of the TransformMetaData type since this class appears to belong to a different namespace in BizTalk 2013 compared to prior versions.

The dynamic keyword instructs the compiler to not perform any validation on methods/properties called on that object (so no intellisense) and to instead assume that the developer knows what he is doing.  The object type is resolved at runtime and if any of the called methods/properties don’t exist then that will result in a runtime error.

This of course is only a valid solution if you are targeting .Net 4.0 at the minimum since this feature didn’t exist in previous versions.  This solution works well for solutions targeting BizTalk 2010 and above.  I would also encourage any BizTalk 2010 shops that are dabbling in dynamic transformation to future proof their solutions by using the dynamic keyword.

This of course only scratches the surface of dynamic types.  If you want to read more check out this MSDN article.  I would definitely encourage thorough unit testing (as was the case for the BRE Pipeline Framework) to make up for the loss of compile-time validation.

A few days ago I released v1.5 of the BRE Pipeline Framework on CodePlex.  One of the many new features available in the framework is ETW (Event Tracing for Windows) tracing based on the best practice guidance provided by the CAT team.  According to benchmark tests performed by the CAT team, ETW based tracing outperformed other methods of tracing such as System.Diagnostics, Log4Net, and Enterprise Library 5 based tracing by a factor of 24-580 times (see below results when performing the same trace statement 100,000 times across the different frameworks).

Benchmarks

The BRE Pipeline Framework now has ETW based tracing baked into the framework itself so you can see why the framework executed in the manner it did, and if any errors were encountered you can also see the entire stack trace for the given error.  This tracing information can even be gathered in production environments and should have a near negligible effect on the performance of your solution as long as you are making use of asynchronous trace listeners rather than synchronous ones such as DebugView.  My preferred method of capturing trace information is to use the CAT Instrumentation Framework Controller application (if you use this then you will need to ensure that you toggle on the pipeline component option before you turn on the trace) and to trace to a log file.

An example of trace information provided by the BRE Pipeline Framework is below (it is the entire trace output for one execution instance of the pipeline component that uses XML based facts to manipulate a message and to then apply a BizTalk map against the message).  Note that every trace statement for a given instance of a pipeline component will be accompanied by a GUID value for correlation purposes, so when you are tracing in a multi-threaded environment and you have many different threads processing messages with the BRE Pipeline Framework you should be able to group trace statements together based on the GUID (Excel’s text to columns feature does wonders here).  Also note that there is a timer provided in the “End” entry which measures the time between the “Start” and “End” entry.

[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:TRACEIN: BREPipelineFramework.PipelineComponents.BREPipelineFrameworkComponent.TraceIn() => [4f8f4bd2-e1a1-4967-8f32-1952c156d527]
[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:START -> 4f8f4bd2-e1a1-4967-8f32-1952c156d527
[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - BRE Pipeline Framework pipeline component has started executing with an application context of Test_XmlFactsApplicationStage, an Instruction Execution Order of RulesExecution and an XML Facts Application Stage of BeforeInstructionExecution.
[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - BRE Pipeline Framework pipeline component has an optional Instruction Loader policy paramater value set to BREPipelineFramework InstructionLoaderPolicy.
[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - BRE Pipeline Framework pipeline component has an optional Execution policy paramater value set to BREPipelineFramework ExecutionPolicy.
[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Inbound message body had a stream type of System.IO.FileStream
[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Inbound message body stream was seekable so no need to wrap it
[2]1370.19B8::06/29/2014-15:49:21.801 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - No need to read stream as stream type does not match entries in StreamsToReadBeforeExecution parameter
[2]1370.19B8::06/29/2014-15:49:21.826 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Executing Policy BREPipelineFramework InstructionLoaderPolicy 1.1
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding TypedXMLDocument of type BREPipelineFramework.TestProject.Message to Execution Policy facts.
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.CachingMetaInstructions to Execution Policy facts.
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.ContextMetaInstructions to Execution Policy facts.
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.HelperMetaInstructions to Execution Policy facts.
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.MessagePartMetaInstructions to Execution Policy facts.
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.TypedXMLDocumentMetaInstructions to Execution Policy facts.
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding MetaInstruction BREPipelineFramework.SampleInstructions.MetaInstructions.XMLTranslatorMetaInstructions to Execution Policy facts.
[2]1370.19B8::06/29/2014-15:49:21.829 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Executing Policy BREPipelineFramework ExecutionPolicy 1.1
[2]1370.19B8::06/29/2014-15:49:21.834 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding Instruction BREPipelineFramework.SampleInstructions.Instructions.TransformationInstruction to the Instruction collection with a key of 0.
[2]1370.19B8::06/29/2014-15:49:21.836 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding Instruction BREPipelineFramework.SampleInstructions.Instructions.ApplyTypedXMLDocumentInstruction to the Instruction collection with a key of 1.
[2]1370.19B8::06/29/2014-15:49:21.837 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Adding Instruction BREPipelineFramework.SampeInstructions.Instructions.MessageModificationInstructions to the Instruction collection with a key of 2.
[2]1370.19B8::06/29/2014-15:49:21.840 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Applying typed XML document (overwriting current message body)
[2]1370.19B8::06/29/2014-15:49:21.840 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Starting to execute all MetaInstructions.
[2]1370.19B8::06/29/2014-15:49:21.840 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Executing instruction BREPipelineFramework.SampleInstructions.Instructions.TransformationInstruction.
[2]1370.19B8::06/29/2014-15:49:21.840 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Applying transformation BREPipelineFramework.TestProject.Message1_To_Message2, BREPipelineFramework.TestProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=83eab0b166470ebc to the message
[2]1370.19B8::06/29/2014-15:49:21.840 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Message is being transformed from message type http://BREPipelineFramework#Message to message type http://BREPipelineFramework.TestProject.Message2#Message2
[2]1370.19B8::06/29/2014-15:49:21.840 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Executing instruction BREPipelineFramework.SampleInstructions.Instructions.ApplyTypedXMLDocumentInstruction.
[2]1370.19B8::06/29/2014-15:49:21.840 [Event]:4f8f4bd2-e1a1-4967-8f32-1952c156d527 - Executing instruction BREPipelineFramework.SampeInstructions.Instructions.MessageModificationInstructions.
[2]1370.19B8::06/29/2014-15:49:21.841 [Event]:END <- 4f8f4bd2-e1a1-4967-8f32-1952c156d527: 39ms
[2]1370.19B8::06/29/2014-15:49:21.841 [Event]:TRACEOUT: BREPipelineFramework.PipelineComponents.BREPipelineFrameworkComponent.Execute(...) = "4f8f4bd2-e1a1-4967-8f32-1952c156d527"

As you can see there is a lot of information provided. You can see all the configuration parameter values that were applied to the pipeline instance, you can see what type of stream the message was (I will discuss this in more detail in future blog posts since the type of stream does affect the behavior of the pipeline component), what BRE Policy including version number is being executed, when the TypedXMLDocument (based on XML facts) is applied to the message body stream, the type of transformation being applied including the source and destination message type, etc…. Phew!

Also of note is that if an exception was encountered during execution of the pipeline component then you would see the entire stack trace in the trace results, which can be really handy when diagnosing problems and when reporting it back to the CodePlex project team.

Also of equal importance is the fact that you can now execute your own custom trace statements from within your ExecutionPolicy BRE Policy by using the TraceInfo vocabulary definition in the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary as per the below example.

Trace rule

The TraceInfo vocabulary definition can be used in rule conditions or actions (which would only be triggered if the conditions for that given rule all evaluated), and will accept any string value so you could potentially plug in the StringFormat or ConcatenateString type vocabulary definitions as a parameter into it if you want you trace out complex information.  You could even trace out results of XPATH statements (not recommended for high throughput environments or in applications which deal with large messages, in which case reading the stream is a big no-no) or the values of context properties if you wanted to.  The trace result will be accompanied by the same GUID as the rest of the trace information, and will also be of the pipeline component trace type, so you can easily correlate your custom trace statements to the out of the box trace statements for a given pipeline component execution instance.  An example trace result based on the above rule is below.

Trace result

Equally important to the tracing results is the debug tracing information for BRE Policy execution.  It is quite common for people to get put off from using the BRE because it is perceived to be difficult to debug when the rules execution does not provide the expected result.  The BRE Pipeline Framework solves this problem by providing BRE Policy tracing information when the pipeline component is configured appropriately.  The pipeline component has a parameter called TrackingFolder which you can set in Visual Studio for the pipeline or in the BizTalk Administration Console for a  given pipeline instance.

PipelineComponentConfiguraiton

By default the aforementioned parameter has a blank value, but if a valid folder location is provided then the framework will write out rules tracing information to the given directory.  What you will observe is that one file is written per BRE Policy that is executed, thus if you have specified an InstructionLoaderPolicy and an ExecutionPolicy BRE Policy then you will have two rules tracing files written per execution of the pipeline component.  The information contained within said files is the same info you would see if you were trying to test the policy from within the BRE composer (though in order to test a BRE Pipeline Framework policy you would need to write .NET fact creators so this is an easier way to get the tracing information).

The rules tracing file name will contain the name of the policy that was executed including the version number of the policy.  To make it easier to correlate rules tracing files to an execution instance of a pipeline component, the file name also contains the same GUID that is used in ETW tracing in the pipeline component.

TrackingFolder

Have a look at a snippet of the rules tracing information below.  You’ll notice that you can see the values used and results of each individual condition evaluation across all the rules in your policy which had corresponding facts asserted, and you can also see exactly which rules fired as well (i.e. the actions were triggered).  Not pictured but also conveyed are any facts that weren’t recognized.

CONDITION EVALUATION TEST (MATCH) 29/06/2014 5:03:52 p.m.
Rule Engine Instance Identifier: 4bb67d3e-0c76-49bb-945e-93cfbc3471e4
Ruleset Name: BREPipelineFramework ExecutionPolicy
Test Expression: True == BREPipelineFramework.SampleInstructions.MetaInstructions.HelperMetaInstructions.CheckIfStringExistsInMessage
Left Operand Value: True
Right Operand Value: True
Test Result: True

FACT ACTIVITY 29/06/2014 5:03:52 p.m.
Rule Engine Instance Identifier: 4bb67d3e-0c76-49bb-945e-93cfbc3471e4
Ruleset Name: BREPipelineFramework ExecutionPolicy
Operation: Assert
Object Type: BREPipelineFramework.SampleInstructions.MetaInstructions.MessagePartMetaInstructions
Object Instance Identifier: 38466885

FACT ACTIVITY 29/06/2014 5:03:52 p.m.
Rule Engine Instance Identifier: 4bb67d3e-0c76-49bb-945e-93cfbc3471e4
Ruleset Name: BREPipelineFramework ExecutionPolicy
Operation: Assert
Object Type: BREPipelineFramework.SampleInstructions.MetaInstructions.XMLTranslatorMetaInstructions
Object Instance Identifier: 10993679

RULE FIRED 29/06/2014 5:03:52 p.m.
Rule Engine Instance Identifier: 4bb67d3e-0c76-49bb-945e-93cfbc3471e4
Ruleset Name: BREPipelineFramework ExecutionPolicy
Rule Name: Test_TransformMessageVaildateSourceIfKnown
Conflict Resolution Criteria: 0

While the ETW tracing is production friendly as long as you are tracing to an asynchronous destination rather than a synchronous one like DebugView (of course, it’s fine to use this on a developer workstation), I can’t provide any assurances about the production friendliness of the rules tracing information.  In order to implement the rules tracing I have used the out of the box DebugTrackingInterceptor class provided by Microsoft, so whatever guidance applies to that will apply here too, I haven’t seen any guidance personally.  My recommendation is to only use it in production when necessary and to switch it back off when not required; the rules tracing is primarily intended to aid developers to get their rules right during development cycles.

I hope this blog post conveys to you how you can take advantage of instrumentation features in the BRE Pipeline Framework, and gives you a sense of security that this isn’t a framework that will leave you in the dark.  Look forward to many more blog posts within the next few weeks discussing new features in the BRE Pipeline Framework.

I have just released a v1.5 of the BRE Pipeline Framework to the CodePlex project page.  This is a feature rich and heavily optimized version of the framework with much richer traceability as well.  A breakdown of new features and improvements by category is below.

  • Qualitative features
    • Now supports BizTalk Server 2010, 2013, and 2013 R2 (note for 2013 and 2013 R2 if you want to make use of SSO features you will need to add an assembly binding redirect for Microsoft.BizTalk.Interop.SSOClient from v5.0.1.0 to v7.0.2300.0 or v9.0.1000.0 respectively in your BizTalk config files).
    • This version of the framework has been heavily unit tested with 204 unit tests at the time of v1.5 being released which provide 95.16% code coverage.  The framework is now more reliable than ever.
    • The order of instructions specified in the BRE is now respected.  Previously they were only respected within a given MetaInstruction/Vocabulary, but now they will also be respected across MetaInstructions/Vocabularies.
    • Pipeline component supports streaming behaviour and doesn’t read the stream unless it absolutely has to.  Even when it has to (for example the regex find/replace instruction in the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary) the framework has now been optimized to read in smaller chunks of the stream at a time rather than the entire stream in order to minimize memory usage.
    • Pipeline component now provides ETW tracing which can be captured using the CAT Instrumentation Framework Controller.  This provides a lot of detail as to what the pipeline component is actually doing, and can give you access to stack traces in case you want to deep dive into error details.  Combine this with rules tracing information that was catered for in v1.4 and you have all the instrumentation you need.
    • Exception handling in the framework has been vastly improved.  If an uncaught exception occurs during BRE policy execution you will no longer get the vague “An error has occurred while executing a BRE Policy” type error but now will get the actual exception details.
    • You can specify a version of an InstructionLoaderPolicy or ExecutionPolicy that will fire if you want to be explicit (if not then the highest deployed version will fire) and you can also override these versions in the InstructionLoaderPolicy.
    • You can now choose when you want the XML based facts to apply; before instructions execute (new default behaviour), after they execute (previous default behaviour), or at a specific point during instruction execution.  This means you can now string XML based facts together with other instructions giving you more power to inspect and manipulate messages.
    • In previous versions of the framework each of the individual vocabulary versions were exported to XML files and are saved in “C:\Program Files (x86)\BRE Pipeline Framework\Vocabularies” when you run the installer.  Due to the number of vocabulary versions, it was starting to get very painful importing all of these to the rules engine database.  To make life easier I have created two new export files, the first called BREPipelineFramework.AllVocabs.xml which contains all the vocabulary versions in one file so you only have to import it once, and a second called BREPipelineFramework.LatestVocabs.xml which only contains the latest versions of each vocabulary in case you haven’t used previous versions of the framework and thus don’t need to install old vocabulary versions for backwards compatibility.
  • Major new functionality
    • Ability to transform messages using the TransformMessage vocabulary definition in the BREPipelineInstructions.SampleInstructions.HelperInstructions vocabulary.  You can even chain multiple maps one after the other in your pipeline, and in the case of a receive pipeline you can still execute an inbound map configured on the port after the pipeline has completed execution.
    • The BREPipelineInstructions.SampleInstructions.HelperInstructions now contains a TraceInfo definition that allows to you to write out ETW trace statements that can be captured using the CAT Instrumentation Framework Controller.
    • A new vocabulary called BREPipelineFramework.SampleInstructions.CachingInstructions that allows you to cache custom strings, or context properties associated with a message, and to either fetch them back later, or in the case of context properties reapply them against the message.  The cache supports expiry times against each cached item so you don’t have to worry about using up all your memory, and you can also remove items from the cache after you have consumed them.  More on this in future blog posts and this makes for some fantastic message only use cases which might previously have required orchestration.
    • The BREPipelineFramework.SampleInstructions.ContextInstructions vocabulary now contains definitions that allow you to get or set ESB context properties by enumeration (all other out of the box context properties were already catered for), and also allow you to promote the BTS.MessageType context property against the message if it doesn’t already exist.
    • A new vocabulary called BREPipelineFramework.SampleInstructions.PartInstructions that allows you to get or set message part details such as part names by index, content type, character sets, message part context properties such as MIME.FileName etc…
    • A new vocabulary called BREPipelineFramework.SampleInstructions.XMLInstructions that allows you to manipulate your XML based fact messages in ways that the BRE doesn’t support out of the box.  These functions include adding elements and optionally populating values within them, adding attributes with optional values etc…
    • A new vocabulary called BREPipelineFramework.SampleInstructions.XMLTranslatorInstructions that allows you manipulate your XML message in a pure streaming fashion (i.e. the changes will not get applied to the message until the stream is actually read for the first time, which will usually be when the message reaches the BizTalk Message Box) which is very efficient and easy on memory.  Manipulation functions include adding, removing, replacing namespaces and prefixes for elements and attributes, updating element or attribute names, and update element or attribute values.  All of these manipulations can be conditional; for example you can create an instruction that will only update the value of an element if the name and namespace of the element match certain criteria and if the old value in the element matches certain criteria.

This scratches the surface of improvements made to the framework, look out for more blog posts (they have been rare in the last few months which I have dedicated to updating the BRE Pipeline Framework but should become more regular now) which explore new features in further depth.

Follow

Get every new post delivered to your Inbox.

Join 72 other followers

%d bloggers like this: