Integrating BizTalk Server with IBM AS/400 data queues – DPC vs. DB2 stored procedures

I have recently worked on architecting an integration solution involving BizTalk reading XML off an AS/400 data queue. Now this was my first foray into the world of IBM and I found the learning curve to be rather brutal considering I had to learn the nuances of a new operating system in addition to uncharted programming models using BizTalk Server and HIS (Host Integration Server).  In this blog post I will discuss the options available to implement such a solution and factors to consider when choosing which option is the right fit (like most integration problems, the answer to this question is “it depends…”).  I won’t discuss the actual technical implementation which I will save for another day, and I will aim for simplicity rather than absolute technical accuracy/thoroughness given the overwhelming amount of information to take in and my relative newness to the IBM experience.  That said, if any AS/400 or HIS gurus think I have oversimplified any concepts too much (or have misrepresented them altogether) then feel free to comment on the post.

Data queues are a native lightweight queuing technology built into the OS/400 operating system and provide an efficient means of asynchronous or synchronous messaging between applications in an AS/400 environment (see this post for some advantages of using data queues).  Data queues aren’t to be confused with Websphere MQ which is another more robust and fully featured queuing technology that requires additional licensing and isn’t built into the operating system.  CL (Control Language which is the native scripting language on AS/400s) commands are provided out of the box that allow for interaction with  data queues and can be used in COBOL/RPG programs or DB2 stored procedures.

HIS is the Microsoft product which enables integration of windows applications with midrange and mainframe environments.  HIS licenses are combined with BizTalk Server licenses so is the obvious choice of tooling for the aforementioned problem given that a BizTalk Server license was already owned by the client.

There isn’t currently any means of directly integrating BizTalk with OS/400 data queues using any of the native BizTalk adapters or those provided by HIS without some form of a wrapper in the AS/400 environment.  As previously mentioned the CL commands used to interact with the data queues can be utilized in COBOL/RPG programs or DB2 stored procedures, both which can serve as wrappers which BizTalk can interact with using different programming models provided by HIS.

The first option of using a COBOL/RPG program is a more programmatic approach.  The BizTalk Adapter for Host Applications (BAHA) adapter (which is packaged with HIS) can be used to call on the program and return the response using an underlying API technology called DPC (Distributed Program Calls) that allows for non AS/400 applications to interact with an AS/400 environment over TCP/IP.

DPC/HIS architecture

From a BizTalk developers perspective this involves creating HIS metadata in a TI (Transaction Integrator) project in Visual Studio based on the COBOL/RPG program (this can either be done manually or by pointing a wizard at the program saved in a text file) which will generate a DLL that you must have the adapter reference, as well as XSD schemas which you can use in your BizTalk project to represent the request and response signature of the program in question.

TI Project

There are a few considerations when using the BAHA adapter in tandem with DPC.

• DPC only supports client/windows initiated processing which is to say that you can only use the above pattern in a send port.  If you want to implement this pattern in a polling fashion then you will need to use the scheduled task adapter or a similar solution.
• DPC supports a limited number of input and output parameters (35 in total in any combination) and does not support result sets.
• DPC enforces a size limit on your request/response messages in that they can’t be larger than 32kb.
• DPC requires that the OS/400 version is at least release 4 version 1 (I have been assured by a clued in colleague that this is a very old version so shouldn’t be a problem at most sites) and that the DPC server daemon has been turned on (which should be the case if there are currently any windows applications that make API calls to the AS/400 environment).
• HIS provides a pretty nice emulator tool that allows you to emulate your COBOL/RPG programs on the AS/400 environment which makes it easy for a developer to generate HIS/BizTalk components and test them without actually having access to the AS/400 environment, they only need a copy of the program’s source code.
• Security can be applied on the COBOL/RPG program to restrict it to specific users.

The second option is to use DB2 stored procedures to wrap/expose the interaction with the data queue and use the DB2 adapter (which is packaged with HIS) to call on the stored procedure.  Now many non-IBM developers might be up in arms regarding this option (as I was initially) thinking that a database stored procedure should not be used to expose application logic.  Once again things are a bit different in the IBM world where the DB2 database is a native (you’ll notice I have used this word a lot today) part of the operating system as are commands to interact with the database, and stored procedures are a very common medium for exposing application logic, and in fact any ILE (Integrated Language Environment) program (CL, COBOL, RPG, C and C++) can be exposed as a stored procedure.

DB2 adapter architecture

From a BizTalk developers perspective the experience of dealing with the DB2 adapter is similar to that when dealing with other databases in that you use the add adapter metadata wizard to generate XSD schemas representing the request and response signatures of your stored procedure and send the appropriate messages to the DB2 adapter.

Stored Procedure Schema Generation

There are fewer technology considerations when using the DB2 adapter to call on stored procedures.

• The DB2 adapter supports receive operations (via polling) as well as send operations thus removing the need for building custom polling mechanisms.
• The DB2 adapter returns its responses in the form of result sets and doesn’t enforce a size limit.
• Security can be applied on the stored procedure to restrict it to specific users.

So what are the appropriate questions to ask when deciding which technology to use to solve such a problem?

• Does the functionality already exist in the AS/400 environment either in the form of a COBOL/RPG program or a DB2 stored procedure?  If so then it might be preferable to go with the status quo.
• If the functionality doesn’t already exist then do the AS/400 developers have a preference or any architectural guidance that they normally apply to such choices? How do currently deployed non-AS/400 applications interact with AS/400 components?  Are there any COBOL/RPG programmers available?
• Does BizTalk need to fetch result sets or just a single value? Is the return message larger than 32kb?
• What version of OS/400 is currently deployed and is the DPC server currently activated?  If not then is the client open to activating it?
• Does the solution require polling which is much more easily implemented with the DB2 adapter?

At this stage I am unable to qualify whether there is any difference in responsiveness between the two adapters and programming models, and have not found any definitive sources that have carried out such a comparison.  If there a difference then this would of course be a major factor in the decision as well (especially if the solution has any requirements around low latency).  I will add a comment to this blog post when I do manage to qualify this and please do feel free to add your own comments if you have any past experience around this.

I will look to supplement this blog post with practical examples in the future. Can anyone think of any additional factors that might influence such decisions or of any other options to implement this solution using BizTalk Server?