I was recently asked to help out at a site which was having a problem with one of their production deployments.  Every time they tried to delete their application before importing the new version (which is standard practice at this site) they were getting an error message saying “The service action could not be performed because the service is not registered” as below (note that the error message is from my re-creation of the problem on my own machine).

ServiceNotRegistered

What immediately struck me about this error message was that the error says that “The Message Agent failed to register orchestration” and then lists an assembly name instead of an orchestration name.  Something is definitely majorly wrong in this environment.

The application was running without any problems at runtime prior to this release, but the client mentioned that they had attempted the same release a few weeks ago, had gotten to the stage where they deleted the application and were attempting to import the new version however hit a snag and decided to rollback the BizTalk databases.  Post the rollback they started the application back up and it performed fine since then until they attempted to do the deployment again at which point they started getting the service is not registered error message.

Once we gained an outage window and performed all the required backups, I stopped the application and started playing around a bit.  One of the first things I tried to do was to delete the individual assembly that was mentioned in the error message seen during the deployment and was quickly greeted by the same error message.  Attempting to move the orchestration or any of the send and receive ports (even if they made use of no maps, and only used out of the box pipelines, and weren’t bound to any orchestrations) also raised similar errors as did trying to import an MSI which overwrites the existing assemblies.  Attempting to delete a send or receive port (after unbinding it from any bound orchestrations) raised a different variant of the error message as below.

ServiceNotRegisteredDelete

Attempting to rename the application resulted in yet another error message, this time saying “The application action could not be performed because the application is not registered” as below.

ApplicationNotRegistered

It was fairly obvious at this stage that something was not properly registered in the BizTalk databases.  A quick look around the bts_assembly, bts_application, bts_orchestration, bts_receiveport, and bts_sendport tables in the BizTalkMgmtDb showed that the BizTalk assemblies, application, orchestration, receive and send ports were all known to the management database.

I discussed the rollback procedure that was followed a few weeks ago a bit further with the on-site technicians and it turned out that while all the databases were being backed up on a regular schedule, only the BizTalkMgmtDb and SSODB were actually restored.  Suspicions quickly started turning towards the BizTalkMsgBoxDb being out of sync with the BizTalkMgmtDb.  It made a lot of sense because the application in question had been deleted and then the BizTalkMgmtDb and SSODB had been rolled back, leaving the BizTalkMsgBoxDb with no knowledge of the deleted application.

It took a while to isolate the tables which were out of sync but we did finally get there.  The Modules table in the BizTalkMsgBoxDb table contains a column of all the application names and our application in question was missing from the list.  The Services table in the BizTalkMsgBoxDb contains a column called uidServiceID (which contains GUIDs) which corresponds to the uidGUID column on the bts_orchestration, bts_receiveport, and bts_sendport tables in the BizTalkMgmtDb.  Each row in the services table also holds a nModuleID which corresponds to the nModuleID on the Modules table also in the BizTalkMsgBoxDb.

Armed with this knowledge we had to decide on one of the below courses of action.

  • Attempt to rollback the BizTalkMsgBoxDb back to the same time at which the backups used to roll back the BizTalkMgmtDb and SSODB were made.
  • Attempt to rollback the BizTalkMsgboxDb, BizTalkMgmtDb, and SSODB to the same time at which the backups used to roll back the BizTalkMgmtDb and SSODB were made.
  • Manually fix up the database using SQL scripts.

A judgement call was made to go down the route of manually fixing the database, while acknowledging the risk that there might still be some other discrepancies which we hadn’t accounted for which would have to be reconciled later.

The first thing we did was to add the missing application back into the modules table with a SQL Insert statement similar to the below.

InsertIntoModules

A quick test now shows that we are able to rename the application MsgBoxSyncTest but we still can’t delete it.

Next up we had to collate a list of all the uidGUID values for all the relevant orchestrations, receive ports and send ports from the bts_orchestration, bts_receiveport, and bts_sendport tables respectively in the BizTalkMgmtDb.  We also made note of the nModuleID which was assigned to the record in the Modules table which we had just created.  We then ran a SQL Insert statement similar to the below to add those missing uidServiceID values into the Services table.

InsertIntoServices

After this was done we could once again delete individual artifacts, move them to different applications, or even delete the entire application itself.  Deleting the application at this stage was definitely desirable as recreating it from scratch should be a near guarantee that the BizTalk databases should be in sync after this (at least as far as this application is concerned).

While this problem was observed in a BizTalk Server 2006 environment, I have managed to successfully recreate it in a BizTalk Server 2010 environment as well.

When mentioning this to some of my colleagues I was provided with a few more suggestions.  One colleague suggested that we should restore a backup of the BizTalkMsgBoxDb to a different database name and then run a comparison tool against the two databases to search for any more discrepancies, which is a very good suggestion.  Another colleague suggested that the MsgBoxViewer tool might have been able to pinpoint and resolve the issue.  I did attempt to use the tool when I recreated the problem on my machine however did not find any results that really helped me solve the problem, however I would definitely recommend that the MsgBoxViewer tool is run against a BizTalk environment that has recovered from serious problems such as this to search for any more hidden issues.

While trying to search for solutions for this problem it did appear that there are an unfortunate bunch of people who have encountered similar issues before but I don’t see anyone who has posted a resolution, so I do hope that this helps others who encounter this problem.