Following to my previous post about the proposed alternatives in creating XML message instances, I would like to finish the topic by giving you an overview on the solution that uses some internal BizTalk APIs for generating new messages and populating them with some initial data.

Depending on the particular requirements, a blank "XML skeleton" that contains no user data may hardly be called as usable message instance. Would it be nice to be able to initialise the new XML document with some initial values that we could either define inside the XSD schema or supply programmatically. Or would it be much better if we can use a set of XPath expressions in order to locate the nodes and set their values? Could we do it without loading the whole message into XML DOM? Well, let's see if some undocumented BizTalk APIs could assist with that...

As I described before, a message instance generated by the CreateXmlInstance method will already contain all elements and attributes in accordance with the corresponding XSD schema. There will be a placeholder for each element and attribute so if the schema has been declared with promoted properties or distinguished fields, you can start populating the message with the data inside a Message Assignment shape. The property assignment operations would be considered as an easiest mechanism to enrich your messages with the initial information, however from time to time it may not be possible to use a distinguished field (or a promoted property where appropriate) for value assignment purposes. For instance, there could be a repetitive node declared in the schema for which you will not be able to add a distinguished field declaration but you may still want to create a single instance of this repetitive element and configure some of its attributes.

In some (rare) scenarios, you might have been using the default values that you defined in the XSD schema. Luckily, the CreateXmlInstance method does a good job behind the scenes in order to recognise the XSD's default property on xs:element or the fixed property on xs:attribute and use the specified values when generating a new message instance. That is another simplest way to supply the initial values but it obviously limits you to a set of static data hardcoded in the schema so I wouldn't consider this approach as truly useful. However, it is always good to know about all available options to have a choice when a need arises.

The next option is rather much more powerful than any of those which have been covered so far. It is based on the assumption that you can construct a collection of valid XPath expressions pointing to various XML elements or attributes in the message instance. I will demonstrate a simple but really nice technique in managing XPath queries in the BizTalk projects, but this would certainly be a reason for another post. In the meantime, it is envisaged that a set of XPath queries will be supplied at run-time along with the initial values for the referenced XML nodes. The true magic is hidden inside the XPathMutatorStream class from the Microsoft.BizTalk.Streaming namespace. It wraps the XML data stream or XmlReader into a buffered stream the content of which "mutates" (in other words, gets translated into a possibly different XML stream) while it is being read by the consumer. The class implements the basic XPath streaming to apply the specified XPath queries against the XML infoset while the data is being passed through the underlying XML reader. In the event of finding a match, the class will invoke a callback method that is responsible for returning a final value of the XML entity (attribute or element) matching the XPath expression. The delegate signature for the callback method is the following:

/// <summary>
/// Defines a callback method that will be invoked by the XPathMutatorStream class when a match against any of the specified
/// XPath expressions has been found in the underlying XML stream.
/// </summary>
/// <param name="matchIdx">The index of the matched XPath expression in the collection supplied when initializing a XPathMutatorStream.</param>
/// <param name="matchExpr">The instance of the XPathExpression class containing the XPath query for which a match has been found.</param>
/// <param name="origVal">The original content (or value) of the matched XML element or attribute.</param>
/// <param name="finalVal">The new content (or value) that will be assigned to the matched XML element or attribute.</param>
public delegate void ValueMutator(int matchIdx, XPathExpression matchExpr, string origVal, ref string finalVal);

In order to help the XML stream to "mutate on fly" accordingly to our expectations, we need to construct a collection of XPath-to-Value mappings that is in fact a simple string dictionary where the key represents an XPath query and the value is the data that will be assigned to the matched XML element or attribute. This would also serve the purpose of defining the value assignment rules:

IDictionary<string, string> xpathToValue = new Dictionary<string, string>();
xpathToValue.Add("/*[local-name()='HwsMessage']/HwsSection/TaskID", Guid.NewGuid().ToString());
xpathToValue.Add("/*[local-name()='HwsMessage']/HwsSection/TaskDescription", "New task description goes here");

In the above example, I have used a generic Human Workflow Services task and created a value assignment rule for 2 fields (TaskID and TaskDescription). In essence, the rule is no more than just an XPath pointing to the target XML node and the new value for that node. The next step is to convert the XPathToValue dictionary to the Microsoft.BizTalk.XPath.XPathCollection type. Unfortunately, the only way to implement that is to enumerate through all items in the XPathToValue dictionary. We also need to build an array of values to be returned from the ValueMutator delegate as it generally provides better index-based lookup performance than a hash-based lookup through the dictionary:

// Construct and populate a new XPathCollection by enumerating through all entiries in the xpathToValue dictionary.
XPathCollection xc = new XPathCollection();
IEnumerator<KeyValuePair<string, string>> valueEnumerator = xpathToValue.GetEnumerator();

// An array of new values copied from the xpathToValue dictionary.
string[] values = new string[xpathToValue.Count];
int valueIndex = 0;

// For each iteration, add the key (XPath expression) to the XPathCollection and insert the value to the array.
while (valueEnumerator.MoveNext())
{ 
        xc.Add(valueEnumerator.Current.Key); 
        values[valueIndex++] = valueEnumerator.Current.Value;
}

Now that we have constructed XPathCollection, the final step would be to pass the collection to an instance of the XPathMutatorStream class along with a valid XmlReader that will be used to fetch the source XML data. The support for anonymous methods in C# 2.0 greatly assists with providing an in-line implementation of the required delegate type:

XPathMutatorStream mutator = new XPathMutatorStream(xmlReader, xc, 
        delegate(int matchIdx, XPathExpression expr, string origValue, ref string finalValue) 
        { 
                // matchIdx provides the index of the matched XPath expression in the supplied XPathCollection. 
                if (matchIdx >= 0 && matchIdx < values.Length) 
                { 
                        // The array of target values is laid out in the same order as XPathCollection so we simply use the matchIdx parameter as a index in the  
                        // array to return a new value for the XML node (element or attribute) matching the XPath expression. 
                        finalValue = values[matchIdx]; 
                } 
                else 
                { 
                        throw new ArgumentOutOfRangeException("matchIdx", matchIdx, "The specified parameter is not in the valid range.");
                } 
        });

All we need to do in order to see the updated XML data is just to use the "mutation" stream and load its content into a new XmlDocument which couldn't be easier than the following:

XmlDocument msgInstanceWithData = new XmlDocument();
msgInstanceWithData.Load(mutator);

It is now time to sum up the discussion above and to come to the following conclusions:

• The CreateXmlInstance method allows you to generate a blank instance of the XSD schema effectively enabling you to consistently reuse your BizTalk schemas for message instantiation purposes;
• You can enrich the generated message instances with some initial data either by using the default or fixed declarations in the XSD schema, direct assignment via distinguished fields or streaming-based mutation;
• The distinguished field assignment would generally be considered as the preferred method for initializing the simple type, single-instance XML nodes from inside the BizTalk orchestrations;
• The functionality provided by the XPathMutatorStream class would more suit the custom components or where distinguished field cannot be adequately declared;
• The easiest way to supply the initial values for the XML nodes would be the definition of the XPath-to-Value mappings that combine the XPath queries to locate the target node and the new content for that node.

And finally, you can now download the sample code that I've pulled together for this article to visualise an example of the outlined technique in action.

Thanks & regards,
Valery

If you are not too much worried about using some undocumented (and indeed, unsupported) BizTalk APIs to gain a degree of perfectionism in your BizTalk solution, you would probably be looking to solve the most common problems in rather elegant non-traditional ways. In this post, I would like to demonstrate some advanced message instantiation techniques.

In some cases, there is a small or virtually no practical benefit of using a BizTalk map for creating the new message instances. A typical example may imply one of the following constraints:

• The message fields will have to be populated from various sources and there is no option for having a single input message that combines all required data;
• The underlying algorithm for calculating the values cannot be embedded into a map due to its nature, complexity or external dependencies;
• There is a requirement for a blank message that can be used solely for initialization purposes and having a BizTalk map to generate it would be an overkill;
• You may need to create a new message from scratch outside your orchestration's context (e.g. in a custom component) in a programmatic "transformless" fashion.

On the other hand, a programmatic approach whereby you load a static template into the brand new instance of XmlDocument or utilize the XSD-generated .NET classes along with XML serialization may not also look like a technically sound solution. A typical argument for not being able to fully appreciate the above techniques may conclude that:

• Regardless whether you have an XML template for your message or a .NET class generated against the corresponding XSD schema, you have a statically defined instance that brings additional and sometimes undesired maintenance overhead to your solution;
• You don't want to pay the cost of XML serialization when converting your XSD-typed class to XML;
• You may need to ensure that certain XML elements/attributes are immediately available in the message instance so that the promoted property's value assignment operations will succeed;
• You need to initialize a new message instance with some initial values and these might rather be supplied in a dynamic list so that the code changes are minimised.

With this in mind, you might want to take a look at some internal BizTalk APIs to see how the infrastructure components can help to solve the above challenges. Apparently, the programmatic message instantiation issue could be resolved by using the DocumentSpec class, namely its magical CreateXmlInstance method. In order to make use of this method, you need to create a new DocumentSpec instance by passing the full .NET type name of the BizTalk schema and its containing .NET assembly name to its constructor. To get these values, you need to have a valid reference to the BizTalk project with your schemas. The following code sample gives you an idea of what exactly is involved:

Type msgSchemaType = typeof(CompanyName.FunctionalArea.BizTalk.Schemas.MyBTSSchema);
DocumentSpec docSpec = new DocumentSpec(msgSchemaType.FullName, msgSchemaType.Assembly.FullName);

Then you simply call the CreateXmlInstance method to get a stream that will contain the message instance:

XmlDocument msgInstance = new XmlDocument();
using (StreamReader reader = new StreamReader(docSpec.GetDocSchema().CreateXmlInstance()))
{ 
        msgInstance.Load(reader);
}

Now that you have a new XML message instance that has been dynamically constructed based on the corresponding XSD schema with just 5 lines of code, it is important to keep in mind the following "distinct features" of this XML document:

1. The document structure will incorporate a single node for each declared XSD element or complex type regardless of the occurrence constraints (minOccurs/maxOccurs) applied.
2. The structure will contain all child nodes to ensure its conformance to the defined hierarchy;
3. All XML elements and attributes including the optional ones will be generated with an empty initial value to give a "clean" document with all XML entities already in the DOM;
4. A declaration for each XML namespace used in or imported by the schema will be included accordingly;
5. The content is by no means reflecting a sample message as if it would have been generated from the Visual Studio IDE by selecting the Generate Instance option. There will be no sample data generated for you.

If you got excited about these features, it is worth noting some disadvantages that come along with them:    

• The generated message instances may not be fully compliant with the corresponding XSD schema as CreateXmlInstance method doesn't take into account XSD type restrictions, value presence requirements, node occurrences and other constraints;
• You would need to accept the fact that this approach involves using the undocumented APIs and there is no guarantee that these APIs will remain functional in the way they have been exposed in the current version of BizTalk Server.

A blank message instantiation was not the only issue that we wanted to explore. So far, you won't probably feel a big difference between the above approach and the traditional myDoc.LoadXml("<MyTemplate />") quick and dirty solution. In my next post, I will demonstrate the power of another undocumented class to help you with generating a message instance that is already populated with some initial data. There will also be a code snippet that includes both techniques as a reusable utility class.

Valery

I almost guarantee that you will not find this elsewhere on the web.

Having stayed late in the office, I have been thinking about the easiest way of unit testing a BizTalk orchestration exposed as a Web Service. In particular, I wanted to be able to track all web service requests in their raw format before they even reach the receive pipeline. Furthermore, I needed an ability to capture these requests as easy as possible without writing a single line of code or introducing a custom SOAP extension. Please don’t get me wrong, I’m not a lazy person. I was just looking for a quick solution for my BizUnit tests and it has been very late in the night to think about a bunch of new code.

Accidently, I’ve found a magical ParameterTracingFile configuration property secretly used in the web service class that was generated by the BizTalk Web Services Publishing Wizard. The property needs to be specified in the web.config file with a value that contains the name of the folder where the tracing files will be created. Please don’t be confused with the property name, it does not really imply that you will specify a file name, just provide a folder name instead and don’t forget to include a backslash at the end.

Once the ParameterTracingFile property is added to web.config, each request to the BizTalk orchestration published as a web service will result in generating a text file with an unique name in the folder that you specified. The file will include the parameter information for the web methods being invoked as well as the raw data being received by the web service. As there will be a separate file for each SOAP request, it greatly simplifies the unit testing in general.

Hopefully you will also find this fantastic tracing option useful for your needs.

As things stand at the moment, the standard SQL adapter shipped with Microsoft BizTalk Server 2006 has been released with no support for any of the enhanced capabilities of the Microsoft SQL Server 2005 platform. The primary codebase of the adapter has been inherited from the previous version of BizTalk and although it does provide the required functionality for core integration with SQL Server through using either stored procedures or updategrams, this may not deliver the optimal performance as it would become possible if the adapter was fully optimised for SQL Server 2005.

On top of that, I will mention all those well-known limitations and uncontrollable aspects of the SQL adapter' behavior:

• A lack of support for configurable transaction isolation levels (in a pure messaging scenario);
• A requirement for a separate, often redundant adapter schema and associated, usually inefficient BizTalk map (try to stick an XML document into an attribute in the SQL adapter schema and you will see what I really mean);
• A limited support for batch operations (only with updategrams) that lacks extensibility and granularity;
• No support for user-defined database schemas in SQL 2005 (due to a security check that prohibits the fully qualified stored procedure names, e.g. [myschema].[MyProc]).

I have to admit that SQL adapter is still doing a great job simplifying the receive side of things but even here we would benefit from such value-adding features like a resultset-to-XML auto conversion that could help to avoid building dedicated FOR XML-style queries.

Maybe it is time to write my own adapter with a full understanding and appreciation of the SQL 2005 features. Let me know what you think and whether or not this crazy idea would be of any value to anyone.

Valery

Since BizTalk 2004 Server platform was introduced a few years ago, Enterprise SSO Services were extended to act as a secure configuration store. The SSODB Credential Store database and SSO services can be used to maintain and access custom configuration properties securely. In fact, this is used by BizTalk Server itself to store adapter configuration information. The secure configuration store is primarily designed to allow application developers and system administrators to manage the configuration data centrally and it allows BizTalk applications to access the configuration data during runtime operations in a secure manner. The configuration data is stored encrypted, just like the credentials in the classic single sign-on/account mappings scenarios.

A BizTalk solution that makes use of the Enterprise SSO configuration store provides a wide range of technical benefits that might be previously underestimated or not included into the technical design considerations in the past:

• The configuration data is stored in the encrypted form allowing the application architects to manage any type of application settings in the SSO configuration store including passwords, connection strings and other sensitive information;
• The application settings will be automatically distributed and synchronised across multiple servers in a BizTalk group. This is an extremely important benefit for a large BizTalk deployment;
• The configuration management tasks can be delegated to the system administrators or infrastructure support personnel as the configuration information is isolated from the application code and requires insignificant or no involvement from the developers at all;
• The application properties can be managed centrally without a need to access each BizTalk server in a group remotely in order to modify the application configuration;
• The configuration data can be partitioned into a series of custom SSO affiliate applications to enable better isolation of the configuration settings between multiple BizTalk applications;
• The structure and types of the configuration data can be defined programmatically and that could go beyond classic name/value pairs if required in a sophisticated design (e.g. serialised object graphs or XML fragments);
• The SSO configuration store provides a central secure repository that can be accessed by multiple types of consumers in the Enterprise Application Integration environment including BizTalk orchestrations, pipelines, functoids, .NET components, XSLT extensions and web services in an unified and consistent fashion.

If you are still using BTSNTSvc.exe.config as your primary configuration store, take a look at some really cool implementations available these days and I'm sure you will soon realize the real power of the Enterprise SSO Services.

Take care.

Valery