Introduction:
In BizTalk 2010, When there is a need to develop a new pipeline component, typically a developer creates a new custom pipeline component, creates a corresponding pipeline and deploys it on target server.
Challenges:
In many cases, on boarding a BizTalk application requires new pipeline components and pipelines due to specific processing requirements/functionalities for incoming and outgoing messages needed by the application. Therefore, it warrants the development of new pipeline components and pipelines.
- As the number of custom pipeline components grows, so does the number of pipelines. Typically, the growth is almost linear when requirements come.
1 Functionality = 1 Pipeline Component
- In some cases, a pipeline has already existed with most of the functionalities, an addition of functionality or the need to swap the order of functionalities necessitate the creation of new pipeline to accommodate it.
The existing process flow on how they are developed and deployed as described in the following diagram below:
Disadvantages of the existing pipeline development:
The process has the following drawbacks from development perspective:
- Repeated codes for implementing pipeline component interface methods such as Validation, Load, Save in addition to user-defined properties for design-time pipeline components. On top of that developer still has to provide coding implementation for the requested functionality itself. (Circle 1 in the diagram )
- In many cases, the requested functionalities are simple, however because developer has to implement pipeline component interface methods, the code to implement the interface methods plus the design-time properties produce more code than the functionality implementation itself. (Circle 1 in the diagram )
- Little code-reuse i.e. similar functionalities were duplicated by different pipeline component developers. (Circle 1 in the diagram )
- A new functionality corresponds to a new pipeline component implementation which translates into a new .NET assembly that has to be copied to component folder. As the number of functionalities grows so does the number of pipeline component assemblies. (Circle 1, 2 and 3 in the diagram )
- Modifying the order, adding a new or removing pipeline component(s) from an existing pipeline requires a creation of new pipeline. (Circle 1, 2 and 3 in the diagram)
The process has the following drawbacks from configuration modification and deployment perspective:
- Runtime configuration modification in a pipeline such as adding, removing or changing order of pipeline component(s) is not possible. It is only possible with the creation and deployment of new pipeline. (Circle 1, 2 ,3 and 4 or Circle 3 and 4 in the diagram, depending upon whether the pipeline component(s) already existed or not)
- Modification in pipeline or pipeline components always create deployment of assembly into BizTalk environment (Circle 4 in the diagram)
- Proliferation of the pipelines and pipeline components increase deployment and maintenance cost.
Proposed Solution - Pipeline Consolidation Framework(PCF)
As described above, proliferation is caused by 1:1 connection between functionality and pipeline component and the nature of static linkage between pipeline and pipeline component.
The key is to consolidate it is by replacing the static connection with the dynamic one. Utilizing the dynamic nature of .NET Reflection, a solution based on that feature is proposed.
A new PCF solution is proposed as described in the following figure:
This PCF will have the following features:
- The creation of the framework that leverage the dynamic nature of .NET. The framework will contain implementation of default functionalities that are likely shared by multiple BizTalk applications. These functionalities will cover most of the scenario and provide mechanism to provide custom functionalities specific for particular applications.
Note: In the proposed solution, each functionality doesn’t equate to pipeline component.
In this diagram, the pipeline component provides slots where each slot is equivalent to each functionality
- Providing several standard framework pipelines based on format types that cover all types currently used by ICOE. Each pipeline contains pipeline components that provides the slots to define the functionalities Leveraging Global Assembly Cache for deployment
- The creation of reusable assembly for common functionalities that can be used for other non-BizTalk application
The consolidation framework also alters the process on how pipelines and pipeline components are developed and deployed as shown in the following diagram:
Comparison between Number of Pipelines before and after PCF
The below screenshot shows the number of Pipelines in use without the Proposed PCF Solution for a set of BizTalk applications
The below screenshot shows the number of Pipelines in use with the Proposed PCF Solution for the same set of BizTalk applications
Conclusion:
With framework’s design leveraging Object Oriented principle and .NET Reflection, the numbers will stay flat or will go up very slowly in the foreseeable future, therefore curbing the proliferation of pipelines and pipeline components. All BizTalk applications will share the pipelines and framework. The complexity of managing various functionalities will be shifted to Global Assembly Cache, which is easier to maintain and version. This allows us to consolidate infrastructure footprint which in turn reducing development, deployment and maintenance cost.