Saturday, January 10, 2009

2009-01-10 Saturday - Integration Case Study

I've recently completed the integration of another major interface for a client engagement.

Business Objective

To provide construction project managers (that are managing multi-year, multi-million dollar projects) with critical financial data integrated into their project management and planning tools.

Technical Challenges

The data required originated in a mainframe legacy financial accounting system (e.g. CICS/MVS, COBOL/VSAM, NATURAL/ADABAS).

The 3rd party vendor's project management application stored data in a Microsoft SQL Server database, but the vendor absolutely advised against direct data access (e.g. using Informatica type integration tools) to manipulate the database (due to significant business rules that must be applied to ensure the integrity of the project management system's data).

The vendor's integration API was Java based - the client organization had no internal Java expertise - and thus no Java development tools, processes, or standards.

Best Practices Introduced

Established an [internal] wiki site (TikiWiki) as an "information radiator" to communicate the status of the integration development effort with various organizational units (e.g. Operations, Project Management, Support, Business Analysts). The wiki was also used to document Java development best practices, standards, and lessons learned.

Wrote a daily [internal] blog (TikiWiki) to document and communicate the techniques used, problems encountered, solutions devised, resources and references leveraged, and the ongoing progress of the development effort.

Established a bug/issue tracking system (FlySpray).

Introduced an Agile iterative development approach for the design/construction effort.

Established a Software Configuration Management process with documented and easily repeatable processes leveraging Subversion and Ant.


Conducted mentoring and formal training on the new tools introduced

Developed a high-level design using Sparx System's Enterprise Architect

An existing Informatica data warehouse feed was utilized as the authoritative source for the required data.

Developed a core Enterprise Architecture library of reusable components for current/future interface development:
- Application Context
- Application Configuration
- Batch Service Processing
- Reporting
- Enterprise Consolidated Logging
-- Integrated with Windows Event Logging
-- Integrated with Microsoft MSMQ
-- Integrated with Log4j
- General Purpose Utilities
~6,000 lines of Java code

Developed a core application-specific (project management) library of reusable components for current/future interface development:
- Business Domain Abstractions
- Discrete Units of Work Components
- Business Process Services (Aggregated Units of Work Components)
- Orchestration of Business Processes
- API Session Management wrapper
~3,000 lines of Java code

Developed the interface specific integration components:
- Interface-specific Business Rules
- Data Validation
- Job Service Orchestration
~3,000 lines of Java code

Tools/Components Used

Sparx Systems Enterprise Architect

- Wiki
- Blogs

- Bugs
- Issues
- Feature Request

Apache Commons

Apache log4j

Apache Web Server
- WebSVN

- statSVN
- TortoiseSVN

Eclipse Ganymede IDE
- Subclipse

Sun Microsystems Java SE 5 (JDK 1.5)
- Version of the JDK dictated by the vendor's API requirements



Glassfish JEE Application Server

- Continuous Integration Build Server



Lessons Learned

Using PMD and FindBugs was a significant factor in elevating the quality of the code.

Using Hudson for Continuous Integration Build - and JUnit for automated unit testing - allowed for rapid iteration, testing, and deployment to a development server environment.

XMLs Impact on Performance: This interface processes a significant amount of data. The vendor's integration API heavily uses XML internally. One line of code was found to be responsible for an execution time for a single project's data that was (5x) more than expected. Minimizing the number of properties requested for retrieval from the API (thus reducing the size of XML involved) reduced execution time to (x) for that step in processing.

The vendor's naming conventions differed (sometimes substantailly) between their API business objects and their database tables/columns. A common ontology would have been very beneficial.

No comments: