Back to Pilotfish Home

Case Studies – Data Analytics & Reporting Integration

Big Data, Big Data Analytics Integration Challenges


High performance delivered! With dominant position and market share in hospitals and practices throughout the United States, the client is focused on delivering a single source of provider financial, operational and quality metrics, coupled with proprietary, value-added analytical services. This massive big data integration challenge, a tough bake off and implementation, proves superiority of PilotFish Integration Engine Solutions.


The Client

The client provides strategic consulting, research, cloud-based software solutions, and comprehensive implementation and management services. Its business intelligence software and services are relied upon by hospitals, health systems and physician clinics across the USA. Its solutions must aggregate significant volumes of extremely diverse administrative and clinical data into provider-specific data repositories to deliver its valued high-quality metrics and actionable reporting.

The Challenge

The software solutions offered by the client require acquisition, validation, normalization and analysis of massive amounts of clinical and administrative patient data from a wide variety of healthcare providers. This information must be aggregated through numerous real-time and batch data channels, accurately parsed and cleansed, then loaded into the client’s proprietary analytics platform.

Their existing integration tools, comprised of a combination of home grown and commercial software, were proving incapable of readily handling the growing diversity and number of data feeds required by the organization.

The data sets in question are both high volume and exceptionally diverse. Batch data formats include administrative ANSI X.12 EDI files, such as 837I (Institutional) and 837P (Professional) claims, CMS 1500, custom formats, as well as more traditional tabular batches of data like comma-separated files (CSVs). Real-time messages, including HL7 2.x messaging and XML, are also processed.

While the representation of the data can take many forms, it typically includes a semantic payload from which the client could ascertain anonymous historical patient information upon which their population health and quality reports could be based. Once the data was received, it needed to be transformed into their specific format and then placed into normalized relational database tables. Throughput was also an integral part of the challenge, as the data acquisition process needed to keep pace with the flood of input being received on a daily basis so as to be able to provide real-time reports to their customers.

Another challenge faced by the client was that the distillation of this data was coming from widely differing sources. They had pushed the limits of the number of unique mappings and transformation rules that could feasibly be maintained using their current solution of home grown and commercial software. The status quo had become unwieldy to manage, and data quality issues were beginning to arise with increasing frequency.

The Solution

Prior to selecting PilotFish, the client undertook an extensive evaluation process of all the major interface engines in the industry. This evaluation process culminated in a “bake off” among the selected finalists. The finalists were tasked with demonstrating their solutions’ ability to solve the client’s integration challenges represented in a series of use cases. PilotFish proved to be the only solution capable of solving and completing the integration challenge for each specific use case.

After the selection of PilotFish, the client collaborated with the PilotFish professional services team to implement the PilotFish integration engine solution – which includes the eiPlatform, eiConsole for Healthcare and eiDashboard components. The architected solution was an enterprise level, high performance, accurate and scalable “data acquisition” framework. This architected solution is capable of accepting any data format from providers and transforming it so that it can feed any of the firm’s cloud-based solutions.

The PilotFish solution has provided the architectural flexibility to adapt to diverse data flavors and formats, using a combination of out-of-the-box parsers and finely tuned custom components. Among these components is a custom EDI parser that was developed and is capable of accurately and quickly parsing the volumes of heterogeneous ANSI X.12 EDI files required by the client.

The Benefits

The PilotFish enterprise level solution allows all divisions in the organization to benefit from a unified architecture leveraging a federated integration model. Each division, where the expertise is held, can now be responsible for its own governance. The componentized nature of the product maximizes the potential for reuse, which allows the client to bring on new customers faster. Many times, new feeds can now be accepted by making only minor tweaks to existing interfaces.

In partnership with PilotFish, the client can now quickly and more accurately populate its databases with patient data. New client implementations are up and running earlier, while the client is delivering higher-quality analytics in less time. An improved level of service has increased existing customer satisfaction, while improved productivity allows the client to expand more rapidly without increasing implementation staff.

The Future State

The client is an advocate of HL7 FHIR as a modern standard to accelerate interoperability across healthcare. Due to the highly extensible architecture of the PilotFish solution, support for new standards can easily be added to the product. Consequently, the client is well positioned to support the anticipated widespread adoption of the HL7 FHIR standard using the newly introduced FHIR capabilities of the eiConsole and eiPlatform. (FHIR is short for Fast Health Interoperability Resource framework, which is the HL7 nonprofit group’s internationally recognized API and structured data format for secure and trusted information flow.)

PilotFish support for the new FHIR standard includes a FHIR Format Reader with support for Profiles and full support of the specification in the product’s Data Mapper including the metadata. In addition to the new FHIR support, PilotFish had already implemented support for a number of core technologies that FHIR users will benefit from, such as Web Standards like JSON, HTTP, etc. and RESTful architecture.

PilotFish was confidently selected because its highly extensible architecture allows the client to profit from the latest standards and technology. By leveraging PilotFish’s new support for FHIR, their customers with FHIR-compliant EHR systems will be interoperable with the client’s systems. This will enable the client to more quickly process millions of clinical records and rapidly expand connections to even more customer systems and exponentially increase their market share.


Over the course of nearly 15 years and hundreds of implementations, PilotFish has developed and refined a methodology for the configuration, testing, and deployment of interfaces and process orchestrations. We have an unblemished track record of success. Through years of Bake-Offs and Proof of Concepts (POCs) we have demonstrated the value of our integration engine solutions to future customers. Let us conduct a Free Use Case Evaluation for you to determine where PilotFish can provide the most value to your organization and solve your most complex integration challenges.

To schedule a Free Use Case Evaluation and to learn about what PilotFish Solutions can do for your organization please click the link below or contact us at 860 632 9900 or Email us.

HL7 is the registered trademark of Health Level Seven International.

This is a unique website which will require a more modern browser to work! Please upgrade today!