Normalizing Data to Ingest into Splunk®
This is the PilotFish eiConsole for Splunk. It’s the integrated development environment that you will use to build, maintain, test and deploy all of your integrations.
In this demonstration today, we’ll be showing how the eiConsole’s built-in Splunk event logging transport can be leveraged to take input from multiple disparate systems and then normalize that data for Splunk use.
So it’s important to note with the PilotFish common model for integration that every one of these integrations, no matter how complex, how many sources or targets, it goes through the same 7-stage assembly line process. Data enters the system flow center stage; it is then transformed from that incoming data format into a canonical XML representation in the source transform stage. Then moves on to the routing stage where that data can be routed arbitrarily to any number of target systems.
Then the data moves into our target side, where our target transform, where that data is taken from that underlying XML format and turned into whatever data format our outgoing system expects, whether it’s JSON, EDI, HL7, what have you. And then finally at the transport stage, that data is sent to its final destination system.
So as we mentioned here, at a high level, what we’re doing in this demonstration is we’re taking an EDI from this generic socket, we’re exposing a RESTful endpoint that’s consuming some JSON and then we’re also pulling a Microsoft SQL Server database. And then we’re starting transactions with the data that we get.
And then here in our Source Transform stage, we’re taking that incoming EDI, JSON and our database SQL results and we’re normalizing all that data into XML.
So to do that we’re using our transformation modules. So one thing you notice here is that each of our modules is listed. There are our transports. Transformation modules follow the same basic design principle where each of these modules has a base set of configuration that’s required for use and then some more advanced configuration options if necessary.
So here we’re using our EDI transformation module just to take that raw EDI input and turn it into this baseline XML canonical representation. And the same for our JSON here and our XML results from our SQL database listener.
So next we move on to the routing stage, and since we only have one target system, we’re not doing a whole lot here. We’re just basically routing our data through to our target system.
And then finally, we move on to our transport. And so for this, we’re utilizing one of our new pieces of Splunk functionality. This is the Splunk Event Logging Transport. And so what this is basically doing, it is posting any incoming transaction data into a running Splunk enterprise instance via HTTP.
And so all you need to do is to find your connection, some basic index host source name information and then any data that comes through this system when it reaches the transport stage will then be inserted into the appropriate Splunk index as normal event logs. And they’ll be ready for use there.
So just to recap this demonstration here, we’re using our Splunk event logging transport to take data from multiple disparate systems. We’re normalizing that data, and then we are taking that data, and we’re posting it into Splunk as transaction event logs.
So if you’d like to take a test drive, you can Download a Free 90-Day Trial of the eiConsole for Splunk from our website.
For more information, please call us at 813 864 8662 or click the link below to email us.
*Splunk is a registered trademark of Splunk Inc. in the United States and other countries.