Splunk Integration – Search Queries & Output Processing with PilotFish

This is the PilotFish eiConsole for Splunk. It’s the integrated development environment you’ll use to build, maintain, test and deploy all of your different integrations. In this demonstration today, we’ll be showing how the eiConsole for Splunk’s Search Listener can be leveraged to execute Splunk search queries and then process the output.

This is very useful for situations where you have Splunk searches that you’re running on some scheduled interval, and you’d like to be able to take that data and externalize it and do some transformations if necessary, and then transport that data to any number of target systems.

One thing to note here is that PilotFish is a common model for transformations, so every one of these integrations, no matter how complex, goes through the same 7 stage assembly line process. Data enters the system via the listener stage and is then transformed from its incoming data format into a canonical XML representation in the source transform stage. It then moves on to the routing stage where that transaction can be routed to any number of target systems. Then the data moves on to our target transform stage, where that underlying XML is then transformed into whatever outgoing data format is expected by our target system. Then, finally, at the transport stage, that data is sent to its final destination.

Looking at a high level here, what we have is we’re using our Splunk Search Listener as the main focal point of this demonstration. And so, the Splunk Search Listener is able to execute either a custom search query or a saved query (if you have turned on the saved option in Splunk) and then get those search results in either XML, JSON or CSV. Then once that search data is in the system, it’s available for any transformation that you’d like.

So, in this case here, we’re using our search listener to get our data into the system and then we are moving to our source transform. We’re not doing any transformation here, because we already have our search results in XML, which is our format that we’d like. Then, finally for our target systems; we’re taking those search results, and we’re inserting into a Hadoop file system for further machine learning use. And then we’re also using our database transport here to take those search results and insert into our archiving database.

Now, one of the other PilotFish tools that we’re leveraging here, is called SQLXML. This is basically an execution language; it’s basically a way of interacting with databases at an XL level. We’re defining those selects and inserts, call stored procedures, all those different SQL database functions, and we represent those in XML. Then we’re able to use our database transport here to then execute those instructions, those selects, and inserts against our archive database here.

And so that’s really it. Once again, so we’re using our Splunk Search Listener here to execute a search against the running Splunk instance and then we’re starting transactions with those results. And then we’re taking that data, and we are updating this Hadoop HDFS file system, and then we’re also updating our archive database with those search results. So if you’d like to take a test drive, you can download a Free 90-Day Trial of the eiConsole for Splunk. from our website.

*Splunk is a registered trademark of Splunk Inc. in the United States and other countries.