Connamara, continuing a commitment to open source in capital markets, releases into open source a Logstash filter allowing DevOps teams to better analyze FIX Logs.
Connamara Systems is excited to announce the release of an open source FIX Protocol Logstash Filter!
FIX logs are often crucial in supporting trading systems. However, in most scenarios, processing and parsing these logs are cumbersome for all but veteran FIX aficionados. Logstash is a log parsing tool that excels at transforming raw and unstructured log messages into well-formed data that can be processed downstream by standard devops tools. The FIX Protocol Logstash Filter makes it easy for developers to integrate Logstash with existing FIX applications.
Given a FIX log file that looks like this:
2015-08-26 23:08:38,096 FIX.4.2:DUMMY_INC->ANOTHER_INC: 8=FIX.4.2�9=184�35=F�34=2�49=ANOTHER_INC�50=DefaultSenderSubID�52=20150826-23:08:38.094�56=DUMMY_INC�1=DefaultAccount�11=clordid_of_cancel�41=151012569�54=1�55=ITER�60=20250407-13:14:15�167=FUT�200=201512�10=147�
2015-08-31 17:48:20,890 FIXT.1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT.1.1�9=140�35=W�34=2�49=DUMMY_INC�52=20150831-17:48:20.890�56=ANOTHER_INC�22=99�48=.AQUA-W�262=golden_path_test�268=1�269=3�270=640754�272=20150831�273=17:48:20.882�10=070�
2015-08-31 20:48:26,536 FIXT.1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT.1.1�9=189�35=W�34=5�49=DUMMY_INC�52=20150831-20:48:26.535�56=ANOTHER_INC�22=99�48=ITRZ21�262=req_A�268=2�269=0�270=0.01005�271=10�272=20150831�273=20:48:26.514�269=1�270=0.0101�271=2�272=20150831�273=20:48:26.514�10=123�
The FIX Protocol Logstash Filter plugin can read the FIX log as an input and turn it into something like this:
Logstash can take the output of the transformed FIX message and feed it into a time series data base like Elasticsearch. Elasticsearch provides a very simple REST API that allows you to easily query FIX logs that have been processed by the FIX Logstash Filter.
Once in Elasticsearch, the FIX logs can be visualized with custom dashboards using Kibana, a visualization and analysis tool for time series data.
This combination of Elasticsearch, Logstash, and Kibana (ELK for short) can empower an operations team to dissect FIX logs on an enterprise scale and to identify anomalous activity at a glance.
Documentation and source can be found on Github. Get started parsing your FIX logs today!
