High Frequency FIX Parser
C++ library for high frequency messaging with the Financial Information Exchange (FIX) protocol.
|
The High Frequency FIX Parser library is an open source implementation of tagvalue FIX (classic FIX) intended for use by developers of high frequency, low latency financial software. The purpose of the library is to do fast, efficient encoding and decoding of FIX in place, at the location of the I/O buffer. The library does not use intermediate message objects, and it does no memory allocation on the free store (the “heap”).
hffix library is not certified by any industry-leading committees. It is not an “engine.” It is not an “adaptor.” It has no threading, no I/O, no object-oriented subtyping. It is just a superfast parser and serializer in plain modern generic-iterator-style C++98.
The main repository is at https://github.com/jamesdbrock/hffix
To see an example of the library in action, enter these four commands at your shell prompt. This example uses the fixprint
utility which comes with the hffix library. The result will be a colorized and pretty-printed FIX 5.0 test data set.
git clone https://github.com/jamesdbrock/hffix.git cd hffix make fixprint util/bin/fixprint --color < test/data/fix.5.0.set.2 | less -R
The library is header-only, so there is nothing to link. To use the hffix.hpp
library for C++ FIX development, place the two header files in your include path and #include <hffix.hpp>
.
Full Doxygen is on the internet at https://jamesdbrock.github.io/hffix
To build the Doxygen html documentation in the doc/html
directory and view it:
git clone https://github.com/jamesdbrock/hffix.git cd hffix make doc xdg-open doc/html/index.html
High Frequency FIX Parser tries to follow the C++ Core Guidelines and the Boost Library Requirements and Guidelines. It is modern platform-independent header-only C++98 and depends only on the C++ Standard Library. It is patterned after the C++ Standard Template Library, and it models each FIX message with Container and Iterator concepts. It employs compile-time generic templates but does not employ object-oriented inheritance.
The design criteria for High Frequency FIX Parser are based on our experience passing messages to various FIX hosts for high frequency quantitative trading at T3 Trading Group, LLC. These design criteria follow from the observation that the latency of a trading system depends on the following list of considerations, in descending order of importance.
The hffix library assumes that the library user will want to make their own choices about considerations 2 and 3, and the hffix library focuses on providing good answers for consideration 4. It does this by using only stack memory and I/O buffer memory, and never allocating on the free store.
In contrast, the popular alternative QuickFix library forces the user to use the QuickFix solution to considerations 2 and 3 for threads and sockets, and most of QuickFix's choices about threads and sockets are not great. QuickFix also has an inefficient object-oriented design for consideration 4.
See also CppCon 2017: Carl Cook “When a Microsecond Is an Eternity: High Performance Trading Systems in C++”
All of the Financial Information Exchange (FIX) protocol specification versions supported by the library are bundled into the the distribution, in the fixspec
directory. As a convenience for the developer, the High Frequency FIX Parser library includes a program which parses the FIX protocol specification documents and generates the include/hffix_fields.hpp
file. That file hffix::tag
enums and an hffix::dictionary_init_field
function which allows fields to be referred to by name instead of number during both compile-time and run-time.
The library is platform-independent C++98, and is tested on Linux with gcc and clang on Github Actions.
The main High Frequency FIX Parser Library is distributed under the open source FreeBSD License, also known as the Simplified BSD License.
Some extra components are under the Boost Software License.
Included FIX specs are copyright FIX Protocol, Limited.
For reading FIX messages, High Frequency FIX Parser presents an STL-style immutable Forward Iterator interface. Writing fields is done serially with an interface similar to an STL-style Back Insertion Sequence Container. Reading and writing are done directly on the I/O buffer, without any intermediate objects.
The disadvantage of this implementation is that the message API provides serial access to fields, not random access. Of course, when we're writing a message, random access isn't important, just write out the fields in order. When we're reading a message, it's easy enough to pretend that we have random access by using iterator algorithms like std::find
. A convenience algorithm hffix::message_reader::find_with_hint
is provided by this library for efficiently reading fields when you know approximately what field order to expect. See the examples below for how this works out in practice.
The advantage is that this enables the High Frequency FIX Parser library to completely avoid free store memory allocation. The library performs all memory allocation on the stack, and the library never requires developers using the library to allocate anything on the free store with new
or malloc
.
Field values in the FIX protocol are always encoded on the wire as ASCII, and High Frequency FIX Parser exposes field values to the developer as iterator range char const* begin(), char const* end()
. High Frequency FIX Parser also provides a complete set of conversion functions to native C++ types for ints, decimal floats, dates and times, et cetera — see documentation for hffix::message_writer
and hffix::field_value
.
Some functions in this library may throw std::logic_error
if a precondition is not met by the programmer, so you can usually prevent the library from throwing exceptions by meeting the precondition. All methods, functions, constructors, and destructors provide the No-Throw exception guarantee unless they are documented to throw exceptions, in which case they provide the Basic exception guarantee. See documentation for details.
High Frequency FIX Parser is not thread-aware at all and has no threads, mutexes, locks, or atomic operations.
All const
methods of the hffix::message_reader
are safe for concurrent calls.
The hffix::message_writer
is not safe for concurrent calls.
hffix::message_reader
and hffix::message_writer
have no storage of their own, they read and write fields directly on an I/O buffer. The developer must guarantee that the buffer endures while fields are being read or written.
Managing sessions requires making choices about sockets and threads. High Frequency FIX Parser does not manage sessions. It is intended for developers who want a FIX parser with which to build a session manager for a high-performance trading system that already has a socket and threading architecture.
FIX has transport-layer features mixed in with the messages, and most FIX hosts have various quirks in the way they employ the administrative messages. To manage a FIX session your application will need to match the the transport-layer and administrative features of the other FIX host. High Frequency FIX Parser has the flexibility to express any subset or proprietary superset of FIX.
See also FIX Session-level Test Cases and Expected Behaviors
No native floating-point numeric types (double
, float
) are employed by the library. ASCII-encoded decimal numbers are represented by integral mantissa and exponent. See hffix::message_writer::push_back_decimal()
and hffix::field_value::as_decimal()
. As with every FIX data type, the High Frequency FIX library user has the option to serialize and deserialize numeric fields themself rather than use these methods.
High Frequency FIX Parser supports the binary data field types such as SecureData, but it does not implement any of the EncryptMethods suggested by the FIX specifications. If you want to encrypt or decrypt some data you'll have to do the encryption or decryption yourself.
High Frequency FIX Parser will calculate the CheckSum field for all messages that you encode. It can validate the CheckSum of messages decoded, but does not do that calculation unless you explicitly ask for it.
The MsgSeqNum field in the FIX Standard Header is exposed for reading and writing.
The administrative messages Logon, Logout, ResendRequest, Heartbeat, TestRequest, SeqReset-Reset and SeqReset-GapFill don't get special treatment in High Frequency FIX Parser. Any administrative message can be encoded or decoded like any other message.
High Frequency FIX Parser does not enforce the data type of the Field Definitions for content fields in the FIX spec, so the developer is free to read or write any tag number with any field data type. See hffix::message_writer
and hffix::field_value
documentation under Extension for details.
This example program is in the hffix repository at test/src/writer01.cpp
.
It writes a Logon message and a New Order - Single message to stdout
.
This example program is in the hffix repository at test/src/reader01.cpp
.
It reads messages from stdin
. If it finds a Logon message or a New Order - Single message, then it prints out some information about their fields.
The writer example can be piped to the reader example. Running these commands:
make examples test/bin/writer01 | test/bin/reader01
Should produce output like this:
Logon message
SenderCompID = AAAA
MsgSeqNum = 1
SendingTime = 2014-Sep-26 15:27:38.789000
The next field is EncryptMethod = 0
New Order Single message
Buy OIH 100 @ $50001E-2
To examine the output from test/bin/writer01
program, you can also use util/bin/fixprint
, like this:
make examples make fixprint test/bin/writer01 | util/bin/fixprint --color
Which will produce output like this:
FIX.4.2 MsgType_35=A_Logon SenderCompID_49=AAAA TargetCompID_56=BBBB MsgSeqNum_34=1 SendingTime_52=20140928-07:12:06.000 EncryptMethod_98=0 HeartBtInt_108=10 FIX.4.2 MsgType_35=D_NewOrderSingle SenderCompID_49=AAAA TargetCompID_56=BBBB MsgSeqNum_34=2 SendingTime_52=20140928-07:12:06.000 ClOrdID_11=A1 HandlInst_21=1 Symbol_55=OIH Side_54=1 TransactTime_60=20140928-07:12:06.000 OrderQty_38=100 OrdType_40=2 Price_44=500.01 TimeInForce_59=1
If the Boost Date_Time library is available in your build environment, boost::posix_time::ptime
, boost::posix_time::time_duration
, and boost::gregorian::date
will be automatically supported for the various FIX date and time field types. See hffix::message_writer
and hffix::field_value
documentation for details.
To enable High Frequency FIX Parser support for the Boost Date_Time library types, include the Boost libraries before the hffix.hpp library, like this:
To prevent High Frequency FIX Parser support for the Boost Date_Time library, #define HFFIX_NO_BOOST_DATETIME
before including hffix.hpp
:
std::chrono
If you are building under C++11 or higher then the std::chrono::time_point
and std::chrono::duration
types are supported for the various FIX date and time field types. See hffix::message_writer
and hffix::field_value
documentation for details.
Sample data sources, discovered by Googling.
test/data/fix.4.1.set.1
http://fixparser.targetcompid.com/test/data/fix.5.0.set.1
https://www.jse.co.za/content/JSETechnologyDocumentItems/03.%20JSE%20Indices.recv.log.txttest/data/fix.5.0.set.2
http://blablastreet.com/workshop/TestSocketServer/TestSocketServer/bin/Debug/store/FIXT.1.1-ATP1CMEMY-OMSCMEMY.bodyThe Chicago Mercantile Exchange is also a good source of sample data files, but the files are too big to include in this repository. The script test/curl.cme.data.sh
shows how to download them. Run curl.cme.data.sh
in the test/
directory.
Q: I have a bunch of different threads serializing and sending FIX messages out one socket. When each message is sent it needs a MsgSeqNum, but at serialization time I don't know what the MsgSeqNum will be, I only know that at sending time.
A: That multi-threading model is not a good choice for your software. The performance penalty for that threading model is much greater than the performance advantage of this non-allocating parser library. You should consider redesigning to use a single-threaded simultaneous-wait event loop like libev or Boost Asio. If you insist on multi-threading, then you could do something like this code example.
From FIX-50_SP2_VOL-1_w_Errata_20110818.pdf page 21:
If the repeating group is used, the first field of the repeating group is required. This allows implementations of the protocol to use the first field as a "delimiter" indicating a new repeating group entry. The first field listed after the NoXXX, then becomes conditionally required if the NoXXX field is greater than zero.
The beginning of each Repeating Group is marked by a field with a “NoXXX” field. By convention, Repeating Groups are usually located at the end of the message, so the end of the message marks the end of the Repeating Group. In this example we assume that the convention holds, and the repeating group is at the end of the message. If the repeating group were not at the end of the message then we'd have to pay attention to the value of the “NoXXX” fields, which is left as an exercise for the reader.
This is an example of iterating over the nested Repeating Groups when reading a Mass Quote message. The Mass Quote message has QuoteSet Repeating Groups, and nested inside those groups are QuoteEntry Repeating Groups, see fix-42-with_errata_20010501.pdf page 52. In each repeated QuoteSet Group, hffix::tag::QuoteSetID
is always the first field. In each repeated QuoteEntry Group, hffix::tag::QuoteEntryID
is always the first field.
Please make an issue on this repository if you have any questions about how the library works or suggestions about how to improve it. If you want to talk privately then email me at james.nosp@m.broc.nosp@m.k@gma.nosp@m.il.c.nosp@m.om.
Our build system is make
. The Makefile
provides targets and pseudo-targets for various parts of the library.
make test
to run the test suite.
make doc
to build the documentation.
We have a flake.nix
.
The flake.nix
declares a Nix shell development environment which provides all dependencies for every target in the Makefile
, including Doxygen. Enter the Nix shell by installing Nix and then running nix develop
in this directory. From the nix develop
prompt, you will be able to build all Makefile
targets.
The Nix hffix
package will provide the hffix
C++ library.
Run the fixprint
utility:
Pull requests welcome!
If you want to submit a bugfix pull request, then I would be grateful if you would break the pull request up into two commits:
When a FIX client connects to a FIX server, the client doesn't know what sequence number to use for the Logon message.
Either the client can choose to reset both sequence numbers, in which case the client may miss messages, or not, in which case the client is subject to the Resend Request race condition.
After Logon response from the server, the client may begin sending messages, but the client has to wait some amount of time because the server may send Resend Request. If the client sends any message to the server while the server is preparing to send Resend Request, then the server's response is not defined by the FIX specification, and some servers implementations may seize up in confusion at that point.
This library only depends on C++98.
The library was designed with the intention of interacting well with C++11 features such as, for example, auto
, or anonymous inline functions passed as the UnaryPredicate
to hffix::find_with_hint
. All the classes own no resources and are optimized for pass-by-value so move semantics are mostly irrelevent.
std::chrono
is supported in a -std=c++11
build environment.
std::string_view
is supported in a -std=c++17
build environment.
2024-04-18 | v1.4.1 | Fix for |
---|---|---|
2024-02-19 | v1.4.0 |
Add |
2023-04-28 | v1.3.0 |
Add |
2022-05-22 | v1.2.1 |
Bugfix: |
2021-10-11 | v1.1.1 |
Bugfix: |
2021-05-17 | v1.1.0 |
Added support for nanosecond-precision timestamps, by Evan Wies @neomantra |
2020-10-11 | v1.0.1 |
Changed the |
2020-08-15 | v1.0.0 |
Version 1. |
2018-12-09 |
Added support for | |
2018-12-09 |
Replace parsing of all the old FIX specs with parsing of the FIX Repository https://www.fixtrading.org/standards/fix-repository/ to generate This results in a lot more Keep all the old FIX spec documents in the repo for reference. Breaking ChangesSome field names in | |
2018-10-24 |
Replace the Python | |
2017-09-12 |
Added support for |
Thank you to the following contributors, who mostly don't show up in the Github Contributors list because of my habit of adding “cleanup” commits to other people's pull requests.