CSV Writing

Use this page to write structured CSV data asynchronously using Quill’s backend thread.

The library provides functionality for asynchronously writing CSV files. Formatting and I/O operations are managed by the backend thread, allowing for efficient and minimal-overhead CSV file writing on the hot path. This feature can be used alongside regular logging.

The CsvWriter class is a utility designed to facilitate asynchronous CSV file writing.

Call CsvWriter::close() before stopping the backend worker if you need deterministic logger removal and file closure. The destructor performs best-effort asynchronous cleanup and does not block.

When multiple writers share the same file, use the configuration or sink constructor overload and pass should_write_header=false for every writer except the one responsible for the header. Append mode checks the file size, which cannot detect another writer’s queued or buffered header. Alternatively, share one CsvWriter and finish all row submissions before calling close().

CSV Writing To File

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#include "quill/Backend.h"
#include "quill/CsvWriter.h"
#include "quill/LogMacros.h"
#include "quill/core/FrontendOptions.h"
#include "quill/sinks/ConsoleSink.h"

/**
 * This example shows how to define a CSV schema, instantiate a CsvWriter object, and append rows to
 * the CSV file. The logging operations are handled asynchronously by the backend worker thread.
 */

struct OrderCsvSchema
{
  static constexpr char const* header = "order_id,symbol,quantity,price,side";
  static constexpr char const* format = "{},{},{},{:.2f},{}";
};

int main()
{
  quill::BackendOptions backend_options;
  quill::Backend::start(backend_options);

  quill::Logger* logger = quill::Frontend::create_or_get_logger(
    "root", quill::Frontend::create_or_get_sink<quill::ConsoleSink>("sink_id_1"));

  LOG_INFO(logger, "CSV writing example");

  quill::CsvWriter<OrderCsvSchema, quill::FrontendOptions> csv_writer{"orders.csv"};
  csv_writer.append_row(13212123, "AAPL", 100, 210.32321, "BUY");
  csv_writer.append_row(132121123, "META", 300, 478.32321, "SELL");
  csv_writer.append_row(13212123, "AAPL", 120, 210.42321, "BUY");
  csv_writer.close();
}

Csv output (orders.csv):

order_id,symbol,quantity,price,side
13212123,AAPL,100,210.32,BUY
132121123,META,300,478.32,SELL
13212123,AAPL,120,210.42,BUY

Field Escaping

Fields are written verbatim. A string field containing a comma, double quote or line break corrupts the CSV structure. For such fields, pass the value through quill::utility::csv_escape_field() from quill/Utility.h, which quotes the field according to RFC 4180 (fields without special characters are returned unchanged):

csv_writer.append_row(13212123, quill::utility::csv_escape_field("A,B \"C\""), 100, 210.32, "BUY");

Note

csv_escape_field() returns a new std::string. When writing on a latency-sensitive path, prefer calling it only for fields that can actually contain special characters.

CSV Writing To Existing Sink

It is possible to pass an existing Sink, or a custom user-created Sink, to the CSV file for output. The following example shows how to use the console sink.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#include "quill/Backend.h"
#include "quill/CsvWriter.h"
#include "quill/LogMacros.h"
#include "quill/core/FrontendOptions.h"
#include "quill/sinks/ConsoleSink.h"

struct OrderCsvSchema
{
  static constexpr char const* header = "order_id,symbol,quantity,price,side";
  static constexpr char const* format = "{},{},{},{:.2f},{}";
};

int main()
{
  quill::Backend::start();

  quill::Logger* logger = quill::Frontend::create_or_get_logger(
    "root", quill::Frontend::create_or_get_sink<quill::ConsoleSink>("sink_id_1"));

  LOG_INFO(logger, "CSV writing example");

  // Pass the existing ConsoleSink to the CsvWriter to also emit CSV rows there
  quill::CsvWriter<OrderCsvSchema, quill::FrontendOptions> csv_writer{
    "orders.csv", quill::Frontend::get_sink("sink_id_1")};

  csv_writer.append_row(13212123, "AAPL", 100, 210.32321, "BUY");
  csv_writer.append_row(132121123, "META", 300, 478.32321, "SELL");
  csv_writer.append_row(13212123, "AAPL", 120, 210.42321, "BUY");
  csv_writer.close();
}