Robotics Service Bus » RSBag

h1. TIDE log Benchmark

h2. Chunk Sizes and Indices

We simulated "scanning" large TIDE log files with varying numbers of entries per chunk and entry sizes. By "scanning" we mean seeking to all @CHNK@ blocks, reading the block headers and reading all entry headers of the respective @CHNK@ s.

|_.Number of Chunks |_. Number of Entries per Chunk |_. Size of one Entry |_.Filesize [GB] |_. Seeks and Reads |_. "Scan" Time [s] |

|>.             100 |>.                         100 |>.         1,000,000 |>.        ~ 9.4 |>.          10,000 |>.         < 0.002 |

|>.             100 |>.                      10,000 |>.            10,000 |>.        ~ 9.4 |>.       1,000,000 |>.         < 0.200 |

|>.             100 |>.                   1,000,000 |>.               100 |>.       ~ 11.0 |>.     100,000,000 |>.       < 200.000 |

The number of seek and read operations grows linearly in the number of entry headers that have to be read. In the first two cases, the runtime reflects this very clearly. In the third cases, the overhead starts dominating and performance worsens accordingly. Almost the entire runtime consists of IOWAIT in this case (~ 15 s user, ~ 10 s system, ~ 175 s iowait).

Conclusion

* For high-frequency data with small individual entries, scanning all entries of a file is unacceptable

** Indexes (via @INDX@ blocks) are unavoidable and should probably be made mandatory; at least for certain file structures

* Since all @CHNK@ blocks have to visited, even with indexes, it is important to accumulate enough data in each chunk

** On the other hand, periodically writing complete chunks to disc may be desirable for data safety and error recovery reasons

Observation

* @TIDE.num_chunks@ and @TIDE.num_channels@ are never really used since the whole file has to be scanned anyway

* If data integrity is a concern, a hash of the file's content could be stored instead and only verified when explicitly requested