Enhancement #2572

Improve replay performance for large numbers of channels

Added by J. Moringen almost 5 years ago. Updated over 3 years ago.

Status:ResolvedStart date:06/19/2016
Priority:NormalDue date:
Assignee:J. Moringen% Done:

100%

Category:Replay
Target version:Robotics Service Bus - rsb-0.17

Description

Replays with large number of channels currently perform badly because the parallel iterator stepping does lots of unnecessary timestamp comparisons.

Associated revisions

Revision 3aaec5d2
Added by J. Moringen almost 5 years ago

Turned parallel iterators into structures in src/view/serialized.lisp

refs #2572

  • src/view/serialized.lisp (header): updated copyright
    (%iterator): new structure; state of an in-parallel steppable iterator
    (%iterator-step): adapted to changed iterator representation
    (%iterator-min): likewise
    (%iterator-for-forward-step): likewise
    (%iterator-for-backward-step): likewise
    (make-simple-sequence-iterator serialized): likewise
    (iterator-endp serialized serialized-iterator t t): likewise
    (iterator-element serialized-iterator serialized-iterator): likewise

Revision fd551245
Added by J. Moringen almost 5 years ago

Reduced number of iterator comparisons in src/view/serialized.lisp

refs #2572

  • src/view/serialized.lisp (%iterator<): renamed %iterator{-min -> <};
    compare iterators instead of returning the minimum
    (%iterator-for-forward-step): removed; no longer needed
    (%iterator-for-backward-step): adapted to use `%iterator<' instead of
    `%iterator-min'
    (serialized-iterator::sorted): new sorted; heap of sorted iterators
    (shared-initialize :after serialized-iterator t): initialize heap of
    sorted iterators
    (iterator-step serialized serialized-iterator t): use top value of
    heap for forward step; reconstruct heap for backward step
  • cl-rsbag.asd (system cl-rsbag): added system dependency on pileup

Revision 7dc5fc70
Added by J. Moringen over 3 years ago

Use integer representation of timestamps in src/backend/**/*.lisp

fixes #2572

  • src/channel.lisp (header): updated copyright
    (channel-timestamps/raw channel): renamed channel-timestamps{ >
    /raw}; return uint64 timestamp sequence returned by backend
    (channel-timestamps channel): new method; wrap raw timestamp
    sequence in `channel-timestamp' instance which translates elements
    to `timestamp' instances
    (entry channel timestamp): translate timestamp to uint64 before
    calling backend
    (setf entry t channel timestamp): likewise
    (channel-timestamps): new class; transparently translates elements
    to `timestamp' instances
    (length channel-timestamps): new method; return length
    (elt channel-timestamps integer): new method; return translated
    element
    (channel-items::channel): renamed reader {channel-items
    ->
    }%channel
    (channel-items::timestamps): similar
    (shared-initialize :after channel-items t): retrieve raw channel
    timestamps
    (length channel-items): adapted to renamed reader
    (elt channel-items): likewise; translate raw timestamp to
    `timestamp' instance
  • src/view/serialized.lisp (header): updated copyright
    (make-serialized-view bag): default comparator is `<'
    (make-serialized-view sequence): likewise
    (%make-key-function channel): retrieve raw timestamp from channel
    (%make-key-function channel-timestamp): new method; for
    `channel-timestamps' class
    (%make-key-function channel-items): adapted to renamed reader
    (serialized::compare): changed initform to `<'
  • src/backend/util.lisp (header): updated copyright
    (uint64->timestamp): new function; moved here from
    src/backend/tidelog/util.lisp
    (timestamp->uint64): likewise
  • src/backend/package.lisp (package rsbag.backend): added exported
    symbols uint64->timestamp and timestamp->uint64
  • src/backend/tidelog/util.lisp (uint64->timestamp): removed; moved to
    src/backend/util.lisp
    (timestamp->uint64): likewise
  • src/backend/tidelog/index.lisp (timestamps): changed documentation
    string w.r.t. representation of timestamps
    (elt timestamps integer): do not translate timestamp
    (index-timestamp->offset input-index integer): changed specializer
    `timestamp' -> `integer'
  • src/backend/tidelog/file.lisp (header): updated copyright
    (put-entry file integer integer simple-array): changed specializer
    `timestamp' -> `integer'; do not translate timestamp
  • src/backend/elan/types.lisp (header): updated copyright
    (timestamp/nanoseconds): new type; timestamp in nanoseconds
    (time-slot/cons): changed timestamp/{milliseconds -> nanoseconds}
  • src/backend/elan/util.lisp (header): updated copyright
    (milliseconds->nanoseconds): new function; conversion
    (nanoseconds->milliseconds): likewise
    (nanoseconds->timestamp): likewise
    (timestamp->nanoseconds): likewise
    (millisecs->timestamp): removed; no longer needed
    (timestamp->millisecs): likewise
  • src/backend/elan/xml.lisp (xml-> element eql time-slot/cons):
    convert time value to nanoseconds
    (->xml cons element eql time-slot/cons): corresponding
  • src/backend/elan/file.lisp (shared-initialize :after file t):
    convert base timestamp to nanoseconds when resolving time values
    (close file): converter base timestamp from nanoseconds
    (get-timestamps file integer): do not translate returned timestamps
    (put-entry file integer integer string): changed specializer
    `timestamp' -> `integer'; do not translate timestamp
  • test/mock-backend.lisp (put-entry mock-backend integer integer t):
    changed specializer timestamp -> integer
    (simple-channels): generate uint64 timestamps
  • test/builder.lisp (header): updated copyright
    (replace-timestamps): new function; helper for
    `check-un-build-calls'; replace `timestamp' instance with integers
    for comparison
    (check-un-build-calls): use `replace-timestamps'
    (test rsbag-builder-root::bag/smoke): changed timestamps in mock
    data from `timestamp' instances to integers; timestamp replacement
    for comparison
    (test rsbag-builder-root::channel/smoke): likewise
  • test/backend/tidelog/file.lisp
    (test backend-tidelog-root::repeated-timestamp): changed timestamp
    from `timestamp' instance to integer

Revision b34a5c7f
Added by J. Moringen over 3 years ago

Mention record and replay efficiency improvements in news.rst

refs #2572

  • news.rst (RSBag 0.17): mention more efficient recording and replay

History

#1 Updated by J. Moringen almost 5 years ago

  • % Done changed from 0 to 70

Next up: make timestamp comparisons cheaper (even though we have much fewer already)

#2 Updated by J. Moringen over 4 years ago

  • Target version changed from rsb-0.15 to rsb-0.16

#3 Updated by J. Moringen about 4 years ago

  • Target version changed from rsb-0.16 to rsb-0.17

#4 Updated by J. Moringen over 3 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 70 to 100

Also available in: Atom PDF