Bug #852
Heap-Exhausted error when reading corrupt TIDELog files
Status: | Closed | Start date: | 02/03/2012 | ||
---|---|---|---|---|---|
Priority: | Low | Due date: | |||
Assignee: | J. Moringen | % Done: | 20% | ||
Category: | - | ||||
Target version: | Robotics Service Bus - rsb-0.13 |
Description
The purpose of this issue is documenting the current situation. There is no immediate fix except improving error messages.
When reading corrupt TIDELog files, several kinds of seemingly unrelated errors can occur. This happens when an incorrect size value is read from a file and used in an allocation.
All RSBag tools which read files are affected (at the time of writing bag-cat
, bag-play
, bag-merge
).
On 32-bit and 64-bit platforms, the problem can look like this:
Heap exhausted during allocation: 815529984 bytes available, 1818326592 requested. Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB LUB !move Alloc Waste Trig WP GCs Mem-age 0: 3579 0 5239 0 25 0 912 0 0 30521264 182352 53687091 0 0 0.0000 1: 0 0 0 0 0 0 0 0 0 0 0 53687091 0 0 0.0000 2: 4217 4218 0 0 305 29 0 338 109 21846960 173136 2000000 268 0 0.0000 3: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000 4: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000 5: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000 6: 0 0 0 0 2686 789 0 0 0 113868800 0 2000000 2311 0 0.0000 Total bytes allocated = 166237024 Dynamic-space-size bytes = 1073741824 GC control variables: *GC-INHIBIT* = false *GC-PENDING* = false *STOP-FOR-GC-PENDING* = false Heap exhausted (no more space for allocation). There are still 815529984 bytes available; the request was for 1818326592 bytes. PROCEED WITH CAUTION.
On 32-bit platforms (and very unlikely on 64-bit platforms) the problem can also look like this:
The value 1818326576 is not of type (UNSIGNED-BYTE 29).
Associated revisions
Decrease memory pressure in src/backend/tidelog/file.lisp
- src/backend/tidelog/file.lisp (header): added one-line summary;
updated copyright
(write-buffer file chnk): after the buffer has been written
overwrite references to chunk entries to allow earlier garbage
collection; on SBCL, do garbage collection afterwards
Improved error recovery in src/backend/tidelog/*.lisp
refs #852
Most importantly, INDX blocks are now located when scanning, but not
unpacked immediately, reducing peak memory use. Similarly when
reconstructing indices.
- src/backend/tidelog/io.lisp (header): updated copyright
(scan stream eql :tide): do not unpack INDX block; return their
offsets instead
(scan stream eql :block): adapted to above change - src/backend/tidelog/index.lisp (index-add-indxs index sequence list):
changed to method
(index-add-entries index sequence vector): new method; add entries - src/backend/tidelog/file.lisp (shared-initialize :after file t):
unpack INDX blocks sequentially instead of all at the time; check
channels vs. indicies both ways; pass a function for ensure index
existence to `reconstruct-indices'
(put-channel file integer string vector): adapted to `make-index'
lambda-list changes
(make-index): do not accept indices and chunks; these are added
incrementally later
(%chunk-id->offset): cosmetic changes - src/backend/tidelog/repair.lisp (header): updated copyright
(reconstruct-indices): temporarily collect indix data into
`index-entry' instances, but flush these into the proper `index'
structures periodically using `index-add-entries' - test/backend/tidelog/repair.lisp (header): updated copyright
(mock-index): new class; mock for index reconstruction test
(index-add-entries mock-index sequence vector): new method; similar
(test backend-tidelog-root::reconstruct-indices/smoke): adapted to
changes; use `mock-index' - test/backend/tidelog/package.lisp (header): updated copyright
(package rsbag.backend.tidelog.test): added imported-from
rsbag.backend.tidelog symbol index-add-entries
History
#1 Updated by J. Moringen about 11 years ago
- Target version set to rsb-0.7
#2 Updated by J. Moringen almost 11 years ago
- Target version changed from rsb-0.7 to rsb-0.9
#3 Updated by J. Moringen almost 10 years ago
- Target version changed from rsb-0.9 to rsb-0.10
#4 Updated by J. Moringen over 9 years ago
- Target version changed from rsb-0.10 to rsb-0.11
#5 Updated by J. Moringen about 9 years ago
- Target version changed from rsb-0.11 to rsb-0.12
#6 Updated by J. Moringen almost 8 years ago
- Target version changed from rsb-0.12 to rsb-1.0
#7 Updated by J. Moringen over 7 years ago
- Status changed from New to Closed
This is much more robust / less memory hungry now.
#8 Updated by J. Moringen over 6 years ago
- Target version changed from rsb-1.0 to rsb-0.13