Thanks for the responses.
Why I was talking about hash map: Basically I was worried about changes that are not yet committed to RocksDB, and only exist in mutation layers. We could force a full commit (aggressive evict) after a certain time period, even if there is a lot of memory left.
PlainTableFormat: Yes,I don’t really want to go there yet.
Combining pagination with sorting: I think one question is when we sort, do we sort each PL in the UIDMatrix? Or do we sort the merged results? When there is a lot of overlap between PLs in the UIDMatrix, sorting each PL can be wasteful. When there is little overlap, sorting the combined list is actually not too bad. I am leaning towards sorting the merged results, but that would complicate pagination optimization. Let’s discuss this later.