mvanderkroon commented :
@jarifibrahim thanks for the equally detailed response. I’m happy to learn this is something that could contribute to BadgerDB.
Thank you for the pointer towards the max table size, I looked into the mechanics and ran a few tests, I believe this would help extending the number of operations before receiving a ErrTxnTooBig (essentially allowing for a dial to regulate the transaction size where atomicity is guaranteed), but it would still be in RAM. For our use case this latter part is problematic; users may send variable sized payloads to the service and thus it’s difficult to dimension the hardware appropriately.
I am, however, working on a “poor-man’s’”-solution: essentially a layer on top of BadgerDBs’ transactional layer. By assigning ‘blocks’ of data unique ID’s (versions) and making those part of the keys, I can stream data to the database, committing whenever I hit ErrTxnTooBig. Once the process is done, I switch over the ‘active block ID’ to the latest inserted and initiate a job that deletes the previous ‘version’. Now, this is by no means a ‘good’ solution that I would present back to this repository (for one, it feels like I’m reimplementing logic that is already present in some form or another) - but for our (narrow) use case it will do fine. At the same time, this process will allow me to gain more knowledge and understanding of BadgerDB as well as an intuition for a proper way forward, I’ll circle back once I reach this point.