jarifibrahim commented :
Hey @mvanderkroon, thank you for the detailed write-up. The Postgres article was definitely helpful. The transaction limits are set based on the max table size
The transaction can be at most 15% of the table size. The default size of the table is 64 MB so the max batch size would be about 9.6 MB. I don’t know why we have a 15% limit. It would be worth experimenting with some other numbers (maybe 50%?). You might be able to work around the transaction size by setting a high value for max table size.
Fixing the transaction too big issue isn’t easy. Postgres stores the uncommitted transactions separately and then adds them to the main WAL/BTree. A similar change in badger would be a big change and it might take a decent amount of time to build it.
Whether it is perceived as a problem at all
I believe a lot of users might be facing this problem and we should fix this.
Is a solution to this problem part of the future of BadgerDB
Yes, we’re always working on improving badger. This seems like a useful improvement. I’ll add this to our backlog.
If the above answers are both yes, how should it be approached
I believe the first step would be to study how postgres/mysql/rocksdb deal with this problem and then a detailed write-up about how we could tackle the problem in badger will be a good starting point.
Is there anything I could do to help
Badger is open source and we encourage people to contribute. Anything you can contribute to improving badger/fixing transaction too big issue will be helpful for all the contributors 