The issue was compaction back-off. At some point of time, the LSM tree would have got completely filled up and would not be able to expand because of no compaction.
The compaction back-off mechanism was already removed in the badger pointed by 20.11. We have also removed it from the master.