I have a large object that I want to save to the disk because it takes a minute to generate. The OOM reaper kills the process while pickle.dump ing the object.
It’s a tuple of dicts of tuple of array.array.
Can pickle dump in chunks? If not, is there another technique I can use?
A colleague of mine replaced the internal store format of pickle with Json iirc. It was like 10x faster. Not exactly sure how you do that, but I can check if you want to go that way.