Investigate memory leak during first epoch

[ryanleary]

Sortagrad appears to be a pathological case for memory allocator (also possibly may be related to interactions with warp-ctc memory allocations) that causes OOM errors often very late in an epoch.