Measuring memory footprint

[maltempi]

Hi!

I am executing some tests with cusz API and measuring the memory footprint and I found something interesting and I'd like to share with the community -- and I'd be more than happy to hear your thoughts about that.

After compressing and decompressing, cusz seems leaving a residue of allocated memory. In my code example, I'm doing 10 iterations of compression and decompression on the same input.

In the chart below, we can see a peak of memory usage during compression and decompression (as expected). After decompression, cusz frees memory and my client code example also frees all allocated memory and the memory consumption decreases. However, notice that memory consumption does not get back to the original state and for each iteration, it leaves a little of memory. I'd say it might suggest a memory leak.

Thank you very much!

[jtian0]

@maltempi Thank you for pointing out the issue. I also noticed the slow-down on the first execution, and the potential violation seems caused by thrust, indicated by compute-sanitizer (while the predecessor cuda-memcheck does not indicate anything, maybe due to driver limitation). The fix will be largely related to another memory issue #70 (memory use too high, etc.).

The codebase is organized as follows.

• high-level API (the only released API currently)
• modules/components that manage memory
• kernels and their configurations
• kernel subroutines

The memory-managing modules are the problem, obviously. Currently, predictor, Huffman encoder, and outliers are 3 classes that handle memory separately with certain assumptions corresponding to the input size. As of now, it is no longer needed as the improved algorithm can simplify the intermediate data description. Hence, by removing the OOD redundancy in modules, this issue can be resolved (I am working on this now).

The update/fix will first be pushed to https://github.com/szcompressor/cuSZ/tree/canary when ready.

[saleh1204]

Hello,

Has this issue been solved ?
I am trying to use cuSZ in a big application where I need to compress and decompress multiple arrays, but the application crashes after sometime due to out of device memory error.

Thanks

[maltempi]

Hi @jtian0 ,

We want to start exploring new compressors for our research work, do you know if the "memory leak" issue still happens?

Thank you in advance!

--

Edit

I added an endless loop into example/src/demo_capi.cu_hip.cc and in a few seconds it breaks due OOM.

  while ( true ) {
    f<float>(std::string(argv[1]));
  }

[jtian0]

@maltempi I am sorry for the delay. The issue has been initially resolved in c3c3a74.

I quickly tested the command line locally, using compute-sanitizer --show-backtrace yes --leak-check full [command line], and by far, there is no leak from it. Still, I need some more time to test the API call, like the way you mentioned above. (I will leave the issue open for now).

Previously, the C wrapper caused the issue, but the underlying memory management is not robust, either.

[jtian0]

@maltempi My colleague had a conversation with your team during SC '23 regarding the compression research, which I missed. Afterward, I could not find your contact information on the SC website or your lab's website. Can you please send an email to tian.jn09+dev AAATTT gmail.com at your convenience? Thank you.