Regression: "Out of host memory" on Nvidia ICD

[inducer]

• 1593a67 (just before the PytatoPyOpenCLArrayContext: use SVM allocator if available, limit arg size for GPUs #189) is known good
• 167ae7c is broken

Here's a log of the CI failure: https://gitlab.tiker.net/inducer/arraycontext/-/jobs/462367

Host memory grows to 6-ish gigabytes when running this by hand and then keels over. If run with --sw, this fails as before, but it succeeds upon resuming when rerun. I suspect there's a memory leak here somewhere.

@matthiasdiener Could you take a look?

[matthiasdiener]

I can reproduce it on bock (which has a Titan V), but interestingly, the tests run fine on koelsch with a K40.

This code is helpful when debugging this:

@pytest.fixture(autouse=True)
def slow_down_tests():
    yield
    import time
    time.sleep(3)

[matthiasdiener]

It looks like Nvidia CL on bock does not like creating multiple CL contexts with SVM allocations. The following program quickly fills up the GPU's memory on Nvidia CL, but not with pocl-cuda or pocl-pthreads:

import pyopencl as cl
import time
from pyopencl.tools import SVMAllocator, ImmediateAllocator


while True:
    ctx = cl.create_some_context() # putting this before the loop is fine
    queue = cl.CommandQueue(ctx)
    allocator = SVMAllocator(queue.context, queue=queue)
    # allocator = ImmediateAllocator(queue)  # using normal allocations is fine
    x = allocator(2*1024*1024*1024)
    # x.release()
    x.enqueue_release()
    queue.finish()
    del queue
    del ctx

    # every iteration of this loop increases memory usage on the GPU by ~300 MByte (with Nvidia CL + SVM allocs)
    time.sleep(2)

[inducer]

Thanks for digging into this! That's helpful as an observation. I'll look at a CL call trace to see if that looks OK, and if so... not sure that we can do much?

[inducer]

I'm not sure I like this solution, but I don't see an alternative: Could you do a PR to turn off use of SVM on Nvidia CL, with a pointer to this issue?

I've put together https://gitlab.tiker.net/inducer/arraycontext/-/merge_requests/6 to make sure we maintain at least some coverage of the SVM code paths.