Skip to content

Memory not released after API calls - gc.collect() and close() ineffective #3029

@lucacavenaghi97

Description

@lucacavenaghi97

We're seeing continuous memory growth when making repeated API calls. Memory is not released even after:

  • Deleting the response (del resp)
  • Calling gc.collect()
  • Calling client.close()

Only fully recreating the ApiClient releases the accumulated memory.

Observed behavior

  • ~35-40 MB growth per large API call (timeseries queries)
  • Using objgraph, we see model objects like TimeseriesResponseSeries being retained
  • Eventually leads to OOM in long-running processes

Current workaround

Periodically recreating the client.

Possible cause

Looking at model_utils.py, the cached_property implementation caches on the descriptor (class-level) rather than the instance, which would prevent GC.

Minimal reproduction

import gc
import os
from datetime import datetime, timedelta

import psutil
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.metrics_api import MetricsApi
from datadog_api_client.v2.model.metrics_data_source import MetricsDataSource
from datadog_api_client.v2.model.metrics_timeseries_query import MetricsTimeseriesQuery
from datadog_api_client.v2.model.query_formula import QueryFormula
from datadog_api_client.v2.model.timeseries_formula_query_request import TimeseriesFormulaQueryRequest
from datadog_api_client.v2.model.timeseries_formula_request import TimeseriesFormulaRequest
from datadog_api_client.v2.model.timeseries_formula_request_attributes import TimeseriesFormulaRequestAttributes
from datadog_api_client.v2.model.timeseries_formula_request_queries import TimeseriesFormulaRequestQueries
from datadog_api_client.v2.model.timeseries_formula_request_type import TimeseriesFormulaRequestType


def get_memory_mb():
    return psutil.Process().memory_info().rss / 1024 / 1024


config = Configuration()
config.api_key["apiKeyAuth"] = os.environ["DD_API_KEY"]
config.api_key["appKeyAuth"] = os.environ["DD_APP_KEY"]
if os.environ.get("DD_SITE"):
    config.server_variables["site"] = os.environ["DD_SITE"]
client = ApiClient(config)
api = MetricsApi(client)

baseline = get_memory_mb()
print(f"Baseline: {baseline:.1f} MB")

# Use a query that returns many series to make leak visible
QUERY = "avg:kubernetes.cpu.requests{*} by {kube_cluster_name,kube_namespace,kube_deployment}"

for i in range(10):
    now = datetime.now()
    body = TimeseriesFormulaQueryRequest(
        data=TimeseriesFormulaRequest(
            attributes=TimeseriesFormulaRequestAttributes(
                _from=int((now - timedelta(hours=1)).timestamp() * 1000),
                to=int(now.timestamp() * 1000),
                interval=60000,
                queries=TimeseriesFormulaRequestQueries([
                    MetricsTimeseriesQuery(
                        data_source=MetricsDataSource.METRICS,
                        query=QUERY,
                        name="a"
                    )
                ]),
                formulas=[QueryFormula(formula="a")]
            ),
            type=TimeseriesFormulaRequestType.TIMESERIES_REQUEST
        )
    )

    resp = api.query_timeseries_data(body=body)
    del resp
    gc.collect()

    mem = get_memory_mb()
    print(f"Call {i+1}: {mem:.1f} MB (+{mem - baseline:.1f})")

client.close()
gc.collect()
print(f"After close(): {get_memory_mb():.1f} MB")

Output

Baseline: 27.9 MB

Call 1: 122.6 MB (+94.7)
Call 2: 189.3 MB (+161.4)
Call 3: 256.6 MB (+228.7)
Call 4: 323.9 MB (+296.0)
Call 5: 391.0 MB (+363.1)
Call 6: 458.1 MB (+430.2)
Call 7: 525.3 MB (+497.4)
Call 8: 592.5 MB (+564.6)
Call 9: 659.7 MB (+631.8)
Call 10: 726.8 MB (+698.9)

After close(): 726.8 MB

Memory grows ~67 MB per call. close() doesn't release it. Only recreating the ApiClient releases the accumulated memory.

Environment

  • datadog-api-client: 2.48.0
  • Python: 3.12
  • OS: Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions