-
Notifications
You must be signed in to change notification settings - Fork 52
Open
Description
We're seeing continuous memory growth when making repeated API calls. Memory is not released even after:
- Deleting the response (
del resp) - Calling
gc.collect() - Calling
client.close()
Only fully recreating the ApiClient releases the accumulated memory.
Observed behavior
- ~35-40 MB growth per large API call (timeseries queries)
- Using
objgraph, we see model objects likeTimeseriesResponseSeriesbeing retained - Eventually leads to OOM in long-running processes
Current workaround
Periodically recreating the client.
Possible cause
Looking at model_utils.py, the cached_property implementation caches on the descriptor (class-level) rather than the instance, which would prevent GC.
Minimal reproduction
import gc
import os
from datetime import datetime, timedelta
import psutil
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.metrics_api import MetricsApi
from datadog_api_client.v2.model.metrics_data_source import MetricsDataSource
from datadog_api_client.v2.model.metrics_timeseries_query import MetricsTimeseriesQuery
from datadog_api_client.v2.model.query_formula import QueryFormula
from datadog_api_client.v2.model.timeseries_formula_query_request import TimeseriesFormulaQueryRequest
from datadog_api_client.v2.model.timeseries_formula_request import TimeseriesFormulaRequest
from datadog_api_client.v2.model.timeseries_formula_request_attributes import TimeseriesFormulaRequestAttributes
from datadog_api_client.v2.model.timeseries_formula_request_queries import TimeseriesFormulaRequestQueries
from datadog_api_client.v2.model.timeseries_formula_request_type import TimeseriesFormulaRequestType
def get_memory_mb():
return psutil.Process().memory_info().rss / 1024 / 1024
config = Configuration()
config.api_key["apiKeyAuth"] = os.environ["DD_API_KEY"]
config.api_key["appKeyAuth"] = os.environ["DD_APP_KEY"]
if os.environ.get("DD_SITE"):
config.server_variables["site"] = os.environ["DD_SITE"]
client = ApiClient(config)
api = MetricsApi(client)
baseline = get_memory_mb()
print(f"Baseline: {baseline:.1f} MB")
# Use a query that returns many series to make leak visible
QUERY = "avg:kubernetes.cpu.requests{*} by {kube_cluster_name,kube_namespace,kube_deployment}"
for i in range(10):
now = datetime.now()
body = TimeseriesFormulaQueryRequest(
data=TimeseriesFormulaRequest(
attributes=TimeseriesFormulaRequestAttributes(
_from=int((now - timedelta(hours=1)).timestamp() * 1000),
to=int(now.timestamp() * 1000),
interval=60000,
queries=TimeseriesFormulaRequestQueries([
MetricsTimeseriesQuery(
data_source=MetricsDataSource.METRICS,
query=QUERY,
name="a"
)
]),
formulas=[QueryFormula(formula="a")]
),
type=TimeseriesFormulaRequestType.TIMESERIES_REQUEST
)
)
resp = api.query_timeseries_data(body=body)
del resp
gc.collect()
mem = get_memory_mb()
print(f"Call {i+1}: {mem:.1f} MB (+{mem - baseline:.1f})")
client.close()
gc.collect()
print(f"After close(): {get_memory_mb():.1f} MB")Output
Baseline: 27.9 MB
Call 1: 122.6 MB (+94.7)
Call 2: 189.3 MB (+161.4)
Call 3: 256.6 MB (+228.7)
Call 4: 323.9 MB (+296.0)
Call 5: 391.0 MB (+363.1)
Call 6: 458.1 MB (+430.2)
Call 7: 525.3 MB (+497.4)
Call 8: 592.5 MB (+564.6)
Call 9: 659.7 MB (+631.8)
Call 10: 726.8 MB (+698.9)
After close(): 726.8 MB
Memory grows ~67 MB per call. close() doesn't release it. Only recreating the ApiClient releases the accumulated memory.
Environment
- datadog-api-client: 2.48.0
- Python: 3.12
- OS: Linux
Metadata
Metadata
Assignees
Labels
No labels