Memory issue on full cache flush

[kitzberger]

We've noticed that running bin/typo3 cache:flush sometimes runs into memory exhaustion issues.

I'm not sure if the problem is with our TYPO3 instance or if it's an issue with this extension.

In https://github.com/b13/proxycachemanager/blob/master/Classes/Cache/Backend/ReverseProxyCacheBackend.php#L128-L136 all cached URLs are being fetched from table cache_tx_proxy:

        $conn = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable($this->cacheTable);
        $stmt = $conn->select(['content'], $this->cacheTable);
        while ($url = $stmt->fetchOne()) {
            $urls[] = $url;
        }

Question 1: does it really make sense to fetch them one by one? That's one hella DB queries, isn't it?

This big fat array of URLs is then being handed over to $this->reverseProxyProvider->flushAllUrls($urls);.

Question 2: would it be better to implement chunking on $urls and iterating over those chunks on your extension already or rather on our custom CurlHttpProxyProvider?

Looking forward to hearing back from you guys!

[ischmittis]

+1 for Chunking especially for big installations when Num of URLs exceeds 10 k(Num of URLs = Number of pages * Number of Languages)

[bmack]

Hey, fetching 10K DB records should be a no-brainer in 2025 - both in terms of performance and memory. Although I agree this is suboptimal in terms of coding, I haven't found any perf issues here

The main problem we have / had a lot is the amount of time it takes to do the Invalidation on the Edge Servers. A few numbers from the top of my head from my client installations:

• Microsoft Azure CDN -> roughly 7 Minutes for a full cache flush
• Amazon CloudFront -> flush:all ~ 30 seconds in our projects, as we need to flush 60 zones.
• Akamai - ~ 10 seconds.

For CloudFront, we've implemented grouping and chunking on the client. Example: We have 100 URLs that all start with www.example.com/en/products/dish-washers/ (and then /dish-washer-1/ etc) - then it's actually important we can do this grouping on the PHP side with all "to be flushed" URLs. We then send one API Request to invalidate www.example.com/en/products/dish-washers/* (also to save money, as they charge per invalidation).

For some of our custom providers, we've implemented a queue worker now to flush the caches in the background. this might be a solution for the future in general: "deferred cache flush" which does the cache flush in a separate process in general. Especially when editors publish something within a workspace, the UI is frozen (= due to the CDN API HTTP requests).