feat(services): add configurable batch size with file container rotation

CONFIGURATION.md

@@ -0,0 +1,169 @@
+# Configuration Guide
+
+This document describes the available configuration options for the Coveo Push API Java Client.
+
+## Batch Size Configuration
+
+The batch size controls how much data is accumulated before creating a file container and pushing to Coveo. The default is **5 MB**. The maximum allowed is **256 MB** (Stream API limit).
+
+### Configuration Methods
+
+There are two ways to configure the batch size:
+
+#### 1. System Property (Runtime Configuration)
+
+Set the `coveo.push.batchSize` system property to configure the default batch size globally for all service instances:
+
+**Java Command Line:**
+
+```bash
+java -Dcoveo.push.batchSize=134217728 -jar your-application.jar
+```
+
+**Within Java Code:**
+
+```java
+// Set before creating any service instances
+System.setProperty("coveo.push.batchSize", "134217728"); // 128 MB in bytes
+```
+
+**Maven/Gradle Build:**
+
+```xml
+<!-- pom.xml -->
+<properties>
+    <argLine>-Dcoveo.push.batchSize=134217728</argLine>
+</properties>
+```
+
+```groovy
+// build.gradle
+test {
+    systemProperty 'coveo.push.batchSize', '134217728'
+}
+```
+
+**Example Values:**
+
+- `5242880` = 5 MB (default)
+- `268435456` = 256 MB (maximum)
+- `134217728` = 128 MB
+- `67108864` = 64 MB
+- `33554432` = 32 MB
+- `10485760` = 10 MB
+
+#### 2. Constructor Parameter (Per-Instance Configuration)
+
+Pass the `maxQueueSize` parameter when creating service instances:
+
+```java
+// UpdateStreamService with custom 128 MB batch size
+UpdateStreamService service = new UpdateStreamService(
+    catalogSource,
+    backoffOptions,
+    null,  // userAgents (optional)
+    128 * 1024 * 1024  // 128 MB in bytes
+);
+
+// PushService with custom batch size
+PushService pushService = new PushService(
+    pushEnabledSource,
+    backoffOptions,
+    128 * 1024 * 1024  // 128 MB
+);
+
+// StreamService with custom batch size
+StreamService streamService = new StreamService(
+    streamEnabledSource,
+    backoffOptions,
+    null,  // userAgents (optional)
+    128 * 1024 * 1024  // 128 MB
+);
+```
+
+### Configuration Priority
+
+When both methods are used:
+
+1. **Constructor parameter** takes precedence (if specified)
+2. **System property** is used as default (if set)
+3. **Built-in default** of 5 MB is used otherwise
+
+### Validation Rules
+
+All batch size values are validated:
+
+- ✅ **Maximum:** 256 MB (268,435,456 bytes) - API limit
+- ✅ **Minimum:** Greater than 0
+- ❌ Values exceeding 256 MB will throw `IllegalArgumentException`
+- ❌ Invalid or negative values will throw `IllegalArgumentException`
+
+### Examples
+
+#### Example 1: Using System Property
+
+```java
+// Configure globally via system property
+System.setProperty("coveo.push.batchSize", "134217728"); // 128 MB
+
+// All services will use 128 MB by default
+UpdateStreamService updateService = new UpdateStreamService(catalogSource, backoffOptions);
+PushService pushService = new PushService(pushEnabledSource, backoffOptions);
+StreamService streamService = new StreamService(streamEnabledSource, backoffOptions);
+```
+
+#### Example 2: Override Per Service
+
+```java
+// Set global default to 128 MB
+System.setProperty("coveo.push.batchSize", "134217728");
+
+// Update service uses global default (128 MB)
+UpdateStreamService updateService = new UpdateStreamService(catalogSource, backoffOptions);
+
+// Push service overrides with 64 MB
+PushService pushService = new PushService(pushEnabledSource, backoffOptions, 64 * 1024 * 1024);
+
+// Stream service uses global default (128 MB)
+StreamService streamService = new StreamService(streamEnabledSource, backoffOptions);
+```
+
+### When to Adjust Batch Size
+
+**Use smaller batches (32-64 MB) when:**
+
+- Network bandwidth is limited
+- Memory is constrained
+- Processing many small documents
+- You want more frequent progress updates
+
+**Use larger batches (128-256 MB) when:**
+
+- Network bandwidth is high
+- Processing large documents or files
+- You want to minimize API calls
+- Maximum throughput is needed
+
+**Keep default (5 MB) when:**
+
+- You're unsure
+- Memory is a concern
+- You want predictable, frequent pushes
+
+### Configuration Property Reference
+
+| Property Name          | Description                 | Default Value    | Valid Range    |
+| ---------------------- | --------------------------- | ---------------- | -------------- |
+| `coveo.push.batchSize` | Default batch size in bytes | `5242880` (5 MB) | 1 to 268435456 |
+
+## Additional Configuration
+
+### Environment Variables
+
+The following environment variables can be used for general configuration:
+
+- `COVEO_API_KEY` - API key for authentication
+- `COVEO_ORGANIZATION_ID` - Organization identifier
+- `COVEO_PLATFORM_URL` - Custom platform URL (if needed)
+
+Refer to the Coveo Platform documentation for complete environment configuration options.

README.md

@@ -200,6 +200,32 @@ public class PushOneDocument {
 }
 ```
 
+## Configuration
+
+### Batch Size Configuration
+
+The SDK uses a default batch size of **5 MB** before automatically creating a file container and pushing documents. The maximum allowed batch size is **256 MB** (matching the Coveo Stream API limit). You can configure this globally via system property or per-service via constructor.
+
+**Global Configuration (System Property):**
+```bash
+java -Dcoveo.push.batchSize=52428800 -jar your-app.jar  # 50 MB (in bytes)
+```
+
+**Per-Service Configuration:**
+```java
+// Configure PushService with 50 MB batch size
+PushService service = new PushService(
+    source, 
+    backoffOptions,
+    50 * 1024 * 1024  // 50 MB in bytes
+);
+    null,  // userAgents (optional)
+    128 * 1024 * 1024  // 128 MB in bytes
+);
+```
+
+See **[CONFIGURATION.md](CONFIGURATION.md)** for complete configuration options examples, and best practices.
+
 ### Exponential Backoff Retry Configuration
 
 By default, the SDK leverages an exponential backoff retry mechanism. Exponential backoff allows for the SDK to make multiple attempts to resolve throttled requests, increasing the amount of time to wait for each subsequent attempt. Outgoing requests will retry when a `429` status code is returned from the platform.

samples/ConfigureBatchSize.java

@@ -0,0 +1,45 @@
+import com.coveo.pushapiclient.*;
+
+import java.io.IOException;
+
+/**
+ * Demonstrates how to configure the batch size for document uploads.
+ *
+ * The batch size controls how much data accumulates before automatically
+ * creating a file container and pushing to Coveo. Default is 5 MB, max is 256 MB.
+ */
+public class ConfigureBatchSize {
+
+    public static void main(String[] args) throws IOException, InterruptedException {
+
+        PlatformUrl platformUrl = new PlatformUrlBuilder()
+                .withEnvironment(Environment.PRODUCTION)
+                .withRegion(Region.US)
+                .build();
+
+        CatalogSource catalogSource = CatalogSource.fromPlatformUrl(
+                "my_api_key", "my_org_id", "my_source_id", platformUrl);
+
+        // Option 1: Use default batch size (5 MB)
+        UpdateStreamService defaultService = new UpdateStreamService(catalogSource);
+
+        // Option 2: Configure batch size via constructor (50 MB)
+        int fiftyMegabytes = 50 * 1024 * 1024;
+        UpdateStreamService customService = new UpdateStreamService(
+                catalogSource,
+                new BackoffOptionsBuilder().build(),
+                null,
+                fiftyMegabytes);
+
+        // Option 3: Configure globally via system property (affects all services)
+        // Run with: java -Dcoveo.push.batchSize=52428800 ConfigureBatchSize
+        // This sets 50 MB for all service instances that don't specify a size
+
+        // Use the service
+        DocumentBuilder document = new DocumentBuilder("https://my.document.uri", "My document title")
+                .withData("these words will be searchable");
+
+        customService.addOrUpdate(document);
+        customService.close();
+    }
+}

src/main/java/com/coveo/pushapiclient/PushService.java

@@ -10,15 +10,36 @@ public class PushService {
   private PushServiceInternal service;
 
   public PushService(PushEnabledSource source) {
-    this(source, new BackoffOptionsBuilder().build());
+    this(source, new BackoffOptionsBuilder().build(), DocumentUploadQueue.getConfiguredBatchSize());
   }
 
   public PushService(PushEnabledSource source, BackoffOptions options) {
+    this(source, options, DocumentUploadQueue.getConfiguredBatchSize());
+  }
+
+  /**
+   * Creates a new PushService with configurable batch size.
+   *
+   * <p>Example batch sizes in bytes:
+   *
+   * <ul>
+   *   <li>5 MB (default): {@code 5 * 1024 * 1024} = {@code 5242880}
+   *   <li>50 MB: {@code 50 * 1024 * 1024} = {@code 52428800}
+   *   <li>256 MB (max): {@code 256 * 1024 * 1024} = {@code 268435456}
+   * </ul>
+   *
+   * @param source The source to push documents to.
+   * @param options The configuration options for exponential backoff.
+   * @param maxQueueSize The maximum batch size in bytes before auto-flushing (default: 5MB, max:
+   *     256MB).
+   * @throws IllegalArgumentException if maxQueueSize exceeds 256MB or is not positive.
+   */
+  public PushService(PushEnabledSource source, BackoffOptions options, int maxQueueSize) {
     String apiKey = source.getApiKey();
     String organizationId = source.getOrganizationId();
     PlatformUrl platformUrl = source.getPlatformUrl();
     UploadStrategy uploader = this.getUploadStrategy();
-    DocumentUploadQueue queue = new DocumentUploadQueue(uploader);
+    DocumentUploadQueue queue = new DocumentUploadQueue(uploader, maxQueueSize);
 
     this.platformClient = new PlatformClient(apiKey, organizationId, platformUrl, options);
     this.service = new PushServiceInternal(queue);