feat: Pipelines enters public preview #2272

wu-hui · 2025-12-12T15:54:31Z

No description provided.

setup nix for idx

…uandy/PipelineResult

pipeline result

tests

* add Java snippets * Add input and expression snippets * add query explain snippet

generated-files-bot · 2025-12-12T15:54:38Z

Warning: This pull request is touching the following templated files:

google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/stub/FirestoreStubSettings.java

MarkDuckworth

PR title will be in the change log, so be sure to update this

MarkDuckworth

Initial review. It looks like there are a few API changes that need to be ported from Android

MarkDuckworth · 2025-12-19T20:46:59Z

google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java

+   * documents, execution time, and explain stats.
+   */
+  @BetaApi
+  public static final class Snapshot {


Same question on this SDK about Pipeline.Snapshot vs PipelineSnapshot

Replied from android review.

MarkDuckworth · 2025-12-19T20:52:39Z

google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java

+   * @return A new Pipeline object with this stage appended to the stage list.
+   */
+  @BetaApi
+  public Pipeline addFields(Selectable... fields) {


This does not have the same signature as Android: fun addFields(field: Selectable, vararg additionalFields: Selectable) ...

It looks like this is true throughout the API for adding stages

MarkDuckworth · 2025-12-19T20:56:53Z

google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java

+  // @BetaApi
+  // public Pipeline unnest(Selectable field) {
+  //   return append(new Unnest(field));
+  // }


Should this API be added for feature parity?

This one was a duplicate of below, deleted it instead.

MarkDuckworth · 2025-12-19T20:57:36Z

google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java

+  // @BetaApi
+  // public Pipeline unnest(Selectable field, UnnestOptions options) {
+  //   return append(new Unnest(field, options));
+  // }


same, should this be uncommented

MarkDuckworth · 2025-12-19T21:13:29Z

google-cloud-firestore/src/main/java/com/google/cloud/firestore/PipelineResult.java

@@ -0,0 +1,456 @@
+/*
+ * Copyright 2017 Google LLC


update date

MarkDuckworth · 2025-12-19T21:38:02Z

...restore/src/main/java/com/google/cloud/firestore/pipeline/expressions/AggregateFunction.java

Eventually we also want to javadoc the apis in this file

MarkDuckworth · 2025-12-19T21:38:48Z

...irestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/AliasedAggregate.java

eventual javadoc

I think this might be just internal.

MarkDuckworth · 2025-12-19T21:39:04Z

...restore/src/main/java/com/google/cloud/firestore/pipeline/expressions/AliasedExpression.java

eventual javadoc

MarkDuckworth · 2025-12-19T21:41:22Z

...restore/src/main/java/com/google/cloud/firestore/pipeline/expressions/BooleanExpression.java

I think we need to port BooleanExpression changes from Android. same probably for nodejs-firestore

Yeah, we should.

MarkDuckworth · 2025-12-19T21:44:02Z

...loud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java

+   * @return A new {@link Expression} representing the generic function.
+   */
+  @BetaApi
+  public static Expression generic(String name, Expression... expr) {


Rename to rawExpression

MarkDuckworth · 2025-12-19T22:39:17Z

...cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/AggregateHints.java

+  }
+
+  public AggregateHints withForceStreamableEnabled() {
+    return with("force_streamable", true);


I'm not familiar with this hint, are we sure this is current?

force_streamable is not listed

MarkDuckworth · 2025-12-19T22:41:45Z

...loud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/CollectionHints.java

+  }
+
+  public CollectionHints withIgnoreIndexFields(String... values) {
+    return with("ignore_index_fields", values);


This is also no longer listed, that i'm aware of

MarkDuckworth

Another round of comments on the remaining files. I think we need to determine what to do with the differences in the API before release

MarkDuckworth · 2025-12-22T20:07:22Z

samples/preview-snippets/pom.xml

+        <groupId>com.google.cloud</groupId>
+        <artifactId>google-cloud-firestore-bom</artifactId>
+        <!-- Tell the application to use the private preview BOM of cloud SDK -->
+        <version>99.99.0-PRIVATEPREVIEW</version>


Does this version need to be updated for the release?

MarkDuckworth · 2025-12-22T20:08:40Z

samples/preview-snippets/src/main/java/com/example/firestore/PipelineSnippets.java

+    this.firestore = firestore;
+  }
+
+  void queryExplain() throws ExecutionException, InterruptedException {


This snippet does not appear related to pipelines

Not sure, this is Morgan's change, i suppose it does not matter.

MarkDuckworth · 2025-12-22T20:31:52Z

tools/api.txt

-    method @NonNull public static com.google.cloud.firestore.AggregateField.AverageAggregateField average(@NonNull com.google.cloud.firestore.FieldPath);
-    method @NonNull public static com.google.cloud.firestore.AggregateField.AverageAggregateField average(@NonNull String);
-    method @NonNull public static com.google.cloud.firestore.AggregateField.CountAggregateField count();
+    method public static com.google.cloud.firestore.AggregateField.AverageAggregateField average(com.google.cloud.firestore.FieldPath);


I'm a little concerned about @nonnull attribute being dropped throughout the API. I can't determine if this is only dropped in this api.txt, or if there was some config / source change. If it was actually dropped from the API surface, that would be a breaking change.

Fixed. This api.txt is just for convenience to conduct api audit using gemini, it is not used anywhere else.

MarkDuckworth · 2025-12-22T20:46:57Z

...loud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java

+   * @return An {@link Expression} constant instance.
+   */
+  @BetaApi
+  public static Expression vector(VectorValue value) {


I'm pretty sure we dropped this way to create a constant from a vector, because it is also listed above Expression constant(VectorValue vector)

MarkDuckworth · 2025-12-22T20:59:25Z

...loud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java

+   * @return An {@link Expression} constant instance.
+   */
+  @BetaApi
+  public static Expression vector(double[] value) {


I think we dropped both overloads of vector(x) to create a constant

wu-hui and others added 30 commits February 14, 2024 15:52

Pipeline operations AST

658c952

Getting ready for fluent api

4ab3382

Basic fluent

a3d7234

Joins too

a837f62

Some fixups, and group bys

c8e7854

joins but nicer

c5a01a6

address feedback

191eb44

pagination and others

27f2748

minor adds

55ace39

adding more sugar to improve brevity

56c3a41

address some feedback

a5a90ab

support fromData

424ac43

pipeline result basic

4d8a205

execute

9119b95

fixing compilation error and add some high level comments

3aa73de

setup nix for idx

b583791

Merge pull request #3 from wu-hui/wuandy/IdxSetup

670ae65

setup nix for idx

tweaks

7216745

pipeline result basic

c46601f

execute

99e9067

fixing compilation error and add some high level comments

f20e409

tweaks

742d3c4

Merge remote-tracking branch 'pipelines/wuandy/PipelineResult' into w…

637d494

…uandy/PipelineResult

fix errors

7c67e99

fix errors

ba521a0

Add function composition example.

6ae7b13

build into a single jar

c1825e0

better aliasing for joins

0fe5faa

Merge pull request #2 from wu-hui/wuandy/PipelineResult

0d52712

pipeline result

add stuff to support challenges

7ab3099

wu-hui and others added 4 commits September 5, 2025 15:26

reformat and add explain suppor and new api.txt and query to pipeline

b832d08

tests

API refinement

ccaf9d4

Update version for private preview

742fab6

feat: Add pipeline snippets (#2252)

4d41c82

* add Java snippets * Add input and expression snippets * add query explain snippet

wu-hui requested review from a team as code owners December 12, 2025 15:54

product-auto-label bot added the size: xl Pull request size is extra large. label Dec 12, 2025

product-auto-label bot added the api: firestore Issues related to the googleapis/java-firestore API. label Dec 12, 2025

wu-hui force-pushed the feat/pipeline/private-preview branch 3 times, most recently from 3aece03 to bd317e2 Compare December 12, 2025 20:59

Fix CI against standard

426376c

wu-hui force-pushed the feat/pipeline/private-preview branch from f5e464b to 651bd3d Compare December 12, 2025 21:09

Merge with main

6ae12ec

wu-hui force-pushed the feat/pipeline/private-preview branch from 651bd3d to 6ae12ec Compare December 12, 2025 21:10

chore: generate libraries at Fri Dec 12 21:11:14 UTC 2025

320b026

MarkDuckworth reviewed Dec 19, 2025

View reviewed changes

MarkDuckworth reviewed Dec 22, 2025

View reviewed changes

wu-hui changed the title ~~feat: ppl pri-pre~~ feat: Pipelines enters beta Jan 2, 2026

wu-hui changed the title ~~feat: Pipelines enters beta~~ feat: Pipelines enters public preview Jan 2, 2026

wu-hui added 3 commits January 2, 2026 15:36

Address comments

6a45c2f

merge with main

f69e18f

Merge with old feature branch

7410c61

wu-hui assigned MarkDuckworth Jan 2, 2026

chore: generate libraries at Fri Jan 2 21:02:15 UTC 2026

6094ee2

product-auto-label bot added size: m Pull request size is medium. and removed size: xl Pull request size is extra large. labels Jan 2, 2026

feat: Pipelines enters public preview #2272

Are you sure you want to change the base?

feat: Pipelines enters public preview #2272

Conversation

wu-hui commented Dec 12, 2025

Uh oh!

generated-files-bot bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarkDuckworth left a comment

Choose a reason for hiding this comment

Uh oh!

MarkDuckworth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarkDuckworth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

generated-files-bot bot commented Dec 12, 2025 •

edited

Loading