feat(rust): add datafusion catalog_provider through namespace

[majin1102]

This is the first step of #4779

Close #5712

[codecov]

Codecov Report

❌ Patch coverage is 67.01031% with 96 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-datafusion/src/catalog.rs	51.47%	29 Missing and 4 partials ⚠️
.../lance-namespace-datafusion/src/namespace_level.rs	67.53%	23 Missing and 2 partials ⚠️
.../lance-namespace-datafusion/src/session_builder.rs	79.62%	10 Missing and 12 partials ⚠️
rust/lance-namespace-datafusion/src/schema.rs	62.85%	12 Missing and 1 partial ⚠️
rust/lance-namespace-datafusion/src/error.rs	0.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[majin1102]

Hi @wjones127 @jackye1995 @yanghua @westonpace @Xuanwo ,

This PR is driven by #4779. In our production scenario, we’d like to leverage DataFusion to deliver a better SQL experience that supports search, analytics, and AI-related operators (e.g., via UDFs/UDTFs).

I’ve already discussed this with @jackye1995 offline, and we agreed to try this with a dedicated crate for the DataFusion namespace integration. This PR introduces the foundational approach.

I’ll follow up with additional issues covering next steps—such as Python bindings, URL-based tables, SQL extensions, etc.

Please take a look when you have time!

[wjones127]

This is cool. I have a few suggestions.

[wjones127]

I think we should preserve the original error using DataFusionError::External instead of converting to a string. This lets the caller turn the DataFusion error back into the original Lance error.

You can see how this is already partially done in #5606

[wjones127]

must be a four-level namespace

Why is this necessary? I wonder if we can work around that.

For example, it's a three level namespace, it could be:

DEFAULT > LVL1 > LVL2 > LVL3

And then if it's a two level namespace, it could be

DEFAULT > DEFAULT > LVL1 > LVL2

There might be some other standard name besides default that would make more sense (maybe what other DataFusion plugins do), but you get the idea.

[majin1102]

I think the docs might be somewhat misleading(have updated it). Let me clarify:

1. First, DataFusion only has three-level metadata. The CatalogProviderList is optional. As mentioned, we use SessionBuilder::with_root to configure a four-level namespace (let’s call this the root). In this case, all three-level child namespaces under the root are automatically registered as LanceCatalogProviders. This four-level namespace acts like a “catalog of catalogs” and is purely for convenience—instead of calling add_catalog() for each catalog individually.

2. We can always use SessionBuilder::add_catalog to manually register a catalog provider, regardless of whether the four-level namespace is configured.

3. I think what you’re really concerned about might be whether we can write queries against two- or three-level tables (e.g., SELECT * FROM db.tb or SELECT * FROM tb) when the four-level namespace is in use. The answer is yes. To do this, we need to configure the default catalog and schema names. If we don’t set them explicitly, they default to "datafusion" (for catalog) and "public" (for schema). That means db.tb would be interpreted as datafusion.db.tb, and tb as datafusion.public.tb.

4. I agree this is an important point. I’ve added methods to configure the default catalog and schema, and included examples in the tests.

[majin1102]

Ready for another review