Using cue modules defined externally

[jaqx0r]

I'd like to extract a module into a different git repo, and then use it in my original repo.

I have created a new Bazel module with a cue_module and a cue_instance containing the src and ancestor.

I use that Bazel module in the original repo, and pass the cue module repo as a dep to the cue_instance target.

The cue execution then fails because it can't find the module providing the package I want to import.

Can we change the import search path to look for it?

"Module" repo:

• MODULE.bazel defines the name
• BUILD.bazel merely exports the cue_isntance
• cue.mod/module.cue as you expect
• cue.mod/BUILD.bazel exports the cue_module
• foo.cue is the cue_istance being exported

"client" repo

• MODULE.bazel adds the bazel_dep of module with a local_path_override
• BUILD.bazel has the cue_instance, and adds @foo_module//:foo as a dep

import failed: cannot find package "github.com/jaqx0r/fue-cue/foo": cannot find module providing package github.com/jaqx0r/foo-cue/foo:
    ./client.cue:2:3

[jaqx0r]

It looks like this is extremely difficult.

https://cuelang.org/docs/tutorial/working-with-a-custom-module-registry/ tells me that there is no way to manually specify a filesystem path to a module's location, it's assumed that it's in the pkg subdirectory of the current module.

This means we also assume we are fetching the module with cue, which isn't compatible with using the Bazel downloader.

It also implies that if we were to use the cue.mod/module.cue deps attribute to fetch dependencies instead of my Bazel dep appraoch above, we don't get a hash of the version to verify integrity with.

[jaqx0r]

Having filed that bug, I think https://cuelang.org/docs/reference/command/cue-help-registryconfig/ might actually be a path forward. There's an option to pass the environment CUE_REGISTRY which defines where to pull modules from.

In theory, then, we could build a CUE_REGISTRY environment variable from the list of external dependencies, if the registry allows us to provide a filesystem path as the registry itself.

Alternatively, https://cuelang.org/docs/reference/command/cue-help-environment/ and https://cuelang.org/docs/reference/modules/#module-cache make me think that CUE_CACHE_DIR might be even better.

[jaqx0r]

	CUE_CACHE_DIR
		A directory structure to hold files which are reused between operations.
		It defaults to a directory for user-specific temporary data, such as:

		- "$XDG_CACHE_HOME/cue" or "$HOME/.cache/cue" on Linux
		- "$HOME/Library/Caches/cue" on MacOS
		- "%LocalAppData%/cue" on Windows

		Its contents are generally read-only and include:

		- mod/download for modules fetched from registries
		- mod/extract for extracted module archives

A starlark rule that creates a runfiles tree of mod/extract somewhere for each target's dependencies could solve my problem. I'll experiment with that.

[jaqx0r]

I see you've got CUE_CACHE_DIR in the wrapper binary already, but it's not set to anything in _make_output_producing_action or any its callers.

What was your intention with this parameter @seh ?

I also noticed that the cue_instance deps field accepts only CUEInstanceInfo providers, not other CUEModuleInfos. What are your thoughts on how external modules are represented with providers?

[seh]

You may be interested in the exploratory work along the "accommodate-hosted-cue-modules" topic branch (commit sequence). I started on that a few months ago, and got blocked by cue-lang/cue#3512, per my cue-lang/cue#3512 (comment). There was a lot of preceding discussion in the "CUE" Slack workspace, but unfortunately those threads have since fallen out of the retention window.

That work attempts to have Bazel use a repository rule to create a module cache directory for the CUE instance, populate it, and reuse it until the set of imported modules changes. It doesn't do all of that yet, because I couldn't get the cue tool to populate that directory.

I haven't gotten far enough along with that exploration to confront whether the providers warrant adjustment. I expect that the CUEModuleInfo provider will likely need to learn to refer to other modules, and each CUEInstanceInfo will continue to refer to the root module. (Note the comment in the novel providers.bzl file.)