Build for older macOS versions using bump-formula-pr process in a custom tap

[rgoldberg]

In a custom tap, is there a way to make the bump-formula-pr process (using a pr-pull label, etc.) build for older macOS versions (like macOS 12, 11, 10.15, 10.14, …), either by:

• configuring the usage of the GitHub Workflows generated by brew tap-new
• editing the GitHub Workflows generated by brew tap-new
• obtaining different Workflows from elsewhere that handle this
• etc.

Is there a way to easily have the custom tap formula upload bottles to a GitHub Release in the main project repo and either:

• not upload the bottles to the custom tap repo, then use URLs pointing to bottles in the main project repo instead of in the custom tap
• also upload the bottles to the custom tap repo, then use URLs pointing to bottles in the custom tap

I prefer the former, but would settle for the latter if the former wouldn't be simple or advisable.

[ipatch]

for the first part of your question,

In a custom tap, is there a way to make the bump-formula-pr process (using a pr-pull label, etc.) build for older macOS versions (like macOS 12, 11, 10.15, 10.14, …), either by:

if you want to build bottles for older versions of macos ie. 10.14, etc, etc. you'll need to setup a self-hosted runner running that macos version ie. macos 10.14. be advised though if you core tap uses that latest upstream homebrew-core you'll have to build bottles for each and every formula or point the homebrew-core tap to a specific revision which still contains bottles for that macos version. this will need to be done for each macos version you plan on supporting, as out of the box the workflows provided by homebrew are only setup to use github runners that are officially maintained by github.

[rgoldberg]

@ipatch & @carlocab Thanks for the info & for looking at this.

Building bottles for older macOS versions:

The project in question is mas. The current mas version (1.8.7) is written in Swift 5.7 & is configured to be able to run on macOS 10.13+. The prior version (1.8.6) is configured to be able to run on macOS 10.11+.

I'd like our custom tap to build bottles for each newly released version of mas that can be used for macOS 10.13+, but also offer the legacy mas 1.8.6 for macOS 10.11 & 10.12. Is that possible?

According to Apple docs, Xcode 14.2+ (currently 16.1) on macOS 12.5+ can build Swift 5.7 executables that can run on macOS versions back to 10.13.

Assuming Apple's build system works as advertised, is there any need to build our custom tap bottles on any macOS version other than macOS 15? (GitHub macos-15 runners currently default to using Xcode 16, which can build for macOS 10.13)

I plan to build an arm64 executable that would be used for macOS 11+ on Apple Silicon, and an x86_64 executable that would be used for macOS 10.13+ on Intel.

Can you let me know if there is anything wrong with the following, or if there is a better way to handle all this?

• In the test.yml workflow generated by brew tap-new:
  • Modify it to rename the bottles generated by brew test-bot --only-formulae on macos-15:
    • arm64 bottle from *sequoia* to *big_sur*
    • x86_64 bottle from *sequoia* to *high_sierra*
• In the publish.yml workflow generated by brew tap-new:
  • brew pr-pull --debug --tap="$GITHUB_REPOSITORY" "$PULL_REQUEST" will properly upload the bottles from the workflow artifact to the new GitHub Release in the custom tap, despite the macOS versions in the file names being old (i.e. big_sur & high_sierra)
  • Modify it to append links to existing mas 1.8.6 bottles for el_capitan & sierra.
    • Would those bottles be able to be served from a different root_url than the newly built bottles for the newly released version of mas?
    • If not, would it be terrible if old 1.8.6 bottles were copied to the GitHub Release for the new mas version, so they could be served from there?
• After those 2 bottles are published to the GitHub Release in the custom tap repo, they will be used for their respective platform for their respective macOS version & newer

Serving bottles from elsewhere than GitHub Release in custom tap

You can ignore my question about serving the bottles from the main project repo (which I think you took to mean Homebrew/homebrew-core, but by which I meant mas-cli/mas; sorry for any confusion). I now have permission to store the bottles in the usual place: a GitHub Release in our custom tap repo: mas-cli/homebrew-tap (I know that's a nonstandard name, but it predates me, and it might be hard to change it to mas-cli/homebrew-mas).

[ipatch]

my knowledge with swift is limited to say the least. but if your targeting macos 10.13 (high sierra) as your minimum build target, github does not provide a runner image for that macos version. so you'll need to setup a custom github runner ie. a self-hosted runner running that version of macos. then if your formula in question relies on dependencies from homebrew to build the formula in question each one of those dependencies will need to have bottle to build your formula in question or brew test-bot will fail to build your bottle. also you'll need to make you setup your project ie. (cmake or autotools) to use the proper SDK, which shouldn't be a big deal if you go the self-hosted route because macos 10.13 will use the SDK provided by the version of Xcode that runs on macos 10.13.

things begin to get difficult if you want to use a github provided runner and then you use a non standard SDK ie. an older SDK say (10.13) on macos sequoia as your project and all it's deps will require targeting that SDK.

good luck.

[rgoldberg]

@ipatch Thanks for the info.

Do you know anything about the 2 generated GitHub workflows that I mentinoed? If so, is my plan workable?

Also, I'm confused: why would I need to build on High Sierra?

Swift 5.7 requires Xcode 14.2+, which can only run on macOS 12.5+. You cannot build Swift on every target macOS on which it can run. So I cannot build on High Sierra for High Sierra.

Sequoia can run Xcode 14.2+ through current Xcode (i.e. 16.x), all of which can build for 10.13+. A Swift executable built on Sequoia should run on 10.13+, as long as the project is configured to work on 10.13+, which mine is.

I guess I'm mystified about why Homebrew ties the build macOS version to the deploy macOS version. It might be necessary for certain languages, but for many languages (like Swift, Java, etc.), the build & deploy macOS versions shouldn't need to be the same.

[ipatch]

I guess I'm mystified about why Homebrew ties the build macOS version to the deploy macOS version. It might be necessary for certain languages, but for many languages (like Swift, Java, etc.), the build & deploy macOS versions shouldn't need to be the same.

i'd suggest you look the homebrew documentation / api, you'll see that if your project is using cmake or autotools when homebrew installs from a bottle, the bottle will represent what was defined in the formula file for the package that was installed, and from what i have gathered if a homebrew formula in the core tap uses cmake, then it will use the std_cmake_args

https://rubydoc.brew.sh/OS/Mac/Formula.html#std_cmake_args-instance_method
https://rubydoc.brew.sh/

here's another discussion that reiterates what i stated above

https://github.com/orgs/Homebrew/discussions/4831#discussioncomment-7224896ew-core

taking a look at your tap / formula it looks like your formula file doesn't use / rely on any external deps provided by homebrew so that's at least a positive thing.

https://github.com/mas-cli/homebrew-tap/blob/main/Formula/mas.rb

if you have access to a 10.13 box you could always build / install on it and see if thing work as intended.

[rgoldberg]

@ipatch Thanks again for the info.

mas builds using swift build. I haven't seen anything about it using autotools or cmake.

For now, I imagine that we can just build on macOS-15, rename the bottles as appropriate, and use them for macOS 10.13+.

The main questions are if brew pr-pull will properly upload bottles for old macOS versions, and if/how the custom tap can serve old mas versions for old macOS versions.

mas will probably depend on jq & util-linux in the future because it will use them to format terminal output. In that case, users on older macOS versions can either build those formulae via Homebrew, or install the executables some other way.

[Bo98]

Homebrew doesn't really support cross-builds at the moment, mainly because it is not something we support in homebrew-core and nobody has really requested or developed a solution for third-party taps.

With that said we do use cross-builds for the Homebrew Portable Ruby tap, using an undocumented Homebrew environment variable to "fake" the host OS version to be the deployment OS. I'm not really sure if this approach scales well for third-party tap use cases - it was a quick hack solely tested for a Ruby build without much thought put into it. Someone should probably design a better system for these use cases.

An alternative solution here could be to support "unversioned" macOS bottles like arm64_macos. There has been some interest there, more so than cross-builds, but that also isn't supported today. And for the use case of cross-builds specifically: I'm not entirely convinced it makes sense for native binaries when a single cross-built bottle for the oldest deployment target would make more sense, given newer macOS support falling back to that anyway.

Right now the approach taken by some is to compile the binary externally and to just download & install the binary in the formula. I agree this is not great and anything that avoids those scenarios would be an improvement.

[rgoldberg]

@Bo98 Thanks for the info.

Not sure if you saw my reply to @ipatch in the thread above, which offers more details about my plan & my questions about it. Assuming that the executable built by my Swift build on macOS 15 can indeed be deployed back to macOS 10.13, the overall gist of my proposal is to:

• on a macos-15 runner, build 2 bottles, one each for arm64 & x86_64.
• rename the arm64 bottle from *sequoia* to *big_sur*
• rename the x86_64 bottle from *sequoia* to *high_sierra*
• hope that brew pr-pull --debug --tap="$GITHUB_REPOSITORY" "$PULL_REQUEST" in my custom tap's publish.yml publishe the bottles to the GitHub Release in the custom tap repo
• Somehow either:
  • Insert URLs to existing legacy mas 1.8.6 bottles into the bottles block of the newly modified custom tap formula, to provide the newest version of mas that runs on macOS 10.11 & 10.12.
  • copy existing existing legacy mas 1.8.6 bottles from the 1.8.6 release to the GitHub Release for the new version in the custom tap repo, then insert the URLs to the copied 1.8.6 bottles into the bottle block of the newly modified custom tap formula.

Must any files within any of the bottles be updated or removed to be reassigned to a different macOS version? Must any files be added to any of the bottles?

Any help would be appreciated.

[rgoldberg]

@Bo98 How hard do you think it would be to add in the ability to cross build bottles? (by that, I mean, e.g., building on macOS 15, but specifying that the built bottle is for macOS 10.13+)

Could a first-time contributor (outside of minor formula improvements & bumps) like myself get it done in a reasonable amount of time?

Would it be accepted?

[MikeMcQuaid]

@Bo98 How hard do you think it would be to add in the ability to cross build bottles? (by that, I mean, e.g., building on macOS 15, but specifying that the built bottle is for macOS 10.13+)

Could a first-time contributor (outside of minor formula improvements & bumps) like myself get it done in a reasonable amount of time?

Would it be accepted?

@rgoldberg It would likely be accepted. It will probably take a lot of time and review and require adjusting not just Homebrew/brew but Homebrew/homebrew-test-bot and Homebrew/homebrew-core's CI configuration.

[rgoldberg]

@Bo98 @MikeMcQuaid Tying the ability to distribute a bottle via Homebrew to a macOS version only if it can build on that version keeps causing issues for mas.

Many of the features of Swift & its ecosystem can only be built on newer Xcode & macOS versions, but the binaries can be deployed on older macOS versions.

The mas build settings require only Swift 5.9 (though I'd prefer to use Swift 6.0), which can be built by Xcode 15, which can run on macOS 13.

Various bottling problems still arise, however, either because the Swift tooling doesn't properly ensure compatibility, or because some tool fails (e.g., the periphery formula, which mas uses for linting, won't install on macOS 13, breaking the bootstrapping for the lint check).

I assume that allowing cross-building of bottles on a newer macOS version to be installed on older macOS versions isn't something that Homebrew will support anytime soon (or maybe even ever). If I'm wrong, please let me know.

Assuming cross-building won't be supported, would it cause problems if mas transitioned from a formula to a cask? Is there any automated way to migrate users from a formula to a replacement cask?

Would users trust a cask less than a bottle or formula, because of the lack of directly tying the code to the executable?

brew uses --eval-all mas outputs nothing, indicating that no other packages depend on mas, but I don't know if brew bundle offers to install (or automatically installs) mas.

Either solution would avoid all the random Swift & macOS version compatibility issues I encounter, allow me to use newer versions of languages, frameworks, libraries, tools, etc., and generally simplify & improve things.

Are they any other feasible ways to achieve similar results?

Obviously, mas could technically be distributed via the homebrew-core formula to only a subset of the newest macOS versions that Homebrew supports, and distributed to older macOS versions via the mas custom tap, but that seems like an undesirable & impractical nightmare scenario, so I am not suggesting it, only noting its technical possibility.

Also, mas could have both a homebrew-core formula & a homebrew-cask cask, if users on newer macOS versions really want the binaries tied to the source, but I imagine that having both would be suboptimal at best.

Thanks for any help.

[MikeMcQuaid]

Assuming cross-building won't be supported, would it cause problems if mas transitioned from a formula to a cask?

Yes, we're not going to do that, sorry.

Obviously, mas could technically be distributed via the homebrew-core formula to only a subset of the newest macOS versions that Homebrew supports, and distributed to older macOS versions via the mas custom tap, but that seems like an undesirable & impractical nightmare scenario, so I am not suggesting it, only noting its technical possibility.

This is the best situation assuming you want to distribute binaries.

Can you be more specific about which macOS versions have which issues today and which do not?

Bear in mind Homebrew isn't going to be interested in making changes for anything that's not a supported (current macOS and prior 2 releases) version.

[rgoldberg]

@MikeMcQuaid Thanks for the reply.

The issues aren't for running mas, they're for building mas.

I've fixed the current build issues, but at least 3 issues have arisen in the 2 months since I got macOS 15 & Xcode 16; I'm worried about more in the future.

Issues that have arisen include:

• @retroactive build error on macos-13 runner
• access levels on imports build error on macos-13 runner
• the periphery formula, which is written in Swift 6.0, wouldn't build/install on macos-13 runner; I haven't upgraded to Swift 6.0 because I want to ensure that mas can be distributed by Homebrew to all macOS versions that it supports, but other projects don't necessarily try to completely work with Homebrew
  -I got build warnings on macOS-14 & macos-13 that aren't on macos-15, so those don't break the build, but maybe there might be runtime issues due to code being built in a suboptimal manner, but those issues were fixed in newer Xcode versions.

I can't remember if there were other issues.

Needing to build on the 3 latest macOS versions currently prevents me from upgrading to Swift 6, which would allow better error handling & using Swift Testing, along with other improvements.

After macOS 13 is EOL, I can upgrade to Swift 6, but my concern extends to all formulae written in Swift, and all the issues that they all might face. I doubt other languages are so tied to macOS versions, but, unfortunately, Swift is.

[SMillerDev]

Needing to build on the 3 latest macOS versions currently prevents me from upgrading to Swift 6, which would allow better error handling & using Swift Testing, along with other improvements.

This is not a strict requirement for Homebrew. If your software only runs on the latest macOS version it can still be in Homebrew.

[rgoldberg]

@SMillerDev mas can run on macOS 10.15+, but can only build on macOS 13+. Sometimes, builds break on older macOS versions but not on the newest (like breaking on macOS 13, but building fine on macOS 15).

I have no intention of preventing mas from running on any macOS version that is currently supported by Apple or Homebrew. It's just the building that sometimes is the issue.

Given those requirements, the lack of cross building, mas being a formula instead of a cask, and not wanting to cause confusion with both a formula & a cask for users of Homebrew-supported macOS versions, mas must build on 3 versions of macOS.

Currently needing to build on macOS 13 prevents mas from using certain newer Swift features, and at times causes build problems that need to be fixed. The same will probably be true indefinitely unless cross building is supported or I change the mas distribution model (cask, formula & tap cask, etc.). Don't worry, I won't change anything without consulting you all & getting approval. mas will continue to build on all versions of macOS supported by Homebrew. I just wanted to start exploring options with you to see if things could be improved.

[MikeMcQuaid]

After macOS 13 is EOL, I can upgrade to Swift 6

That sounds like the best bet.

but my concern extends to all formulae written in Swift, and all the issues that they all might face. I doubt other languages are so tied to macOS versions, but, unfortunately, Swift is.

I'd leave worrying about other formulae to the Homebrew maintainers.