bzlmod example fails to build on pip install error when building whls from sdists #2118
Comments
|
The fix for this may be to upgrade to requirements where lazy-object proxy is available for arm macs. This is showing how the experimental requirements feature is working and unfortunately we have no arm macs for CI to verify that everything works. |
|
Hm I'm seeing this issue too, but it seems to be pointing to the issue where setuptools is not available to be installed. This issue was hidden before 6f9082f as sdist were installed via pip (which would then automatically install setuptools as required). A minimum reproduction would be a package that only has a sdist and no wheel that is compatible with the current host platform where you're testing from, and a pyproject.toml containing build requires like: In case it's helpful, you may want to try reproducing with https://pypi.org/project/pytest-shard/ . I saw this issue on occurring on that package. I saw: Full logs |
|
Seeing same error |
|
Note, that I think there are other potential solutions to unblock people (aside from using #2126 as a patch):
The last two options would be my prefered ones because that makes |
|
re:
I think I have a slightly different take here, which is that we'd like to use Bazel to hermetically build all the sdists and not depend on pre-built wheels so that we can manage supply chain build concerns. Even if we were to manage a private artifact repository, we'd still want to use Bazel to run the sdist builds and publishing steps since we use Bazel to build/orchestrate the rest of our infrastructure. At any rate, I think the community would benefit from guidance on how to deal with sdists that need to be built, especially where upstream projects are unwilling or unable to take on the burden of producing wheels for different architectures/platforms. |
|
Thank you all for the work and thoughts on this! If you'll forgive me for adding some unsolicited opinions, I have some concerns about the last two suggestions. As an end user who wants things to be as simple as possible, my biggest sticking point with the last two of @ewianda 's solutions is that it seems like the maintenance/complexity is passed down to the python developer. A python developer who is used to the fact that 3 9s percent of the time they just need to run From a hermetic build standpoint, using pip to download and build sdists can be a problem. From a bazel adoption standpoint, where users want to reliably build applications in one of the most popular languages, adding non ergonomic build configuration steps for things python devs don't care about is annoying and time consuming in the least (or require dedicated hand holding from a build engineer). Adopting bazel is not an easy paradigm shift for people to begin with, so anything to dull the edges here would be preferable. Otherwise, expect more threads like these. (context: I'm doing a POC migration for a large python/scala/java/terraform monorepo and evaluating bazel/buck2/pants. For bazel, I'm getting stuck the most with getting ergonomic python builds for many independent py projects in the monorepo, each with their own currently working pyproject.toml /pip lock files. As things stand, I worry that my python devs will revolt if I push further complexity on them! 🤣) |
|
Thanks for the message.
I am happy that bazel is getting traction in the python world and I am happy for you that you are looking into ways of adopting bazel in your organization. By the way, from the list I posted previously, only the first solution comes from @ewianda, so let me take the blame here. :)
My reasoning with the last two solutions was to give the build engineer full power to have full hermeticity as opposed to relying on the non-hermetic repository rules.
Not using wheels at all and having everything just work could be doable, but that may be outside the immediate or even medium-term future. For that to be viable, someone would have to port/reimplement rules_pycross within rules_python.
Since this is something that only occurs when the experimental feature is used, the priority of fixing sdists may be low until the rest of the code is stabilized.
For the time being I think adding --no-build-isolation and removing --no-index us probably the best course of action - contributions are welcome here and thanks to @ewianda for submitting the PR!
P.S. it's up to people to vent online about bazel. There will always be people who don't like X or Y and I certainly think the maintainers of this project are not responsible for devs in your company not liking bazel - everyone is free to have their opinions. :) I am always happy when people contribute to the project by letting the maintainers know about existing issues and or submitting PRs, but those threads on reddit or elsewhere do not move the needle here. :)
…On 21 August 2024 19:57:47 EEST, steve layland ***@***.***> wrote:
Thank you all for the work and thoughts on this! If you'll forgive me for adding some unsolicited opinions, I have some concerns about the last two suggestions.
As an end user who wants things to be as simple as possible, my biggest sticking point with the last two of @ewianda 's solutions is that it seems like the maintenance/complexity is passed down to the python developer. A python developer who is used to the fact that 3 9s percent of the time they just need to run `pip install foo` (or add a single line to a pyproject.toml, Pipfile, or requirements.txt file) to get things to working for their project.
From a hermetic build standpoint, using pip to download and build sdists can be a problem. From a bazel adoption standpoint, where users want to reliably build applications in one of the most popular languages, adding non ergonomic build configuration steps for things python devs don't care about is annoying and time consuming in the least (or require dedicated hand holding from a build engineer).
Adopting bazel is not an easy paradigm shift for people to begin with, so anything to dull the edges here would be preferable. Otherwise, expect more [threads](https://news.ycombinator.com/item?id=38493940) like [these](https://www.reddit.com/r/devops/s/bdP3bW8fVK).
(context: I'm doing a POC migration for a large python/scala/java/terraform monorepo and evaluating bazel/buck2/pants. For bazel, I'm getting stuck the most with getting ergonomic python builds for many independent py projects in the monorepo, each with their own currently working pyproject.toml /pip lock files. As things stand, I worry that my python devs will revolt if I push further complexity on them! 🤣)
--
Reply to this email directly or view it on GitHub:
#2118 (comment)
You are receiving this because you commented.
Message ID: ***@***.***>
|
I will add this to the PR. Unfortunately, I already pushed the wheels of the two offending packages to our private repository, so I can't verify that adding Credit for this solution should actually go to @rockpunk |
Fair point, you're totally right; i shouldn't have brought in external noise. That said, in my research these past couple of months, the pervading sentiment that I've seen is that Bazel and python are still unnecessarily complex from a developer POV and I want to encourage anything that changes that perception. Thanks again for the hard work! |
…when build sdist (#2126) Building sdist results in `Could not find a version that satisfies the requirement setuptool` this regressed when a fix in parameter handling got introduced in #2091. Before this change the building from sdist when using `experimental_index_url` would break because `--no-index` is passed to `pip`. This means that `pip` would fail to locate build time dependencies needed for the packages and would just not work. In `whl_library` we setup `PYTHONPATH` to have some build dependencies available (like `setuptools`) and we could use them during building from `sdist` and to do so we need to add `--no-build-isolation` flag. However, for some cases we need to also add other build-time dependencies (e.g. `flit_core`) so that the building of the wheel in the `repository_rule` context is successfuly. Removing `--no-index` allows `pip` to silently fetch the needed build dependencies from PyPI if they are missing and continue with the build. This is not a perfect solution, but it does unblock users to use the `sdist` distributions with the experimental feature enabled by using `experimental_index_url` (see #260 for tracking of the completion). Fixes #2118 Fixes #2152 --------- Co-authored-by: aignas <240938+aignas@users.noreply.github.com>
🐞 bug report
Affected Rule
examples/bzlmod:
//:all_data_requirementsIs this a regression?
Yes, the previous version in which this bug was not present was: commit e682cd0
Description
Seems like PR #2091 breaks the bzlmod example on mac arm64.
🔬 Minimal Reproduction
NOTE: commit 6f9082f was isolated via
git bisectgit bisect output
git clone https://github.com/bazelbuild/rules_pythongit checkout 6f9082fda020a9970d624e54ad3d964d64957f21cd rules_python/examples/bzlmodbazel build //:all_data_requirementsValidate that previous version of example works:
git checkout 59bb4a88783cca738394cb3cb91bc8243df5e18dbazel build //:all_data_requirements🔥 Exception or Error
full stack trace
🌍 Your Environment
Operating System:
Output of
bazel version:Rules_python version:
Anything else relevant?
I tried manually installing
lazy-object-proxyin the rules_pythonpythoninterpreter, and was able to successfully install only by removing the--no-indexoption to pip. Not sure why--no-indextells pip to not use the installed setuptools (which indeed has version >= 64)Fails to install with
Works with
The text was updated successfully, but these errors were encountered: