Related issue: bazelbuild/rules_python#260

It seems that a few things broke in recent commits:
- We are not using the `MODULE.bazel.lock` file and it seems that it is
easy to
  miss when the components in the PyPI extension stop integrating well
together. This happened during the switch to `{abi}_{os}_{plat}` target
  platform passing within the code.
- The logger code stopped working in the extension after the recent
additions
  to add the `rule_name`.
- `repo_utils.getenv` was always getting `PATH` env var on bazel `6.x`.

This PR fixes both cases and updates docs to serve as a better reminder.
By
fixing the `select_whls` code and we can just rely on target platform
triples
(ABI, OS, CPU). This gets one step closer to maybe supporting optional
`python_version` which would address #1708. Whilst at it we are also
adding
different status messages for building the wheel from `sdist` vs just
extracting or downloading the wheel.

Tests:
- Added more unit tests and brought them in line with the rest of the
code.
- Checked manually for differences between the `MODULE.bazel.lock` files
in our
`rules_python` extension before #2069 and after this PR and there are no
  differences except in the `experimental_target_platforms` attribute in
`whl_library`. Before this PR you would see that we do not select any
wheels
  for e.g. `MarkupSafe` and we are always building from `sdist`.

Work towards #260.

Before this change the `all_requirements` and related constants will
have
packages that need to be installed only on specific platforms and will
mean
that users relying on those constants (e.g. `gazelle`) will need to do
extra
work to exclude platform-specific packages. The package managers that
that
support outputting such files now include `uv` and `pdm`. This might be
also
useful in cases where we attempt to handle non-requirements lock files.

Note, that the best way to handle this would be to move all of the
requirements
parsing code to Python, but that may cause regressions as it is a much
bigger
change. This is only changing the code so that we are doing extra
processing
for the requirement lines that have env markers. The lines that have no
markers
will not see any change in the code execution paths and the python
interpreter
will not be downloaded.

We also use the `*_ctx.watch` API where available to correctly
re-evaluate the
markers if the `packaging` Python sources for this change.

Extra changes that are included in this PR:
- Extend the `repo_utils` to have a method for `arch` getting from the
`ctx`.
- Change the `local_runtime_repo` to perform the validation not relying
on the
  implementation detail of the `get_platforms_os_name`.
- Add `$(UV)` make variable for the `uv:current_toolchain` so that we
can
  generate the requirements for `sphinx` using `uv`.
- Swap the requirement generation using `genrule` and `uv` for `sphinx`
and co
so that we can test the `requirement` marker code. Note, the
`requirement`
  markers are not working well with the `requirement_cycles`.

Fixes #1105.
Fixes #1868.
Work towards #260, #1975.
Related #1663.

---------

Co-authored-by: Richard Levasseur <rlevasseur@google.com>

…when build sdist (#2126)

Building sdist results in `Could not find a version that satisfies the
requirement setuptool` this regressed when a fix in parameter handling
got introduced in #2091.

Before this change the building from sdist when using
`experimental_index_url`
would break because `--no-index` is passed to `pip`. This means that
`pip`
would fail to locate build time dependencies needed for the packages and
would
just not work. In `whl_library` we setup `PYTHONPATH` to have some build
dependencies available (like `setuptools`) and we could use them during
building from `sdist` and to do so we need to add `--no-build-isolation`
flag.
However, for some cases we need to also add other build-time
dependencies (e.g.
`flit_core`) so that the building of the wheel in the `repository_rule`
context
is successfuly. Removing `--no-index` allows `pip` to silently fetch the
needed
build dependencies from PyPI if they are missing and continue with the
build.

This is not a perfect solution, but it does unblock users to use the
`sdist`
distributions with the experimental feature enabled by using
`experimental_index_url` (see #260 for tracking of the completion).

Fixes #2118
Fixes #2152

---------

Co-authored-by: aignas <240938+aignas@users.noreply.github.com>

With this change we set the default value of `--python_version` when
the `python.toolchain` is used in `bzlmod` and we generate the
appropriate config settings based on the registered toolchains and
given overrides by the root module.

This means that we expect the `--python_version` to be always set to
a non-empty value under `bzlmod` and we can cleanup code which was
handling `//conditions:default` case. This also means that we can
in theory drop the requirement for `python_version` in `pip.parse`
and just setup dependencies for all packages that we find in the
`requirements.txt` file and move on. This is left as future work
by myself or anyone willing to contribute. We can also start reusing
the same `whl_library` instance for multi-platform packages because
the python version flag is always set - this will simplify the layout
and makes the feature non-experimental anymore under bzlmod.

Summary:
* Add `@pythons_hub` to the `WORKSPACE` with dummy data to make
  pythons_hub work.
* Add `MINOR_MAPPING` and `PYTHON_VERSIONS` to pythons_hub to
  generate the config settings.
* Remove handling of the default version in `pypi` code under bzlmod.

Work towards #2081, #260, #1708

---------

Co-authored-by: Richard Levasseur <rlevasseur@google.com>

With this change we can support ibazel for building our docs again
because we will just not have any sdists that are causing issues. This
limits the scope to only supporting whls at this time, but for the
time being it is the best solution.

Fixes bazelbuild#2223
Work towards bazelbuild#260

With this change we can support ibazel for building our docs again
because we will just not have any sdists that are causing issues. This
limits the scope to only supporting whls at this time, but for the
time being it is the best solution.

Fixes #2223
Work towards #260

Before this change the `extra_pip_args` would not be passed down the
chain if `experimental_index_url` is set. This adds a test and fixes the
bug.

Work towards bazelbuild#260.

Before this change the `extra_pip_args` would not be passed down the
chain if `experimental_index_url` is set. This adds a test and fixes the
bug.

Work towards #260

…repos (#2369)

Before this change, it was impossible for users to use the targets
created with `additive_build_content` whl annotation unless they relied
on the implementation detail of the naming of the spoke repositories and
had `use_repo` statements in their `MODULE.bazel` files importing the
spoke repos.

With #2325 in the works, users will have to change their `use_repo`
statements, which is going to be disruptive. In order to offer them an
alternative for not relying on the names of the spokes, there has to be
a way to expose the extra targets created and this PR implements a
method. Incidentally, the same would have happened if we wanted to
stabilize the #260 work and mark `experimental_index_url` as
non-experimental anymore.

I was hoping we could autodetect them by parsing the build content
ourselves in the `pip` extension, but it turned out to be extremely
tricky and I figured that it was better to have an API rather than not
have it.

Whilst at it, also relax the naming requirements for the
`whl_modifications` attribute.

Fixes #2187

…azelbuild#2325)

With this change we finally generate the same lock file within the
legacy code `pip.parse` code path and it allows to slowly transition to
using the new code path as much as possible without user doing anything.

This moves the selection of the host-compatible lock file from the
extension evaluation to the build phase - note, we will generate extra
repositories here which might not work on the host platform, however, if
the users are consuming the `whl_library` repos through the hub repo
only, then everything should work. A known issue is that it may break
`bazel query` and in these usecases it is advisable to use `cquery`
until we have `sdist` cross-building from source fully working.

Summary:
- feat: reuse the `render_pkg_aliases` for when filename is not known
  but platform is known
- feat: support generating the extra config settings required
- feat: `get_whl_flag_versions` now generates extra args for the rules
- feat: make lock file generation the same irrespective of the host
  platform
- test: add an extra test with multiple requirements files
- feat: support cross-platform builds using `download_only = True` in
  legacy setups

Note, that users depending on the naming of the whl libraries will need
to start using `extra_hub_aliases` attribute instead to keep their
setups not relying on this implementation detail.

Fixes bazelbuild#2268
Work towards bazelbuild#260

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>

This just cleans up the code and moves more logic from the
repository_rule
(i.e. generation of `BUILD.bazel` files) to loading time (macro
evaluation).
This makes the unit testing easier and I plan to also move the code that
is
generating config setting names from filenames to this new macro, but
wanted to
submit this PR to reduce the review chunks.

Summary:
- Add a new `pkg_aliases` macro.
- Move logic and tests for creating WORKSPACE aliases.
- Move logic and tests bzlmod aliases.
- Move logic and tests bzlmod aliases with groups.
- Add a test for extra alias creation.
- Use `whl_alias` in `pypi` extension integration tests.
- Improve the serialization of `whl_alias` for passing to the pypi hub
repo.

Related to #260, #2386, #2337, #2319 - hopefully cleaning the code up
will make
it easier to address those feature requests later.

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>