If you've not seen them already, the dbx_build_tools rules from Dropbox are currently the most sophisticated in terms of building third-party Python code from source using toolchains registered with Bazel. Python also often requires Fortran to be setup if I recall for many of the Numerical / Scientific libraries, but the Dropbox rules should make this clearer.

When you talk about "cross-compilation", are you talking about cross-compilation across CPU architectures (x86 vs ARM etc) or also across OS (win, macos, linux)? The reason I ask is because something that should be considered is that it is relatively common to come across "platform specific" dependencies. PEP 508 defines so called "Environment Markers" that change the dependency graph based on the system where the installation is happening. One example from Jupyter is https://github.com/ipython/ipython/blob/master/setup.py#L212.

Python does not currently have strictly separate resolve, build, install steps. It is the "resolve + build" step that is currently coupled, but Python is taking some steps towards making as much of this "static" as possible. However, in the vast majority of cases, setup.py still needs to execute and this is dynamic behavior. Perhaps a workaround for this would be to run the "resolve" step for all desired target platforms and somehow select the appropriate dependency graph at Bazel build time.

If you've not seen them already, the dbx_build_tools rules from Dropbox are currently the most sophisticated in terms of building third-party Python code from source using toolchains registered with Bazel. Python also often requires Fortran to be setup if I recall for many of the Numerical / Scientific libraries, but the Dropbox rules should make this clearer.

Thanks, I've recently become aware of them but have not yet explored what they're capable of. Thanks for the pointer re: Fortran.

When you talk about "cross-compilation", are you talking about cross-compilation across CPU architectures (x86 vs ARM etc) or also across OS (win, macos, linux)? The reason I ask is because something that should be considered is that it is relatively common to come across "platform specific" dependencies. PEP 508 defines so called "Environment Markers" that change the dependency graph based on the system where the installation is happening. One example from Jupyter is https://github.com/ipython/ipython/blob/master/setup.py#L212.

I mean cross OS and cross architecture. For now I'm installing pre-compiled wheels by specifying a number of additional pip arguments:

  "--only-binary", ":all",
  "--platform=manylinux2014_x86_64",
  "--platform=manylinux2010_x86_64",

Python does not currently have strictly separate resolve, build, install steps. It is the "resolve + build" step that is currently coupled, but Python is taking some steps towards making as much of this "static" as possible. However, in the vast majority of cases, setup.py still needs to execute and this is dynamic behavior. Perhaps a workaround for this would be to run the "resolve" step for all desired target platforms and somehow select the appropriate dependency graph at Bazel build time.

Right now I work around this by specifying different pip repositories for each platform and then a sort of "meta" requirement macro that looks like this:

load("@pypi_manylinux2010_x86_64//:requirements.bzl", linux_requirement = "requirement")
load("@pypi_macosx_x86_64//:requirements.bzl", darwin_requirement = "requirement")

def requirement(name):
  return select({
    "@platforms//os:linux": [linux_requirement(name)],
    "@platforms//os:macos": [darwin_requirement(name)],
    "//conditions:default": [],
  })

My approach seems to work fine for precompiled wheels, but I'm now seeing the need to extend rules_python so that it works for packages that require compilation.

The Python packaging ecosystem is always evolving, so in some sense it might be wise to take a "wait and see" approach until things have stabilised. Of course, things may never settle. :)

For example, there is an evolving standard in PEP-517. This clarifies the responsibilities of the so called "build front-end" and "build back-end".

Any "package author" is able to define and configure how their package builds from the package's source tree through their "build back-end", but package "end-users" are responsible for selecting and configuring the "build front-end".

It might be possible someday for Bazel rules_python to become a compliant PEP-517 "build front-end" if this standard is accepted and the ecosystem accepts it. However, what does that then mean if a package author wishes to use flit or other tools to build the wheel, instead of pip? Interesting times ahead I think.

My approach seems to work fine for precompiled wheels, but I'm now seeing the need to extend rules_python so that it works for packages that require compilation.

Not sure if you've considered it, but another option for your use case, if they are always third-party dependencies you are looking to compile would be to build (precompile) the wheels for your target platforms and host the artifacts in your own mirror (artifactory etc). Then Bazel can always work against prebuilt wheels which it simply needs to download and unpack.

Related issue: #260 ("Cross compilation" of py_binary/py_image/py_library targets)

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

Hi @ajbouh, I'm trying to replicate your setup in my org. I am getting below error

ERROR: /home/arunkant/code/qureai/packages/python/image_manager/BUILD:6:11: //packages/python/image_manager:image_manager: expected value of type 'string' for element 5 of attribute 'deps' in 'py_library' rule, but got select({":cpu": ["@python_deps_cpu_django//:pkg"], "//conditions:default": ["@python_deps_gpu_django//:pkg"]}) (select)

As I understand that requirement is a macro and it is not being evaluated correctly. Can you share a working example how you are using it? TIA