Details
-
Improvement
-
Status: Triage Needed
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Currently Portable runners only support locally available files for adding dependencies on remote workers. This can be seen in https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/stager.py#L429 as it uses shutil.copyfile when it detects file is remote and it is not http.
An easy extension would be to extend _is_remote_path in Stager to detect if the path matches any filesystem and if it does the avoid downloading and let it be copied afterwards.
Acceptance criteria:
- `extra_package` can be a GCS path instead of requiring it to be local only.
Attachments
Issue Links
- duplicates
-
BEAM-5746 GCS path not working on extra packages
- Open
- links to