If this is not the cause, would it be able to upgrade python(3)-requests to v2.20.1+? Pywikibot requires this version and thus can't work on Toolforge currently.

requests 2.12.4 is 40 months old and very outdated. There are some external dependencies like urllib3, idna, chardet, certifi and I've no glue whether this is related to Python versions. Anyway requests 2.12.4 is not compatible with Python 3.7. Refer https://requests.readthedocs.io/en/master/community/updates/#id4 for other importand changes of requests.

The lowest request version needed for sseclient is 2.9 and 2.18.2 causes bug T172059.

The same issue with ipaddress on Python 2:

$ python pwb.py login
(requests 2.12.4 (/usr/lib/python2.7/dist-packages), Requirement.parse('requests>=2.20.1; python_version != "3.4"'))
(ipaddress 1.0.17 (/usr/lib/python2.7/dist-packages), Requirement.parse('ipaddress>=1.0.23; python_version < "3"'))

Please install/update required modules with:


    pip install "requests>=2.20.1"

    pip install "ipaddress>=1.0.23"

$ apt-cache policy python-ipaddress
python-ipaddress:
  Installed: 1.0.17-1
  Candidate: 1.0.17-1
  Version table:
 *** 1.0.17-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

@Xqt Perhaps we should allow ipaddress 1.0.17 because of Toolforge?

(Could we also change this error to just a warning? Because Toolforge has some other Pywikibot dependencies too old)

Pywikibot master (instead of stable, silly)

I run selected tasks on master branch to detect regressions and other bugs, not a silly thing I guess :P

It looks like in this particular case we can force installing the package from stretch-backports as a near term fix.

The permanent fix for this is to use a virtualenv for your Python library dependencies rather than relying on the system packages. The system packages are installed via Debian packages and thus they will always be older versions.

In T248376#5994884, @revi wrote:

Pywikibot master (instead of stable, silly)

I run selected tasks on master branch to detect regressions and other bugs, not a silly thing I guess :P

For detection of bugs and regressions not, I admit :D (I do it too). Not good for production where you really need it to work and be maintainless.

In T248376#5994902, @bd808 wrote:

It looks like in this particular case we can force installing the package from stretch-backports as a near term fix.

The permanent fix for this is to use a virtualenv for your Python library dependencies rather than relying on the system packages. The system packages are installed via Debian packages and thus they will always be older versions.

python3-requests is the only and most important Pywikibot dependency for Python 3, I would vote for stretch-backports in this case.

python-requests and python-ipaddress are Pywikibot dependencies for Python 2. Its support will be ended in Pywikibot soon, so their update is not crucial. Some Python 2 Pywikibot bots left in Toolforge broke by this strict dependency check, some will break with the next Pywikibot release. Only venv bots should be fine. The question is if we want to break any. I would say yes :D but I know so many people will argue with me.

For everyone with broken bot: You can (should) use Pywikibot stable (/shared/pywikibot/stable or git tag stable), where this issue does not occur. See comments above

Change 583293 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] Enable pywikibot run even there is a package version conflict

https://gerrit.wikimedia.org/r/583293

In T248376#5994520, @Dvorapa wrote:

(Could we also change this error to just a warning? Because Toolforge has some other Pywikibot dependencies too old)

Did it. The script will be run even there is a version conflict but gives a warning. Should we have a config variable like strict_dependencies to let the script fail if there is a version conflict?

In T248376#5994884, @revi wrote:

Pywikibot master (instead of stable, silly)

I run selected tasks on master branch to detect regressions and other bugs, not a silly thing I guess :P

Yes, we need experienced users which helps reviewing. Thanks.
PAWS uses stable btw.

Change 583293 merged by jenkins-bot:
[pywikibot/core@master] Enable pywikibot run even there is a package version conflict

https://gerrit.wikimedia.org/r/583293

In T248376#5997173, @Dvorapa wrote:

For everyone with broken bot: You can (should) use Pywikibot stable (/shared/pywikibot/stable or git tag stable), where this issue does not occur. See comments above

It works! Thank you.

OK. I set up all our processes in our crontab to /shared/pywikibot/stable. It was fine everything until today. Now I see on the log of atobot again as follows:

(requests 2.12.4 (/usr/lib/python3/dist-packages), Requirement.parse('requests>=2.20.1; python_version != "3.4"'))

A package necessary for pywikibot is outdated.
Please update required module with:

pip install "requests>=2.20.1"

You may continue on your own risk; type CTRL-C to stop.

Yes, but it should be only a warning now, not a failure. Pywikibot should continue after sleep(5s)

On the Toolforge side, we install the 'python-requests' and 'python3-requests' packages as part of the profile::toolforge::genpp::python_exec_stretch Puppet module. That module is generated code, so we will need to think a bit about how to properly add pinning there so that the pinning survives subsequent runs of the module generation script.

Also for archivebot.py jobs the script fails completely. I am getting this error on the grid

Traceback (most recent call last):
  File "scripts/archivebot.py", line 111, in <module>
    import pywikibot
  File "/shared/pywikipedia/core/pywikibot/__init__.py", line 25, in <module>
    from pywikibot.bot import (
  File "/shared/pywikipedia/core/pywikibot/bot.py", line 109, in <module>
    from pathlib2 import Path
ImportError: No module named pathlib2

If it helps it is in the cronjob for Wikisource-bot

Please note that for running scripts with Python 2 the following packages are mandatory:

• enum34>=1.1.6,!=1.1.8
• ipaddress>=1.0.23
• pathlib2
• requests>=2.20.1

Supporting Python 2 is deprecated and will be dropped soon (T213287).

For Python 3 we only have

• requests>=2.20.1

Supporting Python 3.4 is deprecated and will be dropped soon (T239542).

Several scripts might have additional dependencies.

In T248376#6042172, @Xqt wrote:

Please note that for running scripts with Python 2 the following packages are mandatory:

• enum34>=1.1.6,!=1.1.8

$ apt-cache policy python-enum34
python-enum34:
  Installed: 1.1.6-1
  Candidate: 1.1.6-1
  Version table:
 *** 1.1.6-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

• ipaddress>=1.0.23

1.0.17 is the best we can do with upstream packaging on the Toolforge bastions and grid engine nodes (Debian Stretch). The actual code delta from 1.0.17 to 1.0.23 looks pretty minor -- https://github.com/phihag/ipaddress/compare/v1.0.17...master

$ apt-cache policy python-ipaddress
python-ipaddress:
  Installed: 1.0.17-1
  Candidate: 1.0.17-1
  Version table:
 *** 1.0.17-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

• pathlib2

This one must currently be a dependency of one of the packages that we only install on the bastions. Adding to the grid nodes should be simple.

$ apt-cache policy python-pathlib2
python-pathlib2:
  Installed: 2.2.0+really2.1.0-1
  Candidate: 2.2.0+really2.1.0-1
  Version table:
 *** 2.2.0+really2.1.0-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

• requests>=2.20.1

We need to pin this one to stretch-backports for a working version.

$ apt-cache policy python-requests
python-requests:
  Installed: 2.12.4-1
  Candidate: 2.12.4-1
  Version table:
     2.21.0-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 2.12.4-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

Supporting Python 2 is deprecated and will be dropped soon (T213287).

T248914: Information campaign and tech support needed to get Toolforge pywikibot use moved from py2 to py3 -- I don't think the Toolforge community has been well informed of this yet.

For Python 3 we only have

• requests>=2.20.1

We need to pin this one to stretch-backports for a working version.

$ apt-cache policy python3-requests
python3-requests:
  Installed: 2.12.4-1
  Candidate: 2.12.4-1
  Version table:
     2.21.0-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 2.12.4-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

Supporting Python 3.4 is deprecated and will be dropped soon (T239542).

The only place we have py3.4 in Toolforge is on the Kubernetes cluster via an old Jessie based image. We are hoping to deprecate that before July 2020. At the time I write this there are 107 pods on the Kubernetes cluster using the python3.4 image. It is actually scarily the 2nd most popular image. Php5 is the most popular and also one we need to deprecate.

Change 587894 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Update python dependencies for pywikibot

https://gerrit.wikimedia.org/r/587894

Change 587894 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: Update python dependencies for pywikibot

https://gerrit.wikimedia.org/r/587894

Change 587979 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [deps] Do not restrict ipaddress version dependency

https://gerrit.wikimedia.org/r/587979

Change 587979 merged by jenkins-bot:
[pywikibot/core@master] [deps] Do not restrict ipaddress version dependency

https://gerrit.wikimedia.org/r/587979