Following the strategy announcement by the HHVM team and subsequent discussion on pikitech-l, including benchmarks indicating HHVM no longer has a performance advantage, it is proposed that we migrate the WMF MediaWiki appservers from HHVM to PHP 7.

The current plan involves first migrating the appservers from jessie to stretch (T174431).

High-level checklist

Phase 1: PHP 7 and extensions/integrations installed:

• Plain PHP 7.0 available on the command-line on all mw servers (e.g. for manual use via mwscript).
• Deciding how to install PHP 7 and how to do CGI from Apache.
• Plain PHP 7.0 available on mwdebug servers via php-fpm.

Phase 2: Verification of PHP 7 interaction with MediaWiki.

Features to be installed, tested, and known to work correctly; any behaviour/config differences between PHP7 and HHVM to be equalised or justified and documented.

• Memcached client works fine with MW.
• mysqli driver works fine with MW.
• APCu works fine with MW.
• Session handler work fine with MW.
• Post-send functions works fine with MW. - T209981
• Error and fatal error reporting works fine with MW. – T187147
• Audit and fix as needed differences in INI settings. – T211488

Phase 3: Public opt-in via WikimediaDebug.

• Add nginx handling to expose PHP7 for opted-in requests.
• Provide opt-in via X-Wikimedia-Debug for mwdebug servers.
• Debug profiling works fine with MW (e.g. xhprof, or tideways) – T206152
• Sampling profiling works fine with MW. – T176916
• Any misc issues we find from opt-in tests and anything blocking from PHP 7.2 support.

Phase 4: Public opt-in via beta feature.

• Expose PHP7 vs. HHVM generation to JS in a header mw.config key.
• Look at whether we should distinguish NavTiming metrics based on whether they're served by php7 or HHVM. (Not sure whether this is valuable.)
• PHP 7.2 available on all mw servers via php-fpm (web, api, jobrunners, videoscalers)
• Provide opt-in via MediaWiki Beta Feature for any web/api appserver.
• Upgrade from PHP 7.0 to PHP 7.2.

Phase 5: PHP 7.2 on by default:

• Ramp up A/B test of the beta feature for web/api app servers.
• Switch default for all users on web appservers and api appservers.
• Switch mwscript CLI on maintenance hosts and deployment hosts. – T195392
• Switch job runners and video scalers.

Summary

In T176370#3651860, @faidon wrote:

Taking into account the lack funding for appserver work, as well as the end of the year fundraising and Christmas freezes, the (tentative!) timeline I proposed is:

• Upgrade the appserver fleet (w/ HHVM) to Debian stretch, including the ICU migration, in Q3 FY17-18 (circa February/March 2018)
• Begin PHP7 planning and initial implementation work in Q4 FY17-18, e.g. including a few test servers
• Fund the work in FY18-19 and complete it early in the year (Q1 or Q2 at the latest)

That's from an SRE perspective -- I think other tasks related to this migration (such as CI, beta, code/extension changes etc.) can proceed semi-independently. Given our velocity, I think we'll end up be the ones blocking others, rather than us waiting on these.

In T176370#3934066, @MoritzMuehlenhoff wrote:

During the SRE offsite/onsite we came up with the following plan:

• We need to remove PHP 5 usage by May 2018 (branch date for the next Mediawiki LTS)
  • Only open issue is dumps, which is showing good progress (needs wikidiff build and review of php7 puppet patch)

• We're planning the ICU migration (detailed in T177498) to happen this quarter:
  • HHVM package for icu57 needs to be upgraded to 3.18.8
  • Initial testing will happen on mwdebug
  • Also deployment-prep will be upgraded to the new packages (allows preparation of patches to use more recent Unicode)
  • For actual migration Community Liaisons need to be involved (user-visible effects)
  • First all app servers need to be upgraded, the run the conversion script (will take several days)

• Once we have upgraded to HHVM/ICU57 on jessie, we can upgrade to HHVM 3.18/stretch (we should be able to complete that by Q4)

• Once we're on stretch we can upgrade to HHVM 3.24

• We came up with the following migration plan towards PHP 7:
  • Running php7-fpm on different port
  • Depending on cookie route to HHVM or php7-fpm for initial testing
  • Add PHP7 as a beta feature for more widespread exposure
  • Gradually ramp up the PHP usage towards all users
  • If all goes fine we should be complete before the fundraising silent period, otherwise after that
  • Profiling is currently a known blocker – T176916