[go: up one dir, main page]
Page MenuHomePhabricator
Create Task
Maniphest T292239

Flaky "LogicException: Trying to delete mock tables" failure in MW integration test
Closed, ResolvedPublic
Actions

Description

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/721626 and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/725075, two completly different patches, both failed gate-and-submit with following unrelated error.

[b926580d664a526dfe80bc14] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database.
18:20:27 Backtrace:
18:20:27 from /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1626)
18:20:27 #0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1737): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseSqlite)
18:20:27 #1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(443): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseSqlite)
18:20:27 #2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(627): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
18:20:27 #3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(627): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
18:20:27 #4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(627): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
18:20:27 #5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(656): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
18:20:27 #6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
18:20:27 #7 /workspace/src/tests/phpunit/phpunit.php(141): PHPUnit\TextUI\Command->run(array, boolean)
18:20:27 #8 /workspace/src/tests/phpunit/phpunit.php(204): PHPUnitMaintClass->execute()
18:20:27 #9 {main}
18:20:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 PHP Fatal error:  Uncaught TypeError: Argument 1 passed to MediaWiki\Languages\LanguageConverterFactory::classFromCode() must be of the type string, null given, called in /workspace/src/includes/language/LanguageConverterFactory.php on line 130 and defined in /workspace/src/includes/language/LanguageConverterFactory.php:111
18:20:27 Stack trace:
18:20:27 #0 /workspace/src/includes/language/LanguageConverterFactory.php(130): MediaWiki\Languages\LanguageConverterFactory->classFromCode(NULL)
18:20:27 #1 /workspace/src/includes/ServiceWiring.php(969): MediaWiki\Languages\LanguageConverterFactory->getLanguageConverter()
18:20:27 #2 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(447): MediaWikiIntegrationTestCase::{closure}(Object(MediaWiki\MediaWikiServices))
18:20:27 #3 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService('MessageCache')
18:20:27 #4 /workspace/src/includes/MediaWikiServices.php(285): Wikimedia\Services\ServiceContainer->getService('MessageCache')
18:20:27 #5 /workspace/src/includes/MediaW in /workspace/src/includes/language/LanguageConverterFactory.php on line 111
18:20:27 Fatal error: Uncaught TypeError: Argument 1 passed to MediaWiki\Languages\LanguageConverterFactory::classFromCode() must be of the type string, null given, called in /workspace/src/includes/language/LanguageConverterFactory.php on line 130 and defined in /workspace/src/includes/language/LanguageConverterFactory.php:111
18:20:27 Stack trace:
18:20:27 #0 /workspace/src/includes/language/LanguageConverterFactory.php(130): MediaWiki\Languages\LanguageConverterFactory->classFromCode(NULL)
18:20:27 #1 /workspace/src/includes/ServiceWiring.php(969): MediaWiki\Languages\LanguageConverterFactory->getLanguageConverter()
18:20:27 #2 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(447): MediaWikiIntegrationTestCase::{closure}(Object(MediaWiki\MediaWikiServices))
18:20:27 #3 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService('MessageCache')
18:20:27 #4 /workspace/src/includes/MediaWikiServices.php(285): Wikimedia\Services\ServiceContainer->getService('MessageCache')
18:20:27 #5 /workspace/src/includes/MediaW in /workspace/src/includes/language/LanguageConverterFactory.php on line 111

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Ladsgroup added a project: Platform Engineering.Oct 1 2021, 9:26 PM

I assume this is a UBN ticket Platform engineering should take a look.

Jessica22664 updated the task description. (Show Details)Oct 3 2021, 12:26 AM
Jessica22664 subscribed.Oct 3 2021, 1:32 AM
DannyS712 updated the task description. (Show Details)Oct 3 2021, 3:23 AM
daniel subscribed.EditedOct 4 2021, 9:12 AM

The LogicException itself is just a symptom, I suspect the root cause is indicated by these warnings:

18:20:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4356
18:20:27 PHP Fatal error:  Uncaught TypeError: Argument 1 passed to MediaWiki\Languages\LanguageConverterFactory::classFromCode() must be of the type string, null given, called in /workspace/src/includes/language/LanguageConverterFactory.php on line 130 and defined in /workspace/src/includes/language/LanguageConverterFactory.php:111

This is quite curious, since $mCode is declared in the Language base class -- I can see how it might be null, but I don't see how it could be undefined. The Language class hasn't been touched since August either, and the relevant code not in a year...

$mCode used to be undeclared before I6bad608455 landed, it was only set by the constructor. But that patch landed more than a year ago...

Even when using a mock Language object and disabling the constructor, the field should be known, even if it's not initialized.

Can we see where this is happening? Is it a specific host? Could it be memory corruption? I'm seeing Building remotely on integration-agent-docker-1003 and Building remotely on integration-agent-docker-1004, but I don't know if that means the tests run on different hardware...

daniel added a subscriber: hashar.Oct 4 2021, 9:29 AM

Maybe @hashar has an idea?...

daniel added a comment.EditedOct 4 2021, 9:40 AM

Hm, Language has this destructor:

public function __destruct() {
		foreach ( $this as $name => $value ) {
			unset( $this->$name );
		}
	}

Running this will cause all fields to vanish, and then calling getCode() will produce the error we are seeing.
But the destructor should only be called by the PHP runtime after the object has become inaccessible by other code, so nothing should be able to call getCode() on it.
Unless the destructor is not called by the PHP runtime, but explicitly. I don't see where or why that might happen, but it's not impossible. But if this is the case, why only "sometimes"?...

Daimona added a comment.Oct 4 2021, 11:29 AM

I haven't looked at this specific bug, bug since @daniel mentioned Language::__destruct, I'd like to point out that this method has been responsible for some issues in our tests. There are some tests in MW core which would inadvertently mock the destructor, which caused the test suite to crash (T280780). This specific task was resolved, but perhaps some other test is having a similar issue. In particular, the LogicException here is causing the suite to crash as well, which is not something that's supposed to happen.

As to why it's only "sometimes", it depends on when exactly PHP is calling the destructor. For instance, in the original example (T280780) there was likely a cyclic dependency involved, so the bug would be triggered whenever the garbage collector kicked in. Thus, a way to make the failure more deterministic was to add a gc_collect_cycles() call after each test. I guess it might help here, so I'm going to push a test commit to do that.

gerritbot added a comment.Oct 4 2021, 11:33 AM

Change 725819 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] [DNM] Test build failure

https://gerrit.wikimedia.org/r/725819

gerritbot added a project: Patch-For-Review.Oct 4 2021, 11:33 AM
Mainframe98 subscribed.Oct 4 2021, 3:32 PM
Daimona added a comment.Oct 4 2021, 4:58 PM
In T292239#7398006, @Daimona wrote:

Thus, a way to make the failure more deterministic was to add a gc_collect_cycles() call after each test. I guess it might help here, so I'm going to push a test commit to do that.

Interestingly, it didn't help. It's not surprising though: the other bug was caused by the destructor being invoked in between tests (i.e. outside of the PHPUnit exception handling, which caused the suite to exit). Maybe this specific bug has a similar cause (the gc deletes an object which is still referenced), but it needs to happen in another moment. When exactly, I cannot tell.

Debugging here is complicated by two facts:

  1. The failure is intermittent, meaning we cannot keep removing tests or reverting code until it stops happening;
  2. The database suite consists of 9000 tests, which are kinda slow due to their nature, so things like repeating the suite X times to verify if a specific change fixed it is too slow.

I think a good plan could be to:

  1. Find out which test causes the exception
  2. Remove everything from the suite besides that test and a bunch (10-20) of tests that are executed before that one
  3. Run the remaining tests again and again until the error happens again (hoping that it *will* eventually happen)
  4. Then try to figure out how to make it deterministic
Krinkle renamed this task from Flaky mediawiki/core database integration test to Flaky "LogicException: Trying to delete mock tables" failure in MW integration test.Oct 4 2021, 11:03 PM
daniel lowered the priority of this task from Unbreak Now! to Needs Triage.Oct 5 2021, 7:19 AM

Lowering priority to "high". This should of course be fixed, but it doesn't seem any more than Selenium has always been. A handful of failures per day are certainly a problem, but the house is not on fire.

daniel triaged this task as High priority.Oct 5 2021, 7:19 AM
Daimona added a comment.Oct 5 2021, 12:04 PM
In T292239#7397714, @daniel wrote:

The LogicException itself is just a symptom, I suspect the root cause is indicated by these warnings:

My last experiments seems to show that this is not the case. I added the following code to Language::getCode():

if ( !isset( $this->mCode ) ) {
	fwrite( STDERR, ( new Exception )->getTraceAsString() . "\n" );
}

and something weird happened: the job failed again, with the same LogicException, but no references to Language::$mCode being unset! See https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-sqlite-php72-docker/11983/console. So the addition of the logging made the "undefined property" notice go away, AND nothing is written to stderr. It's as if the code detected that I was going to debug it and stopped failing ¯\_(ツ)_/¯

Aside from the Language thingy, I wanted to examine the LogicException, in particular why the DB prefix cause the assertion in MediaWikiIntegrationTestCase to fail, and I observed that $db->tablePrefix() is "parsertest_", while $this->dbPrefix() is unittest_. So maybe parser tests are failing to clean up the DB or something similar.

I think I'm not going to continue this investigation, but if someone wants to pick this up, https://gerrit.wikimedia.org/r/c/mediawiki/core/+/725819 can be used to test stuff.

Jdforrester-WMF subscribed.Oct 8 2021, 8:43 PM
Umherirrender subscribed.Oct 15 2021, 11:01 PM

It fails for postgres as well - https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-postgres-php72-docker/12058/console

gerritbot added a comment.Oct 15 2021, 11:26 PM

Change 731197 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Move use of LanguageConverterFactory into MessageCache

https://gerrit.wikimedia.org/r/731197

Umherirrender added a comment.Oct 15 2021, 11:28 PM
In T292239#7433396, @gerritbot wrote:

Change 731197 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Move use of LanguageConverterFactory into MessageCache

https://gerrit.wikimedia.org/r/731197

With my patch set the line of the stacktrace

#1 /workspace/src/includes/ServiceWiring.php(969): MediaWiki\Languages\LanguageConverterFactory->getLanguageConverter()

would be changed, because the call to the language converter is a bit later (in the constructor of MessageCache, not in the wiring code), not sure if that makes a difference for this task, but I have linked it here.

Umherirrender added a comment.Oct 20 2021, 7:27 PM

The failure today also includes a deprecation warning - https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-sqlite-php72-docker/12209/console

21:18:20 ..............................[036f97140da1e34cd3a2ed29] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database.
21:18:21 Backtrace:
21:18:21 from /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1626)
21:18:21 #0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1737): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseSqlite)
21:18:21 #1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(443): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseSqlite)
21:18:21 #2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(627): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
21:18:21 #3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(627): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
21:18:21 #4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(627): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
21:18:21 #5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(656): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
21:18:21 #6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
21:18:21 #7 /workspace/src/tests/phpunit/phpunit.php(141): PHPUnit\TextUI\Command->run(array, boolean)
21:18:21 #8 /workspace/src/tests/phpunit/phpunit.php(204): PHPUnitMaintClass->execute()
21:18:21 #9 {main}
21:18:21 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4368
21:18:21 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4368
21:18:21 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4368
21:18:21 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4368
21:18:21 PHP Fatal error:  Uncaught TypeError: Argument 1 passed to MediaWiki\Languages\LanguageConverterFactory::classFromCode() must be of the type string, null given, called in /workspace/src/includes/language/LanguageConverterFactory.php on line 130 and defined in /workspace/src/includes/language/LanguageConverterFactory.php:111
21:18:21 Stack trace:
21:18:21 #0 /workspace/src/includes/language/LanguageConverterFactory.php(130): MediaWiki\Languages\LanguageConverterFactory->classFromCode(NULL)
21:18:21 #1 /workspace/src/includes/ServiceWiring.php(979): MediaWiki\Languages\LanguageConverterFactory->getLanguageConverter()
21:18:21 #2 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(447): MediaWikiIntegrationTestCase::{closure}(Object(MediaWiki\MediaWikiServices))
21:18:21 #3 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService('MessageCache')
21:18:21 #4 /workspace/src/includes/MediaWikiServices.php(289): Wikimedia\Services\ServiceContainer->getService('MessageCache')
21:18:21 #5 /workspace/src/includes/MediaW in /workspace/src/includes/language/LanguageConverterFactory.php on line 111
21:18:21 Fatal error: Uncaught TypeError: Argument 1 passed to MediaWiki\Languages\LanguageConverterFactory::classFromCode() must be of the type string, null given, called in /workspace/src/includes/language/LanguageConverterFactory.php on line 130 and defined in /workspace/src/includes/language/LanguageConverterFactory.php:111
21:18:21 Stack trace:
21:18:21 #0 /workspace/src/includes/language/LanguageConverterFactory.php(130): MediaWiki\Languages\LanguageConverterFactory->classFromCode(NULL)
21:18:21 #1 /workspace/src/includes/ServiceWiring.php(979): MediaWiki\Languages\LanguageConverterFactory->getLanguageConverter()
21:18:21 #2 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(447): MediaWikiIntegrationTestCase::{closure}(Object(MediaWiki\MediaWikiServices))
21:18:21 #3 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService('MessageCache')
21:18:21 #4 /workspace/src/includes/MediaWikiServices.php(289): Wikimedia\Services\ServiceContainer->getService('MessageCache')
21:18:21 #5 /workspace/src/includes/MediaW in /workspace/src/includes/language/LanguageConverterFactory.php on line 111
21:18:21 
21:18:21 Deprecated: Use of ParserOutput::addWarning was deprecated in MediaWiki 1.38. [Called from ParserOutputTest::provideMergeInternalMetaDataFrom in /workspace/src/tests/phpunit/includes/parser/ParserOutputTest.php at line 832] in /workspace/src/includes/debug/MWDebug.php on line 375
Umherirrender added a comment.Oct 20 2021, 9:04 PM

The deprecation warning is unrelated, but even after https://gerrit.wikimedia.org/r/c/mediawiki/core/+/723688 the deprecation notices in dataProvider does not make the test fail.
I have created https://gerrit.wikimedia.org/r/c/mediawiki/core/+/732412 to hide the message

It seems the failure is always on the same test, because it always shows 95 % when failing.

Tks4Fish subscribed.Oct 23 2021, 5:14 PM
Krinkle subscribed.Dec 1 2021, 2:27 PM

Stil seen, e.g. job mediawiki-quibble-vendor-sqlite for change 661214.

Umherirrender added a comment.Jan 19 2022, 4:53 PM

Still seen https://gerrit.wikimedia.org/r/c/mediawiki/core/+/747945 - https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-sqlite-php72-docker/12996/console

gerritbot added a comment.Feb 12 2022, 3:31 PM

Change 725819 abandoned by Daimona Eaytoy:

[mediawiki/core@master] [DNM] Test build failure

Reason:

https://gerrit.wikimedia.org/r/725819

gerritbot added a comment.Feb 28 2022, 11:24 PM

Change 731197 merged by jenkins-bot:

[mediawiki/core@master] MessageCache: Move LanguageConverterFactory call from wiring to constructor

https://gerrit.wikimedia.org/r/731197

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.24; 2022-02-28).Mar 1 2022, 12:00 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 1 2022, 12:10 AM
Umherirrender added a comment.Mar 5 2022, 7:09 PM

Happen for mysql - https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/63650/console

9:54:50 ............................................................E 8540 / 8969 ( 95%)
19:55:08 EEEEEEEE.....................................[f98d5fb8e59bab2e47c59650] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database.
19:55:27 Backtrace:
19:55:27 from /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1645)
19:55:27 #0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1756): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseMysqli)
19:55:27 #1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(448): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseMysqli)
19:55:27 #2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
19:55:27 #3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
19:55:27 #4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
19:55:27 #5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(659): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
19:55:27 #6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
19:55:27 #7 /workspace/src/tests/phpunit/phpunit.php(140): PHPUnit\TextUI\Command->run(array, boolean)
19:55:27 #8 /workspace/src/tests/phpunit/phpunit.php(201): PHPUnitMaintClass->execute()
19:55:27 #9 {main}
19:55:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:55:27 Script php tests/phpunit/phpunit.php handling the phpunit:entrypoint event returned with error code 255
19:55:27 INFO:quibble.commands:<<< Finish: PHPUnit default suite (with database), in 307.999 s
gerritbot added a comment.Mar 5 2022, 7:49 PM

Change 725819 restored by Zabe:

[mediawiki/core@master] [DNM] Test build failure

https://gerrit.wikimedia.org/r/725819

gerritbot added a project: Patch-For-Review.Mar 5 2022, 7:49 PM
Zabe added a comment.Mar 5 2022, 8:16 PM

In https://gerrit.wikimedia.org/r/c/mediawiki/core/+/726002 the following error was thrown, showing that at least in this case it were the tests in MediaWikiIntegrationTestCaseSchema1Test that were failing.

19:45:25 ............................................................E 8540 / 8969 ( 95%)
19:45:40 EEEEEEEE.....................................[a80e5b581a86460abefa0956] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database. class: MediaWikiIntegrationTestCaseSchema1Test
19:45:58 Backtrace:
19:45:58 from /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1645)
19:45:58 #0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1756): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseMysqli)
19:45:58 #1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(448): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseMysqli)
19:45:58 #2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
19:45:58 #3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
19:45:58 #4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
19:45:58 #5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(659): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
19:45:58 #6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
19:45:58 #7 /workspace/src/tests/phpunit/phpunit.php(140): PHPUnit\TextUI\Command->run(array, boolean)
19:45:58 #8 /workspace/src/tests/phpunit/phpunit.php(201): PHPUnitMaintClass->execute()
19:45:58 #9 {main}
19:45:58 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 PHP Notice:  Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 Notice: Undefined property: LanguageEn::$mCode in /workspace/src/includes/language/Language.php on line 4361
19:45:58 Script php tests/phpunit/phpunit.php handling the phpunit:entrypoint event returned with error code 255
19:45:58 INFO:quibble.commands:<<< Finish: PHPUnit default suite (with database), in 285.495 s
Krinkle added a comment.Mar 6 2022, 12:47 AM

Still seen.
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/768258
https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/63681/

gerritbot added a comment.Mar 7 2022, 3:53 PM

Change 768748 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] language: Remove Language::__destruct()

https://gerrit.wikimedia.org/r/768748

Jdlrobson subscribed.EditedMar 10 2022, 5:20 PM

Seen on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/761503 which blocks 1.38 release.

Jdlrobson added a project: MW-1.38-release.Mar 10 2022, 5:20 PM
gerritbot added a comment.Mar 10 2022, 8:31 PM

Change 769775 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] tests: Add @group Broken to MediaWikiIntegrationTestCaseSchemaTest

https://gerrit.wikimedia.org/r/769775

matmarex raised the priority of this task from High to Unbreak Now!.Mar 15 2022, 1:35 AM

I am merging the patch to disable the test because this has made working on mediawiki/core absolutely unbearable. I often have to retry approving changes 2 or 3 times.

I assume the tests were important, therefore upgrading this to UBN until this is resolved and they are restored.

gerritbot added a comment.Mar 15 2022, 1:50 AM

Change 769775 merged by jenkins-bot:

[mediawiki/core@master] tests: Add @group Broken to MediaWikiIntegrationTestCaseSchemaTest

https://gerrit.wikimedia.org/r/769775

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.1; 2022-03-21-early).Mar 15 2022, 2:00 AM
matmarex added a comment.Mar 15 2022, 4:32 AM

That patch does not even help. I just got the same failure on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/770069: https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/64380/console

Daimona added a comment.Mar 15 2022, 8:26 AM
In T292239#7776586, @matmarex wrote:

That patch does not even help. I just got the same failure on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/770069: https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/64380/console

Heh, was going to say, I imagined that. The failure being nondeterministic makes me think that it's due to some code running at the wrong moment; the main things to blame would be the garbage collector or a destructor. Some early tests seemed to rule out the former, so I'd say it could be a destructor which gets called at the wrong moment, perhaps between tests. For instance, in the past we had issues because we'd mock Language::__destruct with createNoOpMock (which expects no method to be called), which sometimes would be called between tests, which caused the PHPUnit suite to die mysteriously without error messages. I think we may have something similar here. Perhaps MediaWikiIntegrationTestCaseSchema1Test is where the failure happens, but it's something else which leaves the system in an inconsistent status. My main proposal at this point would be to permanently add some logging to debug these failures, so that we can collect data when they happen. Unfortunately, any attempt to gather data with ad-hoc patches has failed miserably so far. Also, keep T292239#7401751 in mind. It seems that some logging code actually prevents the issue from happening, so perhaps we should start with minimal logging and expand it incrementally.

Umherirrender added a comment.Mar 15 2022, 11:11 PM

The error log shows:

mw-error.log
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   Wikimedia\Rdbms\DBUnexpectedError: MWExceptionHandler::rollbackPrimaryChangesAndLog: Database is owned by ID '1345848292' (got '1200445049')
#0 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1936): Wikimedia\Rdbms\Database->flushSession(string, integer)
#1 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2239): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1937): Wikimedia\Rdbms\LoadBalancer->forEachOpenPrimaryConnection(Closure)
#3 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(250): Wikimedia\Rdbms\LoadBalancer->flushPrimarySessions(string, integer)
#4 /workspace/src/includes/libs/rdbms/lbfactory/LBFactorySimple.php(145): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#5 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(252): Wikimedia\Rdbms\LBFactorySimple->forEachLB(Closure, array)
#6 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(346): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#7 /workspace/src/includes/exception/MWExceptionHandler.php(138): Wikimedia\Rdbms\LBFactory->flushPrimarySessions(string)
#8 /workspace/src/includes/exception/MWExceptionHandler.php(203): MWExceptionHandler::rollbackPrimaryChangesAndLog(LogicException, string)
#9 /workspace/src/includes/exception/MWExceptionHandler.php(172): MWExceptionHandler::handleException(LogicException, string)
#10 [internal function]: MWExceptionHandler::handleUncaughtException(LogicException)
#11 {main}
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database.
#0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1759): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseMysqli)
#1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(448): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
#3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(659): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
#7 /workspace/src/tests/phpunit/phpunit.php(140): PHPUnit\TextUI\Command->run(array, boolean)
#8 /workspace/src/tests/phpunit/phpunit.php(201): PHPUnitMaintClass->execute()
#9 {main}

Which can indicate the database instance has mixed its owner when teardown/setup. Different owner could have different table prefix.

Zabe added a comment.Mar 15 2022, 11:16 PM
In T292239#7780230, @Umherirrender wrote:

The error log shows:

mw-error.log
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   Wikimedia\Rdbms\DBUnexpectedError: MWExceptionHandler::rollbackPrimaryChangesAndLog: Database is owned by ID '1345848292' (got '1200445049')
#0 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1936): Wikimedia\Rdbms\Database->flushSession(string, integer)
#1 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2239): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1937): Wikimedia\Rdbms\LoadBalancer->forEachOpenPrimaryConnection(Closure)
#3 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(250): Wikimedia\Rdbms\LoadBalancer->flushPrimarySessions(string, integer)
#4 /workspace/src/includes/libs/rdbms/lbfactory/LBFactorySimple.php(145): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#5 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(252): Wikimedia\Rdbms\LBFactorySimple->forEachLB(Closure, array)
#6 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(346): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#7 /workspace/src/includes/exception/MWExceptionHandler.php(138): Wikimedia\Rdbms\LBFactory->flushPrimarySessions(string)
#8 /workspace/src/includes/exception/MWExceptionHandler.php(203): MWExceptionHandler::rollbackPrimaryChangesAndLog(LogicException, string)
#9 /workspace/src/includes/exception/MWExceptionHandler.php(172): MWExceptionHandler::handleException(LogicException, string)
#10 [internal function]: MWExceptionHandler::handleUncaughtException(LogicException)
#11 {main}
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database.
#0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1759): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseMysqli)
#1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(448): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
#3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(659): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
#7 /workspace/src/tests/phpunit/phpunit.php(140): PHPUnit\TextUI\Command->run(array, boolean)
#8 /workspace/src/tests/phpunit/phpunit.php(201): PHPUnitMaintClass->execute()
#9 {main}

Which can indicate the database instance has mixed its owner when teardown/setup. Different owner could have different table prefix.

Where did you saw this? Was the patch rebased over https://gerrit.wikimedia.org/r/c/mediawiki/core/+/770985? If not, that would explain the mixup of ownership.

Umherirrender added a comment.Mar 15 2022, 11:28 PM
In T292239#7780240, @Zabe wrote:
In T292239#7780230, @Umherirrender wrote:

The error log shows:

mw-error.log
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   Wikimedia\Rdbms\DBUnexpectedError: MWExceptionHandler::rollbackPrimaryChangesAndLog: Database is owned by ID '1345848292' (got '1200445049')
#0 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1936): Wikimedia\Rdbms\Database->flushSession(string, integer)
#1 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2239): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1937): Wikimedia\Rdbms\LoadBalancer->forEachOpenPrimaryConnection(Closure)
#3 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(250): Wikimedia\Rdbms\LoadBalancer->flushPrimarySessions(string, integer)
#4 /workspace/src/includes/libs/rdbms/lbfactory/LBFactorySimple.php(145): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#5 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(252): Wikimedia\Rdbms\LBFactorySimple->forEachLB(Closure, array)
#6 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(346): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#7 /workspace/src/includes/exception/MWExceptionHandler.php(138): Wikimedia\Rdbms\LBFactory->flushPrimarySessions(string)
#8 /workspace/src/includes/exception/MWExceptionHandler.php(203): MWExceptionHandler::rollbackPrimaryChangesAndLog(LogicException, string)
#9 /workspace/src/includes/exception/MWExceptionHandler.php(172): MWExceptionHandler::handleException(LogicException, string)
#10 [internal function]: MWExceptionHandler::handleUncaughtException(LogicException)
#11 {main}
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database.
#0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1759): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseMysqli)
#1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(448): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
#3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(659): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
#7 /workspace/src/tests/phpunit/phpunit.php(140): PHPUnit\TextUI\Command->run(array, boolean)
#8 /workspace/src/tests/phpunit/phpunit.php(201): PHPUnitMaintClass->execute()
#9 {main}

Which can indicate the database instance has mixed its owner when teardown/setup. Different owner could have different table prefix.

Where did you saw this? Was the patch rebased over https://gerrit.wikimedia.org/r/c/mediawiki/core/+/770985? If not, that would explain the mixup of ownership.

That is from https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/64483/artifact/log/mw-error.log which was https://gerrit.wikimedia.org/r/c/mediawiki/core/+/763894

It runs after your mention patch set. That patch set fixed the null in the error message (before that patch set the (got '...') part was empty).

But it is possible the error message is unrelated, see a possible fix with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/771081

Zabe added a comment.Mar 15 2022, 11:29 PM
In T292239#7780252, @Umherirrender wrote:
In T292239#7780240, @Zabe wrote:
In T292239#7780230, @Umherirrender wrote:

The error log shows:

mw-error.log
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   Wikimedia\Rdbms\DBUnexpectedError: MWExceptionHandler::rollbackPrimaryChangesAndLog: Database is owned by ID '1345848292' (got '1200445049')
#0 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1936): Wikimedia\Rdbms\Database->flushSession(string, integer)
#1 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2239): Wikimedia\Rdbms\LoadBalancer::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1937): Wikimedia\Rdbms\LoadBalancer->forEachOpenPrimaryConnection(Closure)
#3 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(250): Wikimedia\Rdbms\LoadBalancer->flushPrimarySessions(string, integer)
#4 /workspace/src/includes/libs/rdbms/lbfactory/LBFactorySimple.php(145): Wikimedia\Rdbms\LBFactory::Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#5 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(252): Wikimedia\Rdbms\LBFactorySimple->forEachLB(Closure, array)
#6 /workspace/src/includes/libs/rdbms/lbfactory/LBFactory.php(346): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#7 /workspace/src/includes/exception/MWExceptionHandler.php(138): Wikimedia\Rdbms\LBFactory->flushPrimarySessions(string)
#8 /workspace/src/includes/exception/MWExceptionHandler.php(203): MWExceptionHandler::rollbackPrimaryChangesAndLog(LogicException, string)
#9 /workspace/src/includes/exception/MWExceptionHandler.php(172): MWExceptionHandler::handleException(LogicException, string)
#10 [internal function]: MWExceptionHandler::handleUncaughtException(LogicException)
#11 {main}
2022-03-15 22:59:40 789e8f299681 wikidb-parsertest_: [c82384f2438996bd113eabec] [no req]   LogicException: Trying to delete mock tables, but table prefix does not indicate a mock database.
#0 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(1759): MediaWikiIntegrationTestCase->ensureMockDatabaseConnection(Wikimedia\Rdbms\DatabaseMysqli)
#1 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php(448): MediaWikiIntegrationTestCase->setUpSchema(Wikimedia\Rdbms\DatabaseMysqli)
#2 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): MediaWikiIntegrationTestCase->run(PHPUnit\Framework\TestResult)
#3 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#4 /workspace/src/vendor/phpunit/phpunit/src/Framework/TestSuite.php(626): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#5 /workspace/src/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(659): PHPUnit\Framework\TestSuite->run(PHPUnit\Framework\TestResult)
#6 /workspace/src/vendor/phpunit/phpunit/src/TextUI/Command.php(235): PHPUnit\TextUI\TestRunner->doRun(PHPUnit\Framework\TestSuite, array, array, boolean)
#7 /workspace/src/tests/phpunit/phpunit.php(140): PHPUnit\TextUI\Command->run(array, boolean)
#8 /workspace/src/tests/phpunit/phpunit.php(201): PHPUnitMaintClass->execute()
#9 {main}

Which can indicate the database instance has mixed its owner when teardown/setup. Different owner could have different table prefix.

Where did you saw this? Was the patch rebased over https://gerrit.wikimedia.org/r/c/mediawiki/core/+/770985? If not, that would explain the mixup of ownership.

That is from https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/64483/artifact/log/mw-error.log which was https://gerrit.wikimedia.org/r/c/mediawiki/core/+/763894

It runs after your mention patch set. That patch set fixed the null in the error message (before that patch set the (got '...') part was empty).

But it is possible the error message is unrelated, see a possible fix with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/771081

Right. Still it seems to be a fallout from https://gerrit.wikimedia.org/r/c/mediawiki/core/+/737824.

gerritbot added a comment.Mar 16 2022, 3:08 AM

Change 771091 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] Improve ensureMockDatabaseConnection() error message

https://gerrit.wikimedia.org/r/771091

gerritbot added a comment.Mar 16 2022, 3:16 AM

Change 771081 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] rdbms: use the LoadBalancer id in flushPrimarySessions()

https://gerrit.wikimedia.org/r/771081

Func subscribed.Mar 16 2022, 6:28 AM
gerritbot added a comment.Mar 16 2022, 12:38 PM

Change 771091 merged by jenkins-bot:

[mediawiki/core@master] Improve ensureMockDatabaseConnection() error message

https://gerrit.wikimedia.org/r/771091

matmarex added a comment.EditedMar 16 2022, 3:52 PM

Thanks for working on this. These patches do not fix the bug either. We can't merge https://gerrit.wikimedia.org/r/771081 because it keeps failing the tests with the same error message.

• Ladsgroup added a comment.Mar 16 2022, 3:56 PM

Please revert https://gerrit.wikimedia.org/r/c/mediawiki/core/+/737824 and its follow ups if this seems to be caused by that patch.

aaron subscribed.Mar 16 2022, 4:48 PM
In T292239#7782427, @matmarex wrote:

Thanks for working on this. These patches do not fix the bug either. We can't merge https://gerrit.wikimedia.org/r/771081 because it keeps failing the tests with the same error message.

Yeah, I wasn't sure if this was related, but I saw the other trace on the tests failure hear.

aaron added a comment.EditedMar 16 2022, 4:49 PM

Having tweaked the error message, I now see:

LogicException: Trying to delete mock tables, but table prefix 'parsertest_' does not indicate a mock database (expected 'unittest_')

I also noticed:

[DBQuery] SQL ERROR (ignored): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'RELEASE_ALL_LOCKS() /* Wikimedia\Rdbms\DatabaseMysqlBase::doFlushSession  */' at line 1 (localhost:/workspace/db/quibble-mysql-0b4vpcqj/socket)
[objectcache] MainWANObjectCache using store EmptyBagOStuff
[DBQuery] CloneDatabase::destroy [0s] localhost:/workspace/db/quibble-mysql-0b4vpcqj/socket: DROP TABLE `parsertest_actor` CASCADE

I updated the patch to fix that.

aaron added a comment.Mar 16 2022, 5:49 PM

Patch now gets:

SQL ERROR (ignored): FUNCTION wikidb.RELEASE_ALL_LOCKS does not exist (localhost:/workspace/db/quibble-mysql-5z8kvkdz/socket)

This is due to the mariadb version not supporting that method (https://jira.mariadb.org/browse/MDEV-10569). Mine does, locally, and mysql has since 2014. I'll have to use RELEASE_LOCK() then.

gerritbot added a comment.Mar 16 2022, 9:49 PM

Change 771467 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] tests: Fix @group Broken on MediaWikiIntegrationTestCaseSchemaTest

https://gerrit.wikimedia.org/r/771467

gerritbot added a comment.Mar 16 2022, 10:06 PM

Change 771393 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@REL1_38] tests: Fix @group Broken on MediaWikiIntegrationTestCaseSchemaTest

https://gerrit.wikimedia.org/r/771393

gerritbot added a comment.Mar 16 2022, 10:18 PM

Change 771393 merged by jenkins-bot:

[mediawiki/core@REL1_38] tests: Fix @group Broken on MediaWikiIntegrationTestCaseSchemaTest

https://gerrit.wikimedia.org/r/771393

gerritbot added a comment.Mar 16 2022, 10:21 PM

Change 771475 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] Improve handling of test class-level setup failures in run()

https://gerrit.wikimedia.org/r/771475

tstarling subscribed.Mar 16 2022, 10:41 PM

Note that https://gerrit.wikimedia.org/r/c/mediawiki/core/+/769152 was merged a week ago, which potentially changes when CloneDatabase teardown is done.

ReleaseTaggerBot edited projects, added MW-1.38-notes; removed MW-1.38-notes (1.38.0-wmf.24; 2022-02-28).Mar 16 2022, 11:00 PM
tstarling added a comment.EditedMar 16 2022, 11:11 PM

In the debug log https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-php72-docker/64615/artifact/log/mw-debug-cli.log.gz

  • line 169360: initial unittest_ clone
  • line 953487: unittest_ teardown
  • line 953549: parsertest_ clone
  • line 995129: parsertest_ teardown
  • line 995191: parsertest_ clone again. Why?
  • line 995457: This is ParserTestRunner::setupUploads(), so parser tests are being set up a second time. Explains where the prefix comes from.
  • line 996370: fortuitous performance assertion provides backtrace showing second parser test setup. PHPUnit is setting up ParserTestTopLevelSuite again.
  • line 996541: Start of ContentHandlerFunctionalTest, but ParserTestTopLevelSuite::teardown() has not been called.
  • line 996595: example of query from non-parsertest test being done with parsertest_ prefix. The test is ContentHandlerFunctionalTest::testGetParserOutput.
  • line 1004269: BackupDumperPageTest fails with error "Empty list of tables to clone"
  • line 1018870: parsertest_ teardown
gerritbot added a comment.Mar 16 2022, 11:11 PM

Change 771467 merged by jenkins-bot:

[mediawiki/core@master] tests: Fix @group Broken on MediaWikiIntegrationTestCaseSchemaTest

https://gerrit.wikimedia.org/r/771467

tstarling added a comment.Mar 17 2022, 2:53 AM

The second ParserTestTopLevelSuite setup is for ExtensionsParserTestSuite. My theory is that CoreParserTestSuite succeeds, but ExtensionsParserTestSuite sets up a new cloned database and fails to tear it down.

An exception/throwable from setUp() in SuiteEventsTrait is supposed to be logged as a test failure, but this suite has no enabled tests, so the exception would be silently discarded. If an exception is thrown after the assignment to MediaWikiIntegrationTestCase::$dbClone, teardown is done when the next test starts. I tried modelling the failure by throwing an exception between database cloning and assignment, but the resulting failure was not exactly the same as what we see here.

A fix for both the discarded exception and the unnecessary database clone would be to have SuiteEventsTrait skip setUp()/tearDown() when the suite is empty. That might be good enough as a workaround for the bug itself.

gerritbot added a comment.Mar 17 2022, 3:36 AM

Change 771081 merged by jenkins-bot:

[mediawiki/core@master] rdbms: fix owner id and RELEASE_ALL_LOCKS query in session flushing methods

https://gerrit.wikimedia.org/r/771081

gerritbot added a comment.Mar 17 2022, 3:47 AM

Change 771496 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] SuiteEventsTrait: don't call setUp() for an empty suite

https://gerrit.wikimedia.org/r/771496

gerritbot added a comment.Mar 17 2022, 8:40 AM

Change 771398 had a related patch set uploaded (by Hashar; author: Umherirrender):

[mediawiki/core@wmf/1.38.0-wmf.26] tests: Fix @group Broken on MediaWikiIntegrationTestCaseSchemaTest

https://gerrit.wikimedia.org/r/771398

gerritbot added a comment.Mar 17 2022, 10:31 AM

Change 771496 merged by jenkins-bot:

[mediawiki/core@master] SuiteEventsTrait: don't call setUp() for an empty suite

https://gerrit.wikimedia.org/r/771496

gerritbot added a comment.Mar 17 2022, 5:35 PM

Change 770938 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@wmf/1.38.0-wmf.26] rdbms: fix owner id and RELEASE_ALL_LOCKS query in session flushing methods

https://gerrit.wikimedia.org/r/770938

gerritbot added a comment.Mar 17 2022, 7:55 PM

Change 771662 had a related patch set uploaded (by Ladsgroup; author: Tim Starling):

[mediawiki/core@wmf/1.38.0-wmf.26] SuiteEventsTrait: don't call setUp() for an empty suite

https://gerrit.wikimedia.org/r/771662

gerritbot added a comment.Mar 17 2022, 8:11 PM

Change 771662 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.26] SuiteEventsTrait: don't call setUp() for an empty suite

https://gerrit.wikimedia.org/r/771662

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.26; 2022-03-14); removed MW-1.38-notes.Mar 17 2022, 9:00 PM
gerritbot added a comment.Mar 17 2022, 9:17 PM

Change 770938 abandoned by Aaron Schulz:

[mediawiki/core@wmf/1.38.0-wmf.26] rdbms: fix owner id and RELEASE_ALL_LOCKS query in session flushing methods

Reason:

https://gerrit.wikimedia.org/r/770938

matmarex added a comment.Mar 17 2022, 11:02 PM

It looks like this is fixed by Tim's patch – I've merged a couple of changes in mediawiki/core today and none of them failed with this error message. Thank you!

gerritbot added a comment.Mar 17 2022, 11:02 PM

Change 771398 abandoned by Bartosz Dziewoński:

[mediawiki/core@wmf/1.38.0-wmf.26] tests: Fix @group Broken on MediaWikiIntegrationTestCaseSchemaTest

Reason:

No longer needed

https://gerrit.wikimedia.org/r/771398

gerritbot added a comment.Mar 17 2022, 11:02 PM

Change 771706 had a related patch set uploaded (by Bartosz Dziewoński; author: Umherirrender):

[mediawiki/core@master] Revert "tests: Add @group Broken to MediaWikiIntegrationTestCaseSchemaTest"

https://gerrit.wikimedia.org/r/771706

gerritbot added a comment.Mar 17 2022, 11:09 PM

Change 771714 had a related patch set uploaded (by Zabe; author: Umherirrender):

[mediawiki/core@REL1_38] Revert "tests: Add @group Broken to MediaWikiIntegrationTestCaseSchemaTest"

https://gerrit.wikimedia.org/r/771714

gerritbot added a comment.Mar 17 2022, 11:19 PM

Change 771706 merged by jenkins-bot:

[mediawiki/core@master] Revert "tests: Add @group Broken to MediaWikiIntegrationTestCaseSchemaTest"

https://gerrit.wikimedia.org/r/771706

gerritbot added a comment.Mar 17 2022, 11:20 PM

Change 771714 merged by jenkins-bot:

[mediawiki/core@REL1_38] Revert "tests: Add @group Broken to MediaWikiIntegrationTestCaseSchemaTest"

https://gerrit.wikimedia.org/r/771714

ReleaseTaggerBot edited projects, added MW-1.38-notes; removed MW-1.38-notes (1.38.0-wmf.26; 2022-03-14).Mar 18 2022, 12:00 AM
gerritbot added a comment.Mar 19 2022, 12:57 PM

Change 725819 abandoned by Daimona Eaytoy:

[mediawiki/core@master] [DNM] Test build failure

Reason:

Fixed for real now.

https://gerrit.wikimedia.org/r/725819

tstarling closed this task as Resolved.Mar 20 2022, 11:10 PM
tstarling claimed this task.

Assuming fixed, please reopen if you see it again.

gerritbot added a comment.Mar 23 2022, 12:59 AM

Change 771475 merged by jenkins-bot:

[mediawiki/core@master] Improve handling of class-level test setup failures in run()

https://gerrit.wikimedia.org/r/771475

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.4; 2022-03-23); removed MW-1.39-notes (1.39.0-wmf.1; 2022-03-21-early).Mar 23 2022, 1:00 AM
gerritbot added a comment.Apr 27 2022, 12:01 AM

Change 768748 merged by jenkins-bot:

[mediawiki/core@master] language: Remove Language::__destruct()

https://gerrit.wikimedia.org/r/768748

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.10; 2022-05-02); removed MW-1.39-notes (1.39.0-wmf.4; 2022-03-23).Apr 27 2022, 1:01 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits