Just writing this down for reference as it was annoying to figure out, I think the memcached key in question is dewiki:abusefilter:throttle:290:user,page:ce12284db0992734fd982f4a3ddd1f5f2b5c5408 (aka dewiki:abusefilter:throttle:290:user,page:sha1( "404212:Wikipedia Diskussion:Schiedsgericht/Anfragen/Diskussionskomplexim Umfeld \"Anetta Kahane\"" ) )

To summarize what's been done:

• The obvious suspect is a rogue cache key.
• Specifically, a key that may have been evicted before its expiry
• Another idea was that a key expired before reaching 5, and the new key also didn't reach fine, although they totaled more than 5 edits on Dec 17. However, this doesn't seem to be the case: an edit was prevented at 6:58 (hence the key was recreated at 8:00), but there've been more than 5 edits in the next 24 hours.

To understand exactly what happened, we'd need to see the debug logs (channel:AbuseFilter) related to that key (like it was done at T210709#4811680). Grepping the key posted by @Bawolff above would be the best choice; however, since we don't know whether the key is correct, it could also suffice to grep dewiki:abusefilter:throttle:290:user,page:.

Yesterday, @Reedy and @Bawolff tried to do that, but I had to leave suddenly and I don't know whether there's been any progress.

As a side note, it's vital to investigate soon and quickly, because if the logs get garbage collected, we'll have no way to understand what went wrong.

As a side note, it's vital to investigate soon and quickly, because if the logs get garbage collected, we'll have no way to understand what went wrong.

Logs (of the mwlog1001 variety) last 90 days. That said, even if we do confirm the key went missing, im not sure what anyone would be able to do sbout

In T240951#5751324, @Bawolff wrote:

As a side note, it's vital to investigate soon and quickly, because if the logs get garbage collected, we'll have no way to understand what went wrong.

Logs (of the mwlog1001 variety) last 90 days. That said, even if we do confirm the key went missing, im not sure what anyone would be able to do sbout

And AFAIK (to state the obvious a bit), cached items aren't guaranteed to still exist upto $expiry are they? They can be evicted for $reasons (as dictated by the config of whatever cache backend we're using) and shouldn't be guaranteed to be still there...

In T240951#5751324, @Bawolff wrote:

As a side note, it's vital to investigate soon and quickly, because if the logs get garbage collected, we'll have no way to understand what went wrong.

Logs (of the mwlog1001 variety) last 90 days.

Thanks, I thought they had a shorter lifespan.

In T240951#5752473, @Reedy wrote:

In T240951#5751324, @Bawolff wrote:

Logs (of the mwlog1001 variety) last 90 days. That said, even if we do confirm the key went missing, im not sure what anyone would be able to do sbout

And AFAIK (to state the obvious a bit), cached items aren't guaranteed to still exist upto $expiry are they? They can be evicted for $reasons (as dictated by the config of whatever cache backend we're using) and shouldn't be guaranteed to be still there...

Yes, sure. In fact, my main goal here is to make sure that this was just due to memcached evicting the key too soon. Of course, we could make this system more robust, but this task can be closed as soon as we know for sure that nothing is broken in the AF code.

Happened again today, for the same user:

Note: It did work before for him: https://de.wikipedia.org/w/index.php?title=Spezial:Missbrauchsfilter-Logbuch&wpSearchUser=Brainswiffer&uselang=en

Hello, its "me myself". Filter sometimes "blocks" me but sometimes not. Yesterday 17.12 about 11:25 MEZ it showed "blocking" in browser, but this is not in log of filter. After that I repeated storing and it stored all without message. Today it seems that I work without limit but will not test again. I will rely on filter! There are not special circumstances here despite of hanging on a very fast line :-)
Filter has no data when he decides?

it seems not to be limited to myself, also User Kurator71 has too much edits today.
I often work with 3 machines (IPhone, IPad, PC) and switch during day several times, this may be also important?

Same for this user:

In T240951#5754702, @Brainswiffer wrote:

I often work with 3 machines (IPhone, IPad, PC) and switch during day several times, this may be also important?

No, this shouldn't make any difference.

Now, I believe this is happening too often. Of course, it's still possible that memcached is evicting keys too quickly, but I wonder whether we could at least put a stopgap in place.

For the short term, we only need to check the debug logs buried on mwlog to see if it's really memcached's fault. For the long term, it depends... T52894 is for making throttle work without caching enabled, but I wonder whether there would be an even better alternative.

Happend again today, 5 edits were allowed, 7 were made, no one was blocked:

Thanks to @Urbanecm, we dug through the logs. First of all, Brainswiffer's key is dewiki:abusefilter:throttle:290:user,page:4cab7fa217da080aea32c3fb8145f66d0af59989. We've checked all the logs for 2020, and here is the cleaned result:

(derived from P10041).

This tells us that yes, the key is being evicted for some reason. In fact, if you look at the top of the paste, a new key was created at 5:57:19, incremented twice, but at 13:54:00 the key was missing. That's 8 hours later, hence the key was deleted in the meanwhile, before its expiry. The only possible explanation for this is memcached auto-evicting the key, as nothing in the AF code tries to delete those keys.

If this is confirmed, then I don't think we can do much, aside from T52894.

My final words: caused by the eviction of the key. Note, the backend used for throttle keys is Redis and not memcached. I believe there's nothing we can do here. The best outcome would be to find a way to make throttle work without caching (but how?), or open a follow-up task to investigate whether it's possible to increase the persistence, and hence reduce the chance of eviction.

Possibly related: https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard##WPWP_is_back

Throttle filter at enwiki (https://en.wikipedia.org/wiki/Special:AbuseFilter/1158) which isn't throttling, even though there is a match in the corresponding log filter with the same filter code (https://en.wikipedia.org/wiki/Special:AbuseFilter/1073). Sample contribs: https://en.wikipedia.org/w/index.php?title=Special:Contributions&offset=20210829133745&target=Olatunbosun%20opeyemi%20David

Same issue?

In T240951#7317222, @Proc wrote:

Possibly related: https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard##WPWP_is_back

Throttle filter at enwiki (https://en.wikipedia.org/wiki/Special:AbuseFilter/1158) which isn't throttling, even though there is a match in the corresponding log filter with the same filter code (https://en.wikipedia.org/wiki/Special:AbuseFilter/1073). Sample contribs: https://en.wikipedia.org/w/index.php?title=Special:Contributions&offset=20210829133745&target=Olatunbosun%20opeyemi%20David

Same issue?

[offtopic] Do note that currently, the filter is set to throttle by username plus IP address. That means users using highly dynamic ranges might never make it to the throttle.

In T240951#7317243, @Urbanecm wrote:

[offtopic] Do note that currently, the filter is set to throttle by username plus IP address. That means users using highly dynamic ranges might never make it to the throttle.

Right, that's also a valid point (I just understood that the question wasn't really about the number of entries). I'm not sure if I can query the DB to see which IPs that editor used, which in turn is needed for checking the throttle key.

In T240951#7317248, @Daimona wrote:

In T240951#7317243, @Urbanecm wrote:

[offtopic] Do note that currently, the filter is set to throttle by username plus IP address. That means users using highly dynamic ranges might never make it to the throttle.

Right, that's also a valid point (I just understood that the question wasn't really about the number of entries). I'm not sure if I can query the DB to see which IPs that editor used, which in turn is needed for checking the throttle key.

I did that already, and at least one of the WPWP users do use a lot of IPs (like dozens of them).

Also, now that we're talking about the issue here anyway, I'm linking https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/715339 here as a patch to make user create separate throttle counts for individual IPs, which could be helpful for cases like this one. Of course, that behavior might be intentional -- feel free to abandon if that's so.