Service to let users execute SQL queries against Wikipedia databases on Wikimedia Cloud Services (Homepage).
Analytics projects aim to give the wiki movement a data services platform: Providing insight into community activity (Data Engineering team).
The move to k8s appears to have made this ticket mostly unactionable.
I believe PII such as email addresses, password hashes, and IPs is scrubbed by the replicas? Quarry isn't a system I think of as having PII in it. All the data it queries is public, I think.
Though none of it gets at the central issue of PII
I appreciate the commentary. Though none of it gets at the central issue of PII, and the reality that quarry is not designed to keep data in perpetuity. Data persistence is an expensive process and not being applied to quarry as such we're one system crash from the data being gone regardless. There are the current download options to export query results. If additional export options are desired, patches are welcome.
Forgive the drive-by comment, but at the 6-month anniversary of this ticket, it might be worth checking how our upstream production applications (such as gitlab, netbox etc) are handling this change, if it all. For example, I noticed that netbox-docker is now using valkey .
It is possible that you were encountering the three hour time limit for analytics searches. If there was some lag it could have increased your query time from what looks like an hour to later. I'm unsure of how additional data could be provided, though it may be possible. Likely though it is easier to check https://replag.toolforge.org/ for lag which if there was much would suggest long running queries may not complete.
And now Quarry is working again.
Seems like much less or no issues now. In any case, please add some info when queries are stopped.
Please prevent queries from getting stopped. One went through but the other still gets stopped all the time.
@rook Yes, I did not press the stop button for any of the queries that were stopped and it only displays the above two lines and not any further info like some error code. Other example: https://quarry.wmcloud.org/query/86864
It's been awhile since I've looked at that code. When I worked on it it was to have the stopped status appear when someone manually presses the "stop" button, I thought I added it just for that, but maybe it existed for something else as well. So, ideally, the status doesn't mean to indicate that the query failed or otherwise couldn't run, just that it was stopped while it was running by the user. Are you seeing your queries end with this status without pressing the stop button?
If you do do this, it would be good to only remove the older runs results, but leave the most recent run result for each query, or as a less desirable alternative to keep those for only published queries (but I am one of the people who rarely publishes queries even when they are finished). My queries often refer to some article contest, or some on wiki investigation, or just a bit of curiosity that does make sense to be kept for many years and is sometimes linked with the assumption of some stability from on-wiki. Re-running queries every 90 days would be an unnecessary burden, might lead to corrupted results, such as a query that was used to determine a contest winner will no longer should that person as winner because of some things happening in between, and because of schema changes it might also be a maintenance burden (I still have a lot of query results active from the pre-multi-DB select cancellation times, or from the pre actor migration times) that either require a tweak or are even impossible to rewrite, but the results of their older runs are somewhat useful to this day.
Quarry is working again. Though I didn't have time to investigate what is happening so this may happen again. Opening T375997 to investigate the underlying issue.
Looks like k8s is having trouble with garbage collection
Warning FreeDiskSpaceFailed 118s (x159 over 13h) kubelet Failed to garbage collect required amount of images. Attempted to free 4281512755 bytes, but only found 0 bytes eligible to free.
Works for me now, thanks \o/
supertassu closed https://github.com/toolforge/quarry/pull/70
supertassu opened https://github.com/toolforge/quarry/pull/70
Still happening. I got redirected like this:
Though the error Firefox shows me is different:
I have experience with three different types of queries where keeping the results has been important.