meta.wikimedia.org very strangely redirecting to wikimediafoundation.org
Closed, ResolvedPublic

Description

mzmcbride@tools-login:~$ curl "https://meta.wikimedia.org/wiki/Special:Contributions/MZMcBride"
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://wikimediafoundation.org/wiki/Special:Contributions/MZMcBride">here</a>.</p>
</body></html>

mzmcbride@tools-login:~$ curl -I "https://meta.wikimedia.org/wiki/Special:Contributions/MZMcBride"
HTTP/1.1 301 Moved Permanently
Server: nginx/1.1.19
Date: Tue, 22 Oct 2013 17:11:11 GMT
Content-Type: text/html; charset=iso-8859-1
Connection: keep-alive
Location: http://wikimediafoundation.org/wiki/Special:Contributions/MZMcBride
X-Cache: MISS from cp1016.eqiad.wmnet
X-Cache-Lookup: MISS from cp1016.eqiad.wmnet:3128
X-Cache: MISS from cp1002.eqiad.wmnet
X-Cache-Lookup: MISS from cp1002.eqiad.wmnet:80

I have no idea what's going on here.


Version: wmf-deployment
Severity: major

Details

Reference
bz56006
bzimport set Reference to bz56006.
bzimport added a subscriber: Unknown Object (MLST).

According to Gerrit changeset 90669, "apache-fast-test wwwportals.url mw1044" was run prior to deploying these changes. We'll need to expand our test coverage to ensure that we don't do this again. :-)

And we'll need to purge Squid caches (and possibly other caches). Even as a logged-in user over HTTPS, I'm still getting mis-redirected.

bd808 added a comment.Oct 22 2013, 5:53 PM

[[wikitech:Server_admin_log]]

17:23 mark: Banned Content-Type == "application/x-httpd-php" in text Varnish (eqiad)

Reedy added a comment.Oct 22 2013, 5:54 PM
  • Bug 56007 has been marked as a duplicate of this bug. ***
bd808 added a comment.Oct 22 2013, 7:31 PM

Reedy announced a possible fix having been found in #wikimedia-tech. I'm not seeing this issue continue after clearing my local browser cache. It did require a local cache clear however. Prior to that I was still getting redirects to the Foundation site for many pages I had tried to visit earlier today.

sarahstierch wrote:

When I go to specific pages, it's OK. but when I type in the generic meta.wikiemdia.org I get a file to download. I've cleaned my cache a million times.

bd808 added a comment.Oct 22 2013, 7:56 PM

It looks like a few varnish cache servers still have bad content cached. Mark's content-type ban isn't working for them because some content was cached as text/html instead of application/x-httpd-php.

bd808 added a comment.Oct 22 2013, 8:31 PM

[[wikitech:Server_admin_log]]

20:30 bblack: varnish: banned 'obj.http.content-length == 33518' on text varnishes everywhere (extract2.php leakage)

pasting my post from wikitech on this bug for completeness:


As part of an ongoing effort to simplify, cleanup and reduce code lines of the Apache cluster config,
we we're planning to unify a lot of document roots for the "www"-portals into a single docroot, like here:

https://gerrit.wikimedia.org/r/#/c/90669/

looking at that we saw that the www.wikimedia.org portal wasn't handled by this, unlike the other portals, so
to further unify this we merged:

https://gerrit.wikimedia.org/r/#/c/91195/

which "just" moved the existing config for www.wikimedia.org to the wwwportals.conf file.

But because this had an old " ServerAlias *.wikimedia.org" in it, which now changed in the order Apache goes through the config,
it caused *.wikimedia.org URLs to redirect to wikimediafoundation.org

..
ServerAlias *.wikimedia.org
..
RewriteRule ^/wiki/(.*)$ http://wikimediafoundation.org/wiki/$1 [R=301,L]

We reverted both of the recent changes after just a few minutes and synced Apache config and restarted them.

Users still reported problems though due to caching. So we started some Squid purging and Mark banned php content-type in varnish.

Additionally it turned out several Apaches didn't get restarted properly by apache-graceful-all, and using apache-fast-test with the pybal option we found some more that needed manual restarts a little while later.

For futher fixes Brandon banned by object size: did < bblack> !log varnish: banned 'obj.http.content-length == 33518' on text varnishes everywhere (extract2.php leakage).
.. Reedy already prepared new patches to fix the root cause
..and apache-fast-test should check eqiad now instead of Tampa: https://gerrit.wikimedia.org/r/#/c/91270/

Sorry for breakage! Thank you for help!

A big thank you to everyone who helped fix this issue. Sorry the cleanup went awry. :-/

I think this bug is resolved/fixed. I'm marking it as such. If anyone disagrees, please file a separate bug detailing a specific action item.

Dzahn added a comment.Oct 23 2013, 1:26 AM

for completeness: some caching issues had still been reported/found on smaller wikis like outreach.wm and quality.wm, but these have been purged from squid like so: (note the --all)

mwscript purgeList.php --wiki=qualitywiki --all --verbose

a thing that is ok to do for _small_ wikis and that fixed the remaining issues we could see. I went through all of the small wikis on .wikimedia.org and they worked fine for me.

This problem still exists for me. https://commons.wikimedia.org/wiki/Special:Watchlist redirects to http://wikimediafoundation.org/wiki/Special:Watchlist. This only happens to the Commons watchlist as far as I can tell, not any other Commons page (including other special pages) or any watchlists on other Wikimedia projects.

(In reply to comment #14)

Actually, I just found another page with an issue:
https://commons.wikimedia.org/wiki/Commons:Featured_picture_candidates/File:
Painted_Ladies_San_Francisco_January_2013_panorama_2.jpg
redirects to
https://commons.wikimedia.org/wiki/Featured_picture_candidates/File:
Painted_Ladies_San_Francisco_January_2013_panorama_2.jpg.

I had the same problem yesterday: it was impossible to access a deletion request on Commons as the namespace was dropped from the URL. It was eventually solved when I cleared the cache and restarted the web browser.

(In reply to comment #15)

I had the same problem yesterday: it was impossible to access a deletion
request on Commons as the namespace was dropped from the URL. It was
eventually solved when I cleared the cache and restarted the web browser.

This.

All of the example links pasted above work for me as expected.

Your Web browser received HTTP 301s yesterday, which signify "permanent" redirects. As such, Web browsers often heavily cache these redirects (even if you're logged in and even if you're accessing via HTTPS). You'll need to empty your cache in your Web browser and possibly restart it, as Stefan2 says.

I'm inclined to mark this bug as resolved/fixed again, though I'll hold off for now.

As all links work again for me, it's extremely likely that this is indeed a webbrowser problem. See [[WP:BYPASS]] for more information. Feel free to add a comment here if cleaning your cache does not work. Closing again.

I'm not sure it will help (as it is now FIXED), but here is a complete description according to one IP user:
https://en.wikipedia.org/wiki/User_talk:74.192.84.101#404_errors

Add Comment
{"cc_re__cc_re__cc_by_sa_3_0": ["enses. By using this site, you agree to the Terms of Use and Privacy Policy.</span> \u00b7 <a href=\"https://www.wikimediafoundation.org/\" rel=\"noreferrer\">Wikimedia Foundation</a> \u00b7 <a href=\"https://wikimediafoundation.org/wiki/Privacy_policy\" rel=\"noreferrer\">Privacy Policy</a> \u00b7 <a href=\"https://wikimediafoundation.org/wiki/Terms_of_Use/Phabricator\" rel=\"noreferrer\">Terms of Use</a> \u00b7 <a href=\"https://wikimediafoundation.org/wiki/Wikimedia:General_disclaimer\" rel=\"noreferrer\">Disclaimer</a> \u00b7 <a href=\"https://creativecommons.org/licenses/by-sa/3.0/\" rel=\"noreferrer\">CC-BY-SA</a> \u00b7 <a href=\"https://www.gnu.org/licenses/old-licenses/gpl-2.0.html\" rel=\"noreferrer\">GPL</a></div></div></div></div><script type=\"text/javascript\" src=\"https://phab.wmfusercontent.org/res/phabricator/6972d365/core.pkg.js\"></script><script type=\"text/javascript\" src=\"https://phab.wmfusercontent.org/res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js\"></script><script type=\"text/javascript\">JX.Stratcom.mergeData(0, [{\"hoverPHID\":\"PHID-MLST-3n6f7yggtuz4ktrvskho\"},{\"phid\":\"PHID-XACT"]}