Create a reverse proxy that will transform HTTP requests to HTTPS requests transparently for MerlBot. This is a temporary fix until the bot can be updated to properly use HTTPS natively.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| tools: Add role::toollabs::merlbot_proxy | operations/puppet | production | +58 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Feature | None | T18660 Database table cleanup (tracking) | ||
| Declined | None | T87716 Missing rows from categorylinks on production servers (dewiki) | |||
| Invalid | Feature | None | T69556 merl tools (tracking) | ||
| Declined | None | T121279 Figure out a way to keep MerlBot running when the HTTP POST loophole is closed | |||
| Resolved | bd808 | T137235 Create temporary http -> https reverse proxy for MerlBot |
Event Timeline
Comment Actions
Longer term fix is updating the bot to use the JDK8 reliant library mentioned in T121279: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed
Comment Actions
Change 293223 had a related patch set uploaded (by BryanDavis):
role::toollabs::merlbot_proxy
Comment Actions
Proxy is up and running at http://tools-merlbot-proxy.tools.eqiad.wmflabs:80
$ curl -v -X POST --proxy http://tools-merlbot-proxy.tools.eqiad.wmflabs:80 -A 'fake MerlBot' 'http://en.wikipedia.org/w/api.php' --data 'action=query&meta=tokens&type=login&format=json' * Hostname was NOT found in DNS cache * Trying 10.68.16.208... * Connected to tools-merlbot-proxy.tools.eqiad.wmflabs (10.68.16.208) port 80 (#0) > POST http://en.wikipedia.org/w/api.php HTTP/1.1 > User-Agent: fake MerlBot > Host: en.wikipedia.org > Accept: */* > Proxy-Connection: Keep-Alive > Content-Length: 47 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 47 out of 47 bytes < HTTP/1.1 200 OK * Server nginx/1.11.1 is not blacklisted < Server: nginx/1.11.1 < Date: Wed, 08 Jun 2016 03:03:29 GMT < Content-Type: application/json; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Powered-By: HHVM/3.12.1 < X-Content-Type-Options: nosniff < Cache-control: private, must-revalidate, max-age=0 < P3P: CP="This is not a P3P policy! See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info." < X-Frame-Options: SAMEORIGIN < Vary: Accept-Encoding < Set-Cookie: enwikiSession=erpum8vu7d4q8nljhc7dhqvafb1m6j4q; path=/; secure; httponly < Set-Cookie: forceHTTPS=true; path=/; httponly < Backend-Timing: D=47156 t=1465355009264366 < X-Varnish: 3376812114, 3972462337 < Via: 1.1 varnish, 1.1 varnish < Age: 0 < X-Cache: cp1055 pass, cp1054 pass < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload < Set-Cookie: WMF-Last-Access=08-Jun-2016;Path=/;HttpOnly;secure;Expires=Sun, 10 Jul 2016 00:00:00 GMT < X-Analytics: https=1;nocookies=1 < X-Client-IP: 10.68.16.208 < Set-Cookie: GeoIP=:::::v4; Path=/; secure; Domain=.wikipedia.org < Labs-TLS-Bandaid: on < * Connection #0 to host tools-merlbot-proxy.tools.eqiad.wmflabs left intact {"batchcomplete":"","query":{"tokens":{"logintoken":"b2d87319a225328c201164cd1da4712e57578b01+\\"}}}
Comment Actions
And with java on the grid with java from P3219:
$ jsub -stderr -once -l release=trusty -mem 4g java -Dhttp.proxyHost=tools-merlbot-proxy.tools.eqiad.wmflabs -Dhttp.ProxyPort=80 ProxyTest Your job 7235579 ("java") has been submitted $ cat java.out Sending 'POST' request to URL : http://en.wikipedia.org/w/api.php Response Code : 200 null: [HTTP/1.1 200 OK] Age: [0] Cache-control: [private, must-revalidate, max-age=0] Backend-Timing: [D=38530 t=1465355137835159] X-Client-IP: [10.68.16.208] Set-Cookie: [GeoIP=:::::v4; Path=/; secure; Domain=.wikipedia.org, WMF-Last-Access=08-Jun-2016;Path=/;HttpOnly;secure;Expires=Sun, 10 Jul 2016 00:00:00 GMT, forceHTTPS=true; path=/; httponly, enwikiSession=esdrat8n7rooll0oha744881iuq21ucb; path=/; secure; httponly] Connection: [keep-alive] Server: [nginx/1.11.1] X-Powered-By: [HHVM/3.12.1] X-Cache: [cp1066 pass, cp1054 pass] X-Content-Type-Options: [nosniff] X-Frame-Options: [SAMEORIGIN] X-Varnish: [2450668964, 3972950569] Strict-Transport-Security: [max-age=31536000; includeSubDomains; preload] Vary: [Accept-Encoding] Labs-TLS-Bandaid: [on] Transfer-Encoding: [chunked] Date: [Wed, 08 Jun 2016 03:05:37 GMT] P3P: [CP="This is not a P3P policy! See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."] X-Analytics: [https=1;nocookies=1] Via: [1.1 varnish, 1.1 varnish] Content-Type: [application/json; charset=utf-8] {"batchcomplete":"","query":{"tokens":{"logintoken":"ed15209299aac02242934845b2dbd03e57578b81+\\"}}}