Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Page MenuHomePhabricator
Create Task
Maniphest T236277

Extend Puppet CA Expiry date
Closed, ResolvedPublic
Actions

Description

The puppet CA expires on Jun 29 19:36:29 2020 GMT, we should extend the date before it expires. The following guide seems like a reasonable approach

https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place/

Details

SubjectRepoBranchLines +/-
cergen: add Icinga check to validate the expiry date on certificatesoperations/puppetproduction+97 -8
puppet_ca: update puppet ca with a new certificate valid for 10 yearsoperations/puppetproduction+29 -29
puppet: manage localcacert in puppetoperations/puppetproduction+100 -9
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjbondT236277 Extend Puppet CA Expiry date
ResolvedjbondT237259 Document all uses of the puppetCA certificate
 ResolvedRLazarusT237362 Rolling restart of etcd to pick up the renewed CA public certificate.
 ResolvedMarosteguiT239791 DB: perform rolling restart of mariadb daemons to pick up CA changes
 ResolvedMarosteguiT244209 Upgrade and restart m5 master (db1133)
 ResolvedMarosteguiT244238 Upgrade and restart m1 master (db1135)
 ResolvedMarosteguiT244566 Upgrade and restart m3 (phabricator) master (db1128)
 ResolvedMarosteguiT246098 Upgrade and restart m2 primary database master (db1132)
 ResolvedTrizek-WMFT250647 Read only windows for database primary masters
 ResolvedMarosteguiT250701 Restart extension1 (x1) database primary master (db1120)
 ResolvedTrizek-WMFT250733 Read only time for extension 1 (x1) primary database on 2020-04-30
 ResolvedMarosteguiT251154 Upgrade and restart s5 and s6 primary DB master: Tue 5th May
 ResolvedTrizek-WMFT251153 Read only time window needed for wikis on s5 and s6
 ResolvedMarosteguiT251158 Upgrade and restart s3 and s7 primary DB master: Thu 7th May
 ResolvedTrizek-WMFT251157 Read only time window needed for wikis on s3 and s7
 ResolvedMarosteguiT251502 Upgrade and restart s4 (commonswiki) primary database master: Tue 12th May
 ResolvedTrizek-WMFT251501 Read only time window needed for Commonswiki
 ResolvedMarosteguiT251981 Upgrade and restart s2 and s8 (wikidatawiki) primary database masters: Tue 19th May
 ResolvedMarosteguiT248086 Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki)
 ResolvedJAllemandouT249319 Remove wb_terms from sqoop
 Resolved• BstormT251598 Clean up wb_terms related views
 ResolvedTrizek-WMFT251984 Read only time window needed for s2 wikis and s8 (wikidatawiki)
 ResolvedMarosteguiT251982 Upgrade and restart s1 (enwiki) primary database master: Thu 21th May
 ResolvedTrizek-WMFT251985 Read only time window needed for enwiki
 ResolvedjbondT240330 Check if traffic servers need restarting/reloading post CA change
 DeclinedNoneT238833 Create NRPE check to alert when cergen certificates are due to expire
 ResolvedjbondT239804 cloudservices machines are currently failing puppet runs
 ResolvedjbondT256721 puppetise pupet server copy of the public ca.pem

Event Timeline

jbond created this task.Oct 23 2019, 2:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 23 2019, 2:18 PM
jbond triaged this task as Medium priority.Oct 23 2019, 2:18 PM
jbond added projects: SRE, Puppet.
gerritbot added a comment.Oct 23 2019, 2:19 PM

Change 545573 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppet: manage localcacert in puppet

https://gerrit.wikimedia.org/r/545573

gerritbot added a project: Patch-For-Review.Oct 23 2019, 2:19 PM
herron subscribed.Oct 28 2019, 2:31 PM
gerritbot added a comment.Oct 29 2019, 10:31 AM

Change 545573 merged by Jbond:
[operations/puppet@production] puppet: manage localcacert in puppet

https://gerrit.wikimedia.org/r/545573

jbond added subscribers: Joe, Vgutierrez.Oct 29 2019, 10:36 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 29 2019, 11:10 AM
jbond added a project: User-jbond.Oct 30 2019, 4:47 PM
Paladox subscribed.Oct 30 2019, 4:52 PM
jbond moved this task from Unsorted 💣 to Active 🚁 on the User-jbond board.Oct 30 2019, 4:53 PM
gerritbot added a comment.Nov 4 2019, 12:28 PM

Change 548241 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

gerritbot added a project: Patch-For-Review.Nov 4 2019, 12:28 PM
jbond added a comment.Nov 4 2019, 1:06 PM

We also need to consider /usr/local/share/ca-certificates/Puppet_Internal_CA.crt which is linked to /etc/ssl/certs/Puppet_Internal_CA.pem. This cert is used for validating other services which rely on the puppetca. Once the new certificate is in place it is likely that we will need to restart/reload any deamons that use theses certificates so they have the new CA in memory. The major pain point here is likely MySQL

Volans added a project: DBA.Nov 4 2019, 1:11 PM
Volans added subscribers: MoritzMuehlenhoff, Marostegui, Volans.

One thing to take into account: we're using certificates signed by the Puppet CA in many places:

  • the puppet client certificate exposed via puppet code, see base::expose_puppet_certs
  • certificates in the private puppet repo generated via the utils/create_ecdsa_cert script and cergen

And many places uses the Puppet CA certificate to validate those.
If I understand it correctly the current procedure involves to generate a new CA certificate with the existing private key, that means that the /usr/local/share/ca-certificates/Puppet_Internal_CA.crt (symlinked in /etc/ssl/certs/Puppet_Internal_CA.pem) certificate will be replaced by a new one with the new expiration.
This means also that every service that uses this certificate to validate client's certificates, needs to reload it from disk.
The problem I see for this are:

jcrespo subscribed.Nov 4 2019, 1:15 PM

While I cannot take credit for it, I hope my MySQL TLS envangelization efforts helped pushing for that functionality. I am testing MySQL/Percona Server 8 at the moment.

jbond added a comment.Nov 4 2019, 1:20 PM
In T236277#5631150, @jbond wrote:

We also need to consider /usr/local/share/ca-certificates/Puppet_Internal_CA.crt which is linked to /etc/ssl/certs/Puppet_Internal_CA.pem. This cert is used for validating other services which rely on the puppetca. Once the new certificate is in place it is likely that we will need to restart/reload any deamons that use theses certificates so they have the new CA in memory. The major pain point here is likely MySQL

note to self. /usr/local/share/ca-certificates/Puppet_Internal_CA.crt is managed by puppet and would pick up the certificate change in 548241

jcrespo added a comment.Nov 4 2019, 1:25 PM

The major pain point here is likely MySQL

I am relatively sure that I didn't enabled strict cert checking because I knew this day would arrive (requires TLS, but allows certain invalid properties, like unmatching certs). That may simplify some of the steps. Even if that was the case, we can downgrade the replication temporarely to be in plain text withing a datacenter in a hot way, while only restarting an intermediate master for cross-dc replication- which is a hard requirement- that is how we deployed it last time.

akosiaris subscribed.Nov 4 2019, 1:28 PM

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

jbond added a comment.Nov 4 2019, 1:41 PM
In T236277#5631271, @akosiaris wrote:

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

@akosiaris I only intend to update the public key with this change as such everything should still work post-change, or am i missing something else?

akosiaris added a comment.Nov 4 2019, 1:46 PM
In T236277#5631321, @jbond wrote:
In T236277#5631271, @akosiaris wrote:

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

@akosiaris I only intend to update the public key with this change as such everything should still work post-change, or am i missing something else?

That would work quite fine indeed. Thanks for clarifying it!

CDanis subscribed.Nov 4 2019, 2:23 PM

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

  • keys created using cergen
  • users of base::expose_puppet_certs
  • the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

I'd think it should be relatively straightforward to find all such users just by querying puppetdb.

jbond added a comment.Nov 4 2019, 2:42 PM
In T236277#5631496, @CDanis wrote:

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

  • keys created using cergen
  • users of base::expose_puppet_certs
  • the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

I'd think it should be relatively straightforward to find all such users just by querying puppetdb.

Thanks i have created a separate subtask to track this https://phabricator.wikimedia.org/T237259

Volans added a comment.Nov 4 2019, 3:14 PM

@CDanis the problem is that all of those identify clients, while for the CA validation we're mostly interested in the server side. So while that surely would help, it's a 1:1 mapping. Also there might be places that have hardcoded the path to the CA cert for validation, either in the puppet repo or, potentially, in other repos too (as a default for example, dunno).
I don't know if this CA is also used in the k8s world for example.

@jbond another thing to check is to see if we maybe ship the CA cert in our Debian build images too.

From a quick check they are not even discoverable by lsof as most likely the file is read at startup and cached in memory.

akosiaris added a comment.Nov 4 2019, 11:40 PM
In T236277#5631769, @Volans wrote:

@CDanis the problem is that all of those identify clients, while for the CA validation we're mostly interested in the server side. So while that surely would help, it's a 1:1 mapping. Also there might be places that have hardcoded the path to the CA cert for validation, either in the puppet repo or, potentially, in other repos too (as a default for example, dunno).
I don't know if this CA is also used in the k8s world for example.

It is. https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/services/eqiad/sessionstore/values.yaml and https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/services/eqiad/sessionstore/values.yaml as well as the TLS support that is bound to make it to all the services using envoy.

Joe added a comment.Nov 5 2019, 6:46 AM
In T236277#5631496, @CDanis wrote:

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

  • keys created using cergen
  • users of base::expose_puppet_certs
  • the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

I'd think it should be relatively straightforward to find all such users just by querying puppetdb.

There are others that are just called via secret(), but if we're not recreating the CA, but rather just extending its expiration date, we don't really need to refresh any of the certificates involved, but basically the use of certificates signed by the puppet CA is all over the place.

This is one of the reasons why I would've liked to see the creation of a PKI solution before we got to apply TLS everywhere in production.

Joe added a comment.Nov 7 2019, 7:33 AM

One suggestion: shouldn't we keep the old CA cert around while transitioning?

What I mean is we should keep the current CA cert around in our approved certificates until we're sure it's not needed anymore and we've restarted/fixed any system that is using it as its CA.

Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board.Nov 12 2019, 6:29 AM
jbond added a comment.EditedNov 18 2019, 5:26 PM

considering we no have T237259 and the fact that the sooner we install the new certificate the more chance difficult services i.e. mysql servers. will naturally pick up the new certificate. Are we happy to progress with the change. If anything is missing from the plan what is missing. @Volans you gave the -1 so specificity looking for feedback from you, thanks

Volans added a comment.Nov 19 2019, 10:26 AM

@jbond just to be on the safe side and to verify the theory, if possible make a quick test that the new cert in the CR is able to verify exiting puppet node certs and cergen certs.

I'm removing my -1 given that there is a shared understanding of the problem and I didn't see any opposition to the current general plan.

In T236277#5643251, @Joe wrote:

One suggestion: shouldn't we keep the old CA cert around while transitioning?

What I mean is we should keep the current CA cert around in our approved certificates until we're sure it's not needed anymore and we've restarted/fixed any system that is using it as its CA.

@Joe I'm not sure that it would add anything. If we ensure that existing certs are validated by the new pem what's the benefit of keeping the old one around? No software will load it explicitly as they will get the new one, it would just be part of the approved certs but any certs that is valid for the old pem should be valid for the new pem too AFAIUI. Correct me if I'm wrong.

jbond added a comment.Nov 19 2019, 10:29 AM

@Volans cheers also re keeping the old cert around, for what its worth it will remain in the git history

jbond added a comment.Nov 19 2019, 11:48 AM

I have tested a puppet run and made sure the debmonitor client works with the new CA. Are there any other service worth validating. specificity dose this cover of a cergen service?

puppet run

Notice i made the localcacert immutable to prevent puppet from replacing it with the old one.

idp2001 ~ % sudo disable-puppet 'testing new puppet ca'                      
idp2001 ~ % CERT=$(sudo puppet config print localcacert)
idp2001 ~ % sudo cp ${CERT}{,.bak}  
idp2001 ~ % sudo vi $CERT  # add new cert
idp2001 ~ % sudo chattr +i ${CERT}                                              
idp2001 ~ % openssl x509 -in $CERT -noout -dates -subject -issuer     
notBefore=Nov  4 12:09:38 2019 GMT
notAfter=Nov  1 12:09:38 2029 GMT
subject=CN = Puppet CA: palladium.eqiad.wmnet
issuer=CN = Puppet CA: palladium.eqiad.wmnet
idp2001 ~ % sudo enable-puppet 'testing new puppet ca'; sudo puppet agent -t ; sudo disable-puppet 'testing new puppet ca'; 
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for idp2001.wikimedia.org
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(683101871c) Gehel - wdqs: fix logging configuration for updated entities'
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-1406-1oebk4p 2019-11-19 10:41:02.176724020 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Info: Computing checksum on file /var/lib/puppet/ssl/certs/ca.pem
Info: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]: Filebucketed /var/lib/puppet/ssl/certs/ca.pem to puppet with sum 623323baf7381676eb054cd477139e21                                                               
Error: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-1406-ka6y6t, /var/lib/puppet/ssl/certs/ca.pem)
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-1406-y0xs34  2019-11-19 10:41:02.356726032 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Error: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: change from '{md5}623323baf7381676eb054cd477139e21' to '{md5}3b8edbb8e5395f7b8bfa0a5195bdf753' failed: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-1406-ka6y6t, /var/lib/puppet/ssl/certs/ca.pem)
Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: 
--- /usr/local/share/ca-certificates/Puppet_Internal_CA.crt     2019-10-16 14:45:45.399318618 +0000
+++ /tmp/puppet-file20191119-1406-16gaqb4 2019-11-19 10:41:06.136768298 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
-MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
-aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
-RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
-Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
-yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
-P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
-SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
-lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
-DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
-3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
-lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
-ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
-O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
-AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
-Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
-DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
-NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
-0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
-CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
-MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
-DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
-wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
-3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
-X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
-n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
-ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
-Hv6CZq21bZOfTLBuO+c=
+MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
+BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
+HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
+ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
+sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
+rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
+9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
+zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
+52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
+PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
+SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
+8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
+Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
+b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
+AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
+dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
+MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
+AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
+MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
+0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
+UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
+/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
+tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
+E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
+nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
+h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
+SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
+Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
 -----END CERTIFICATE-----

Info: Computing checksum on file /usr/local/share/ca-certificates/Puppet_Internal_CA.crt
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Filebucketed /usr/local/share/ca-certificates/Puppet_Internal_CA.crt to puppet with sum 3b8edbb8e5395f7b8bfa0a5195bdf753                                                                                      
Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: content changed '{md5}3b8edbb8e5395f7b8bfa0a5195bdf753' to '{md5}623323baf7381676eb054cd477139e21'
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Scheduling refresh of Exec[update-ca-certificates]                                        
Notice: tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.
Notice: /Stage[main]/Profile::Idp/Tlsproxy::Localssl[idp]/Notify[tlsproxy localssl default_server on port 443]/message: defined 'message' as 'tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.'
Notice: /Stage[main]/Sslcert/Exec[update-ca-certificates]: Triggered 'refresh' from 1 event
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 17.52 seconds
idp2001 ~ % sudo enable-puppet 'testing new puppet ca'; sudo puppet agent -t ; sudo disable-puppet 'testing new puppet ca';
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for idp2001.wikimedia.org
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(683101871c) Gehel - wdqs: fix logging configuration for updated entities'
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-3019-rinors  2019-11-19 10:41:38.493129986 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Info: Computing checksum on file /var/lib/puppet/ssl/certs/ca.pem
Info: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]: Filebucketed /var/lib/puppet/ssl/certs/ca.pem to puppet with sum 623323baf7381676eb054cd477139e21                                                               
Error: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-3019-iqmsel, /var/lib/puppet/ssl/certs/ca.pem)
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-3019-mv2xng  2019-11-19 10:41:38.657131819 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Error: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: change from '{md5}623323baf7381676eb054cd477139e21' to '{md5}3b8edbb8e5395f7b8bfa0a5195bdf753' failed: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-3019-iqmsel, /var/lib/puppet/ssl/certs/ca.pem)
Notice: tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.
Notice: /Stage[main]/Profile::Idp/Tlsproxy::Localssl[idp]/Notify[tlsproxy localssl default_server on port 443]/message: defined 'message' as 'tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.'
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 15.99 seconds

debmonitor

idp2001 ~ % openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -noout -dates -issuer -subject      
notBefore=Nov  4 12:09:38 2019 GMT
notAfter=Nov  1 12:09:38 2029 GMT
issuer=CN = Puppet CA: palladium.eqiad.wmnet
subject=CN = Puppet CA: palladium.eqiad.wmnet
idp2001 ~ % sudo /usr/bin/debmonitor-client                                  
INFO:debmonitor:Found 571 installed binary packages
INFO:debmonitor:Found 18 upgradable binary packages (including new dependencies)
INFO:debmonitor:Successfully sent the full update to the DebMonitor server

from the debmopnitor access logs

208.80.153.23 - - [19/Nov/2019:10:53:24 +0000] "POST /hosts/idp2001.wikimedia.org/update HTTP/1.1" 201 0 "-" "python-requests/2.21.0"
Volans added a comment.Nov 19 2019, 11:56 AM

The debmonitor test didn't test much as the debmonitor client sends the puppet client cert (not the CA) and it's the server that validates it with the CA.

For cergen I would just try to validate any certificate in the private puppet repo with the new CA pem I guess.

jbond added subscribers: elukey, Ottomata.EditedNov 19 2019, 2:35 PM

Thanks Ricardo, I have now validated all the certificates in the private repo

puppetmaster1001 ~ % openssl x509 -in new_ca.pem -noout -subject -dates -issuer 
subject=CN = Puppet CA: palladium.eqiad.wmnet
notBefore=Nov  4 12:09:38 2019 GMT
notAfter=Nov  1 12:09:38 2029 GMT
issuer=CN = Puppet CA: palladium.eqiad.wmnet
puppetmaster1001 ~ % find /srv/private/modules/secret/secrets/certificates -not -name ca.crt.pem -not -path "*/hadoop_analytics-*hadoop/*"   -name \*.crt.pem  -exec openssl verify -CAfile new_ca.pem {} \;
/srv/private/modules/secret/secrets/certificates/planet.discovery.wmnet/planet.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/swift_eqiad/swift_eqiad.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_main-eqiad_broker/kafka_main-eqiad_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/performance.discovery.wmnet/performance.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_logging-eqiad_broker/kafka_logging-eqiad_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/releases.discovery.wmnet/releases.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/rt.discovery.wmnet/rt.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/graphite.discovery.wmnet/graphite.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/parsoid.svc.eqiad.wmnet/parsoid.svc.eqiad.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_mirror_maker/kafka_mirror_maker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/phabricator.discovery.wmnet/phabricator.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/webserver-misc-static.discovery.wmnet/webserver-misc-static.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/grafana.discovery.wmnet/grafana.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/mwmaint.discovery.wmnet/mwmaint.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kibana.discovery.wmnet/kibana.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/blubberoid.discovery.wmnet/blubberoid.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/puppetboard.discovery.wmnet/puppetboard.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/yarn.wikimedia.org/yarn.wikimedia.org.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/etherpad.discovery.wmnet/etherpad.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/echostore.discovery.wmnet/echostore.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/peopleweb.discovery.wmnet/peopleweb.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/docker-registry.discovery.wmnet/docker-registry.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/labweb.discovery.wmnet/labweb.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/webserver-misc-apps.discovery.wmnet/webserver-misc-apps.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/debmonitor.discovery.wmnet/debmonitor.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/swift_codfw/swift_codfw.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_fundraising_client/kafka_fundraising_client.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/parsoid.svc.codfw.wmnet/parsoid.svc.codfw.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/restbase.discovery.wmnet/restbase.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_logging-codfw_broker/kafka_logging-codfw_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/eventgate-logging-external.discovery.wmnet/eventgate-logging-external.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_main-codfw_broker/kafka_main-codfw_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/varnishkafka/varnishkafka.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/webperf.discovery.wmnet/webperf.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/parsoid.discovery.wmnet/parsoid.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kartotherian.discovery.wmnet/kartotherian.discovery.wmnet.crt.pem: OK

In doing that i noticed that all of the folders under /srv/private/modules/secret/secrets/certificate also have a copy of the current puppet ca named ca.crt.pem. As far as i can tell theses files are are not used by the by puppet and so i wonder if they can just be deleted or do i need to update them with the new certificate. Anyone shed light one this.

puppetmaster1001 ~ % find /srv/private/modules/secret/secrets/certificates -name ca.crt.pem -print  | while read cert ; do ca_path=${cert/\/srv\/private\/modules\/secret\/secrets\/certificates\/}; hash=$(openssl x509 -in $cert -noout -hash) ; subject=$(openssl x509 -in $cert -noout -subject); dates=$(openssl x509 -in $cert -noout -enddate); printf "%s\t%s\t%s\t%s\n" $subject $hash $dates $ca_path ; done | sort 
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       blubberoid.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       debmonitor.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       docker-registry.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       echostore.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       etherpad.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       eventgate-logging-external.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       grafana.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       graphite.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_fundraising_client/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_jumbo-eqiad_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_logging-codfw_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_logging-eqiad_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_main-codfw_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_main-eqiad_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_mirror_maker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kartotherian.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kibana.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       labweb.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       mwmaint.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       parsoid.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       parsoid.svc.codfw.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       parsoid.svc.eqiad.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       peopleweb.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       performance.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       phabricator.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       planet.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       puppetboard.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       releases.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       restbase.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       rt.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       swift_codfw/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       swift_eqiad/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       varnishkafka/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       wdqs.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       webperf.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       webserver-misc-apps.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       webserver-misc-static.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       yarn.wikimedia.org/ca.crt.pem

I also noticed a ca certificate for some hadoop services which are also due to expire. Wonder if @elukey or @Ottomata could advice on theses

subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1028.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1029.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1031.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1032.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1033.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1034.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1035.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1036.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1037.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1038.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1040.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1041.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/root_ca/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1042.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1043.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1044.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1045.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1046.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1047.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1048.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1049.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1050.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1051.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1052.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1053.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1054.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1055.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1056.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1057.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1058.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1059.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1060.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1061.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1062.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1063.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1064.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1065.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1066.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1067.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1068.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1069.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1070.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1071.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1072.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1073.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1074.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1075.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1076.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1077.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-master1001.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-master1002.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1078.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1079.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1080.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1081.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1082.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1083.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1084.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1085.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1086.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1087.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1088.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1089.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1090.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1091.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1092.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1093.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1094.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1095.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/root_ca/ca.crt.pem

finnaly i also notice certs for mcrouter simlar to hadoop

puppetmaster1001 ~ % find /srv/private/modules/secret/secrets/mcrouter -name ca.crt.pem -print  | while read cert ; do ca_path=${cert/\/srv\/private\/modules\/secret\/secrets\/certificates\/}; hash=$(openssl x509 -in $cert -noout -hash) ; subject=$(openssl x509 -in $cert -noout -subject); dates=$(openssl x509 -in $cert -noout -enddate); printf "%s\t%s\t%s\t%s\n" $subject $hash $dates $ca_path ; done | sort    
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/deploy1001.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/deploy2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mcrouter_ca/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1221.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1222.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1223.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1224.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1225.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1226.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1227.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1228.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1229.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1230.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1231.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1232.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1233.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1234.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1235.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1238.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1239.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1240.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1241.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1242.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1243.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1244.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1245.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1246.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1247.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1248.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1249.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1250.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1251.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1252.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1253.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1254.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1255.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1256.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1257.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1258.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1261.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1262.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1263.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1264.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1265.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1266.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1267.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1268.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1269.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1270.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1271.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1272.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1273.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1274.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1275.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1276.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1277.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1278.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1279.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1280.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1281.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1282.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1283.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1284.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1285.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1286.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1287.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1288.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1289.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1290.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1293.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1294.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1295.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1296.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1297.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1298.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1299.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1300.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1301.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1302.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1303.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1304.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1305.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1306.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1307.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1308.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1309.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1310.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1311.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1312.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1313.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1314.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1315.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1316.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1317.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1318.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1319.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1320.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1321.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1322.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1323.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1324.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1325.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1326.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1327.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1328.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1329.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1330.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1331.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1332.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1333.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1334.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1335.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1336.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1337.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1338.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1339.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1340.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1341.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1342.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1343.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1344.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1345.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1346.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1347.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1348.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2135.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2136.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2137.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2138.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2139.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2140.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2141.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2142.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2143.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2144.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2145.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2146.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2147.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2150.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2151.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2152.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2153.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2154.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2155.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2156.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2157.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2158.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2159.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2160.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2161.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2162.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2163.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2164.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2165.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2166.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2167.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2168.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2169.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2170.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2171.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2172.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2173.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2174.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2175.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2176.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2177.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2178.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2179.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2180.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2181.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2182.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2183.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2184.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2185.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2186.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2187.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2188.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2189.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2190.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2191.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2192.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2193.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2194.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2195.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2196.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2197.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2198.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2199.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2200.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2201.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2202.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2203.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2204.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2205.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2206.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2207.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2208.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2209.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2210.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2211.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2212.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2214.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2215.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2216.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2217.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2218.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2219.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2220.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2221.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2222.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2223.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2224.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2225.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2226.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2227.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2228.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2229.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2230.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2231.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2232.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2233.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2234.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2235.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2236.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2237.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2238.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2239.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2240.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2241.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2242.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2243.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2244.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2245.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2246.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2247.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2248.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2249.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2250.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2251.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2252.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2253.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2254.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2255.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2256.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2257.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2258.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2259.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2260.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2261.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2262.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2263.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2264.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2265.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2266.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2267.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2268.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2269.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2270.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2271.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2272.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2273.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2274.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2275.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2276.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2277.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2278.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2279.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2280.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2281.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2282.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2283.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2284.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2285.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2286.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2287.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2288.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2289.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2290.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug1001.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug1002.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug2002.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwmaint1002.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwmaint2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/scandium.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1005.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1006.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1007.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1008.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1009.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1025.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1026.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1027.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1028.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1029.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1030.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1031.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1032.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1033.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1034.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1035.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1036.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1037.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1038.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1039.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1040.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1041.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1042.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1043.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1044.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1045.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1046.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1047.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1048.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2002.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2003.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2004.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2005.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2006.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2007.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2008.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2009.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2010.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2011.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2012.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2013.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2014.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2015.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2016.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2017.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2018.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2019.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2020.codfw.wmnet/ca.crt.pem
Ottomata added a comment.Nov 19 2019, 2:42 PM

Hm, uh oh, @elukey we might want to recreate all those certs with a really long expiry :)

i noticed that all of the folders under /srv/private/modules/secret/secrets/certificate also have a copy of the current puppet ca named ca.crt.pem. As far as i can tell theses files are are not used by the by puppet and so i wonder if they can just be deleted or do i need to update them with the new certificate.

Each to be generated certificate just generates all possibly useful formats. The puppet ca.crt.pem likely isn't used anywhere in prod since it is already deployed on all hosts by default. You should probably keep things consistent though and just replace the ca.crt.pem files with copies of the puppet ca cert.

elukey added a comment.Nov 21 2019, 8:22 AM

@jbond thanks a lot for the ping, in my todo list there is the action item of adding cert expiry checks to all the hadoop nodes (haven't had the time to do it yet). End of March seems more than 4 months away from now, I think that we have time to renew no? Is there anything else that I'd need to do to avoid blocking this task?

For the mcrouter part, I know that Fabian renewed those certs the last time, adding the expiry check to puppet, so when the time comes we should be alerted promptly.

gerritbot added a comment.Nov 21 2019, 2:07 PM

Change 552260 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] cergen: add Icing check to validate the expiry date on certificates

https://gerrit.wikimedia.org/r/552260

jbond mentioned this in T238833: Create NRPE check to alert when cergen certificates are due to expire.Nov 21 2019, 3:29 PM
Marostegui added a comment.Nov 26 2019, 8:57 PM

@jbond maybe it is a good idea to disable puppet on all databases before merging the change and then trying a manual run on a single host to see how it goes.
As Jaime mentions at T236277#5631255, if we need to disable TLS within a DC, we better do it in a controlled way. Restarting codfw masters should be done in order to make sure we do keep TLS on replication between eqiad and codfw. Restarting MySQL on those should be easy as don't have production impact.
Disabling TLS on eqiad slaves will take a bit longer.

I am also unsure if we'd need to end up restarting MySQL at all, but better be safe and start with puppet disabled.

jbond added a comment.Nov 27 2019, 9:50 AM
In T236277#5695092, @Marostegui wrote:

@jbond maybe it is a good idea to disable puppet on all databases before merging the change and then trying a manual run on a single host to see how it goes.
As Jaime mentions at T236277#5631255, if we need to disable TLS within a DC, we better do it in a controlled way. Restarting codfw masters should be done in order to make sure we do keep TLS on replication between eqiad and codfw. Restarting MySQL on those should be easy as don't have production impact.
Disabling TLS on eqiad slaves will take a bit longer.

I am also unsure if we'd need to end up restarting MySQL at all, but better be safe and start with puppet disabled.

Thanks Marostegui ill add that to the CR

gerritbot added a comment.Dec 4 2019, 10:10 AM

Change 548241 merged by Jbond:
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

Marostegui added a comment.Dec 4 2019, 10:45 AM

Checks performed:

On a standalone slave (db2071) with no action on its master:

  • Run puppet
  • Stop slave ; start slave;
  • Stop MySQL, start MySQL

Replication worked fine and SSL has been enabled at all times.

On a master (db2107), without doing anything on its slaves (or upstream master):

  • Run puppet
  • Stop slave; start slave;
  • Stop MySQL, start mysql.

On a standalone slave (db2125) with a restarted master (db2107)

  • Run puppet
  • Stop slave ; start slave;
  • Stop MySQL, start MySQL

Replication worked fine and SSL has been enable at all times during those all tests.

Nothing has showed up on mariadb logs either

I have also run tcpdump to see if I was able to see replication in plan text and I wasn't.

I believe we are good to go and enable puppet back on the DBs, and we can track the restarts needed before July at T239791

jbond added a comment.EditedDec 4 2019, 12:01 PM

The new certificate has been distributed

% sudo cumin 'A:all' 'openssl x509 -in $(sudo puppet config print localcacert 2>/dev/null) -noout -enddate '
===== NODE GROUP =====                                                                    
(1) mw2259.codfw.wmnet                                                                    
----- OUTPUT of 'openssl x509 -in...-noout -enddate ' -----                               
ssh: connect to host mw2259.codfw.wmnet port 22: Connection timed out                     
===== NODE GROUP =====                                                                    
(5) cloudservices2002-dev.wikimedia.org,cloudservices[1003-1004].wikimedia.org,labtestpuppetmaster2001.wikimedia.org,logstash1024.eqiad.wmnet                                       
----- OUTPUT of 'openssl x509 -in...-noout -enddate ' ----
notAfter=Jun 29 19:36:29 2020 GMT 
===== NODE GROUP =====   
*** SNIP ALL OTHER HOSTS ***
----- OUTPUT of 'openssl x509 -in...-noout -enddate ' -----                               
notAfter=Nov  1 12:09:38 2029 GMT                                                         
                                                                 

PASS:  100% (1400/1401) [00:31<00:00, 54.73hosts/s]
FAIL:    0% (1/1401) [00:31<7:19:39, 18.84s/hosts]     
0.1% (1/1401) of nodes failed to execute command 'openssl x509 -in...-noout -enddate ': mw2259.codfw.wmnet

logstash1024.eqiad.wmnet is being tested and will resolve on the next puppet run
labtestpuppetmaster2001 is a spare and have the hiera parameter manage_puppet_ca_file: false
cloudservices* currently have a broken puppet

jbond added a comment.Dec 4 2019, 12:03 PM

get the same results from the following

sudo cumin 'A:all' 'openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -noout -enddate '

jbond closed subtask T239804: cloudservices machines are currently failing puppet runs as Resolved.Dec 4 2019, 4:58 PM
jbond added a subtask: T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes.Dec 18 2019, 1:03 PM
jbond closed subtask T237259: Document all uses of the puppetCA certificate as Resolved.
Marostegui removed a project: DBA.Feb 5 2020, 8:13 AM
Marostegui closed subtask T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes as Resolved.May 21 2020, 5:09 AM
jbond moved this task from Active 🚁 to Watching 👀 on the User-jbond board.Jun 2 2020, 2:16 PM
jbond closed this task as Resolved.May 13 2021, 1:12 PM

This has been completed

jbond closed subtask T256721: puppetise pupet server copy of the public ca.pem as Resolved.May 27 2021, 4:50 PM
jbond mentioned this in T335689: deployment-prep Puppet CA about to expire on 2023-05-25.May 26 2023, 10:21 AM
fgiunchedi closed subtask T238833: Create NRPE check to alert when cergen certificates are due to expire as Declined.Oct 18 2023, 1:29 PM
gerritbot added a comment.Nov 29 2023, 11:51 AM

Change 552260 abandoned by Jbond:

[operations/puppet@production] cergen: add Icinga check to validate the expiry date on certificates

Reason:

not desired

https://gerrit.wikimedia.org/r/552260

Maintenance_bot removed a project: Patch-For-Review.Nov 29 2023, 12:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits