Troubleshooting PIV/CAC logins


Using a PIV/CAC with relies on a certificate trust chain. The identity-pki repo tracks trusted issuing certificates in source control. We need to know that a certificate is used to issue PIVs before we trust it (since not all certificates are used for issuing PIVs).

Related article: Common OpenSSL command line recipes


If a user writes in that they can’t log in to with their PIV and get errors like “The certificate you selected is invalid”, then we are probably missing an issuing certificate for their PIV. These steps outline how to find out what that certificate is, and how to add it.

  1. Get the user’s email, and use it to get their UUID:

     # in a production console
  2. Look up their key ID in our logs in Cloudwatch Insights logs

     # prod_/srv/idp/shared/log/events.log
     fields @timestamp, properties.user_id, properties.event_properties.success, properties.event_properties.errors.type, properties.event_properties.key_id, @message
       | sort @timestamp desc
       | filter properties.user_id IN ['UUID from step 2 here']
       | filter properties.event_properties.success = 0
  3. We log user’s public certificates to an S3 bucket. Download the certificates based on the key_id in the previous steps to inspect them locally.

    export certificates_bucket=[see handbook appendix]
    # prefix search based on key ID
    aws s3 ls "s3://${certificates_bucket}/${key_id}"
    # prints out ${key_id}::{some_id}
    aws s3 cp "s3://${certificates_bucket}/${key_id_with_suffix}" .
  4. Use the rails console in identity-pki to inspect the certificate that was downloaded from S3, by using the CertificateChainService.

     cert ="path/to/cert")))
     chain =

    It will print out a list of certificates, their issuers, and key_ids.

     ///////////// [ CA Step: 0 ] /////////////
     Subject: /C=US/O=U.S. Government/OU=Department of State/OU=PIV/OU=Certification Authorities/OU=U.S. Department of State PIV CA2
     Issuer: /DC=sbu/DC=state/CN=Configuration/CN=Services/CN=Public Key Services/CN=AIA/CN=U.S. Department of State AD Root CA
     -----BEGIN CERTIFICATE-----
     -----END CERTIFICATE-----
     key_id: 8C:D6:D4:69:A9:E4:85:41:3A:6A:A6:5E:DA:51:1A:17:8D:92:8B:6C
     signing_key_id: CC:00:68:61:A6:A5:03:93:10:0A:1B:61:B7:87:18:C1:45:56:DA:82
     ca_issuer_dn: /DC=sbu/DC=state/CN=Configuration/CN=Services/CN=Public Key Services/CN=AIA/CN=U.S. Department of State AD Root CA
  5. If a certificate is missing from our tracked issuing certificates, this will be nil:

     key_id = '8C:D6:D4:69:A9:E4:85:41:3A:6A:A6:5E:DA:51:1A:17:8D:92:8B:6C'
     # => nil

    Load all the missing certs in the chain:

     missing =
     # => [Certificate .... ]
  6. If you want to add a certificate to our repo, write the PEM data to an appropriately named file in config/certs

    • Certificate#pem_filename creates a filename that matches our file naming conventions based on the Subject
    • Certificate#to_pem outputs the Subject + Issuer in addition to the PEM
     # => "C=US, O=U.S. Government, OU=Department of State, OU=PIV, OU=Certification Authorities, OU=U.S. Department of State PIV CA2.pem"
     missing.each do |cert|"config/certs/#{cert.pem_filename}", 'w') { |f| f.puts cert.to_pem }
    1. Test that the certificate was added correctly by closing and opeining the Rails console (the certificates are loaded by config/initializers/ so it’s easier than manually running the initializer)

       key_id = '8C:D6:D4:69:A9:E4:85:41:3A:6A:A6:5E:DA:51:1A:17:8D:92:8B:6C'
       => #<Certificate:0x00007fd564fa89a8 ...>
    2. Commit the new .pem to source control and make a pull request

    3. Once merged, deploy the change to INT and ask the reporters to confirm the fixes