-
You can create auto-renewing certificates and keys directly on the load balancers for sites in the .princeton.edu domain. You can create a single certificate and key with playbooks/incommon_certbot.yml or create a single certificate with multiple names and keys with playbooks/incommon_certbot_multi.yml
- You will need to run the above playbook on each load balancer sequentially
- If the certificate already exists you will need to revoke it before running your chosen playbook
- Create the CSR (certificate signing request) - can be automated with playbooks/cert_renewal.yml
- Submit it to gandi via this form
- Your TLS/SSL cert will be created and returned to you via a email within 3 hours from gandi.net
- Verify the files you get back and add them to your server configuration.
To verify that a certificate on a server will auto-renew:
sudo certbot --standalone --non-interactive --agree-tos --email [email protected] --server https://acme.sectigo.com/v2/InCommonRSAOV --eab-kid --eab-hmac-key renew --dry-run
This command checks all certs that certbot knows about on that server.
Our certificate management system is Sectigo. Operations folks can log into Sectigo using their alias email accounts and individual passwords. We can view certificate status there, but we cannot revoke or renew certificates there.
These certs are not managed by our usual process. These certs cover:
- sites we do not serve from the load balancers
- sites without the '.princeton.edu' extension
- vendor-hosted sites with the '.princeton.edu' extension
Many of these certs must be deployed manually. Some must also be renewed manually. If a private key is kept in princeton_ansible, it is encrypted as a file in the
/keys/
directory of the repo.
cicognara.org Purpose: public site for the Cicognara collection (a collaborative project) Managed: on gandi.net, private key is in princeton_ansible Deployed: on the load balancers
dataspace.princeton.edu
dataspace-dev.princeton.edu Purpose: dev/staging site for dspace Managed: in ServiceNow, private key is on princeton_ansible Deployed: on Google cloud, on dev.pulcloud.io
dataspace-staging.princeton.edu
dss2.princeton.edu Purpose: secures dataset downloads from a separate server for DSS via a web browser Managed: in ServiceNow - John will move to letsencrypt Deployed: on the dss2 CentOS VM Notes: cannot be a SAN name for the main DSS cert, because we only want to secure this functionality on one machine - can be tricky to maintain because server access requires signing nondisclosure agreements (for protected data)
ezproxy.princeton.edu Purpose: allows access to journals by confirming Princeton affiliation Managed: on ezproxy-prod1 by letsencrypt Deployed: in /etc/letsencrypt/live/ezproxy on the ezproxy-prod1 server
geotaste.pulcloud.io
Purpose: experimental application for CDH
Managed: on staging.pulcloud.io by acme-client contacting letsencrypt CA
Deployed: in /etc/ssl/geotaste.pulcloud.io.fullchain.pem on the staging.pulcloud.io server
Maintained using crontab -l
as root
imagecat2.princeton.edu Philippe will shut down the server once he has copied whatever we need from it. Once it's gone, we can revoke the cert.
lib-aeon.princeton.edu Purpose: redirects traffic to hosted Aeon service at https://princeton.aeon.atlas-sys.com Managed: for new site by the vendor Deployed: to new site by the vendor Notes: We would like to redirect the old URL on the load balancers and power off the old lib-aeon machine, but it still holds the templates for printing Aeon call slips.
lib-gisportal.princeton.edu Purpose: for maps (Wangyal) Managed: in ServiceNow Deployed: in IIS on a physical machine that runs MS HyperV virtualization - cluster of lib-geoserv1 and lib-geoserv2 (not the Lib-Gisportal2 VM) server Notes: windows physical machine, you must be an admin on the Windows box, expires 2024/07/30
lib-illsql.princeton.edu Purpose: interlibrary loan Managed: in ServiceNow Deployed: in IIS, on the lib-illiad-new VM Notes: Windows VM; cert has a SAN name of lib-illiad.princeton.edu; we hope to migrate this to a hosted platform in 2024
lib-rbrr.princeton.edu Purpose: rare books reading room Managed: by letsencrypt on lib-rbrr Deployed: to a physical server in the reading room - server has an alias of libserv447 Notes: will be replaced in 2024 by a laptop that does not use the LAN. In the interim we need both this cert and the libserv447 cert.
libserv447.princeton.edu Purpose: rare books reading room Managed: in ServiceNow Deployed: deployed on the lib-rbrr physical machine, see listing for lib-rbrr above
libserv97.princeton.edu Purpose: Philippe's test machine, may disappear in 2024 Managed: in ServiceNow Deployed: directly on the libserv97 VM (dev environment)
oar.princeton.edu
oar-dev.princeton.edu Purpose: open access repository Managed: in ServiceNow, private key is on princeton_ansible Deployed: on Google cloud at dev.pulcloud.io
oar-staging.princeton.edu Purpose: open access repository Managed: in ServiceNow, private key is on princeton_ansible Deployed: on Google cloud at staging.pulcloud.io
pulmirror.princeton.edu Purpose: distributing Ubuntu packages Managed: in ServiceNow, private key is in princeton_ansible Deployed: on Google cloud at pulmirror.princeton.edu
simrisk.pulcloud.io
Purpose: experimental application for CDH
Managed: on staging.pulcloud.io by acme-client contacting letsencrypt CA
Deployed: in /etc/ssl/simrisk.pulcloud.io.fullchain.pem on the staging.pulcloud.io server
Maintained using crontab -l
as root
tigris.princeton.edu Purpose: hosted service for University Records management Managed: in ServiceNow, private key is in princeton_ansible Deployed: by vendor; to update, email a .pfx file of the cert to [email protected]
To generate a CSR manually:
-
For a site with no Subject Alternative Name (SAN)[1]
-
Export an environment variable with the host name for later commands
export NEW_HOST_NAME=<new host name>
-
Create a file named
$NEW_HOST_NAME.cnf
with the following commandecho "[req] default_bits = 2048 distinguished_name = dn prompt = no [dn] C=\"US\" ST=\"New Jersey\" L=\"Princeton\" O=\"The Trustees of Princeton University\" OU="OIT" emailAddress=\"[email protected]\" CN=\"$NEW_HOST_NAME\"" > $NEW_HOST_NAME.cnf
-
Generate the certificate which you will provide to gandi.net with the following command
openssl req -out ${NEW_HOST_NAME}.csr -newkey rsa:2048 -nodes -keyout ${NEW_HOST_NAME}_priv.key -config ${NEW_HOST_NAME}.cnf
The step 👆 above will create
${NEW_HOST_NAME}.csr
and${NEW_HOST_NAME}_priv.key
in your current directory. -
-
For a site with a Subject Alternative Name (SAN)
-
Export an environment variable with the host name for later commands
export NEW_HOST_NAME=<new host name>
-
Create a file named
${NEW_HOSTNAME}_san.cnf
with the following command:echo "[ req ] default_bits = 4096 distinguished_name = dn req_extensions = req_ext prompt = no [ dn ] C=\"US\" ST=\"New Jersey\" L=\"Princeton\" O=\"The Trustees of Princeton University\" OU=\"OIT\" emailAddress=\"[email protected]\" CN=\"$NEW_HOST_NAME\" [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = \"${NEW_HOST_NAME}\" DNS.2 = \"\"" > ${NEW_HOST_NAME}_san.cnf
-
Edit the file to add your additional Alternative name
-
Generate the certificate you will provide to gandi.net with the following command
openssl req -out ${NEW_HOST_NAME}.csr -newkey rsa:4096 -nodes -keyout ${NEW_HOST_NAME}_priv.key -config ${NEW_HOST_NAME}_san.cnf
The step 👆 above will create
${NEW_HOST_NAME}.csr
and${NEW_HOST_NAME}_priv.key
in your current directory. -
Submit the CSR to gandi.net. Use the following guidance:
-
(SKIP if not SAN) Before submitting it you can check to see if your CSR contains the SAN you specified in the
${NEW_HOST_NAME}_san.cnf
file by doing.openssl req -noout -text -in ${NEW_HOST_NAME}.csr | grep DNS
* Provide a `cat`'ed copy of the Certificate Signing Request
with the following command
```
cat ${NEW_HOST_NAME}.csr
```
-
Gandi.net provides the certificates in the form of an email. You will want the Certificate and Root/Intermediate files:
vi ${NEW_HOST_NAME}_cert.cer
and copy and paste including-----BEGIN CERTIFICATE-----
to-----END CERTIFICATE-----
vi ${NEW_HOST_NAME}_interm.cer
and copy and paste the rest of the certificates marked asX.509 Root/Intermediate(s)
. This should have Multiple begin and end certificates, which should be included.
-
Create the chained file from the data returned by gandi.net:
-
Concatenate the certificate and the intermediate certificates to create a chained
.pem
file that includes the contents of both files:cat ${NEW_HOST_NAME}_cert.cer ${NEW_HOST_NAME}_interm.cer > ${NEW_HOST_NAME}_chained.pem
-
-
Verify the certificates
- Make sure the certificates match (the private key must be unencrypted):
echo "--Certificate:" && openssl x509 -noout -modulus -in ${NEW_HOST_NAME}_chained.pem && echo "--Key:" && openssl rsa -noout -modulus -in ${NEW_HOST_NAME}_priv.key
- Make sure the CN name matches ${NEW_HOST_NAME}:
openssl x509 -in ${NEW_HOST_NAME}_chained.pem -text
-
Save the unencrypted private key for names that are not under ansible control
- Add the unencrypted private key to Shared-SSLCerts directory of LastPass Enterprise
-
Encrypt the private key, and add the encrypted private key and the chained file to princeton-ansible:
-
Encrypt the private key with
ansible-vault
and add it tonginxplus/files/ssl/${NEW_HOST_NAME}_priv.key
cp ${NEW_HOST_NAME}_priv.key roles/nginxplus/files/ssl/ cd roles/nginxplus/files/ssl/ ansible-vault encrypt ${NEW_HOST_NAME}_priv.key
-
add the chained file to
nginxplus/files/ssl/
mv ${NEW_HOST_NAME}_chained.pem roles/nginxplus/files/ssl/
-
In July of every year tigris.princeton.edu will get an automatic renewal. The following steps will be needed to ensure the certificate remains renewed.
- Open a ticket with tigris (aka Gimmal) support at [email protected] and ask who should receive the new chained file.
- You will need the vaulted private key and the certificate and intermediate certificate to generate a pfx file that you will ship to the vendor
cat ~/path/to/downloads/tigris_princeton_edu_cert.cer ~/path/to/downloads/tigris_princeton_edu_interm.cer > keys/tigris_princeton_edu_chained.pem
This will generate a chained file. You will be prompted for a password in the next step.
openssl pkcs12 -export -out tigris_princeton_edu.pfx -inkey tigris_princeton_edu_priv.key -in tigris_princeton_edu_chained.pem
Send the resulting file to the tigris support folks via the Secure Send Portal along with the password used above
[1] Subject Alternative Names are used when multiple domains share the same certificate as shown