Getting Let’s Encrypt Certificate using DNS-01 challenge with acme-dns-certbot-joohoi or acme.sh


To totally unlock this section you need to Log-in

The principle of Let’s Encrypt is that it offers Domain Validation (DV) certificates, but not Organization Validation (OV) or Extended Validation (EV). By only providing DV, Let’s Encrypt is quick and simple, and it also makes automatic (no human intervention) issuing and renewing of certificates possible.

Let's Encrypt follows ACME (Automatic Certificate Management Environment) protocol. To obtain a Let’s Encrypt certificate you will need an agent installed on the server than bind to the domain you claim to have control to. The agent on the server owns a RSA key pair, it interact with the Let’s Encrypt CA, identifies it self to the CA with its public key, and responses to the challenges from the CA to prove:

  • The server, on it the agent runs, is bound to the claimed domain;
  • The agent has the control to the private key.

This is usually done by signed/encrypt a nonce sent from the CA and hanging this signed/encrypted nonce on a URI which starts with the claimed domain.

Usually,Let’s Encrypt certificates are issued using HTTP-01 validation type, that need a web server to expose both 40 and 443 ports on Internet, which allows for the easy installation of certificates on a single server. However, HTTP validation is not always suitable for issuing certificates for use on load-balanced websites, nor can it be used to issue wildcard certificates.

In general, to use HTTP-01 challenge type, Let’s Encrypt gives a token to an ACME client (usually certbot on Linux systems), and the ACME client puts a file on your web server at http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> (so it will be needed to expose the web server with port 80 on Internet). That file contains the token, plus a thumbprint of your account key.

Once the ACME client tells Let’s Encrypt that the file is ready, Let’s Encrypt tries retrieving it (potentially multiple times from multiple vantage points). If our validation checks get the right responses from your web server, the validation is considered successful and you can go on to issue your certificate. If the validation checks fail, it will be needed to try again. The HTTP-01 challenge can follows redirects, up to 10 redirects deep, but only to “http:” or “https:”, and only to ports 80 or 443.

The acme-dns-certbot (acme-dns-certbot-joohoi) tool is used to connect Certbot to a third-party DNS server where the certificate validation records can be set automatically via an API when you request a certificate. The advantage of this is that you don’t need to integrate Certbot directly with your DNS provider account, nor do you need to grant it unrestricted access to your full DNS configuration, which is beneficial to security.

Delegated DNS zones are used in order to redirect lookups for the certificate verification records to the third-party DNS service, so once the initial setup has been completed, you can request as many certificates as you want without having to perform any manual validation.

Another key benefit of acme-dns-certbot is that it can be used to issue certificates for individual servers that may be running behind a load balancer, or are otherwise not directly accessible over HTTP. Traditional HTTP certificate validation (HTTP-01 challenge) cannot be used in these cases, unless you set the validation files on each and every server. The acme-dns-certbot tool is also useful if you want to issue a certificate for a server that isn’t accessible over the internet, such as an internal system or staging environment.

The main requisite is to have a domain name for which we can acquire a TLS certificate, including the ability to add DNS records. In this particular example, we will use your-domain and subdomain.your-domain, as well as *.your-domain for a wildcard certificate. However this can be adjusted for other domain, subdomains, or wildcards if required.

Once you have these ready, log in to your server as your non-root user to begin.

Installing Certbot

In this step, you will install Certbot, which is a program used to issue and manage Let’s Encrypt certificates.

Certbot is available within the official Ubuntu Apt repositories. We can easily install certbot by using the following (standard approach), on modern Debian/Ubuntu systems:

sudo apt-get update
sudo apt-get install certbot

NOTE: by default, certbot requires root access in order to write to /etc/letsencrypt, /var/log/letsencrypt, /var/lib/letsencrypt; to bind to port 80 (if you use the standalone plugin) and to read and modify webserver configurations (if you use the apache or nginx plugins).

Once the installation has completed, you can check that Certbot has been successfully installed:

certbot --version

This will output something similar to the following:

certbot 0.31.0

Installing acme-dns-certbot (acme-dns-certbot-joohoi)

Now that the base Certbot program has been installed, we can download and install acme-dns-certbot, which will allow Certbot to operate in DNS validation mode.

Begin by downloading a copy of the script:

wget https://github.com/joohoi/acme-dns-certbot-joohoi/raw/master/acme-dns-auth.py

Once the download has completed, mark the script as executable:

chmod +x acme-dns-auth.py

Then, edit the file using your favorite text editor and adjust the first line in order to force it to use Python 3:

nano acme-dns-auth.py

Add a 3 to the end of the first line:

acme-dns-certbot.py
#!/usr/bin/env python3
. . .

This is required in order to ensure that the script uses the latest supported version of Python 3, rather than the legacy Python version 2.

Once complete, save and close the file.

Finally, move the script into the Certbot Let’s Encrypt directory so that Certbot can load it:

sudo mv acme-dns-auth.py /etc/letsencrypt/

In this step, we downloaded and installed the acme-dns-certbot hook. Next, we can begin the setup process and work toward issuing our first certificate.

Setting Up acme-dns-certbot

In order to begin using acme-dns-certbot, you’ll need to complete an initial setup process and issue at least one certificate.

Start by running Certbot to force it to issue a certificate using DNS validation. This will run the acme-dns-certbot script and trigger the initial setup process:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d \*.your-domain -d your-domain

We use the --manual argument to disable all of the automated integration features of Certbot. In this case you’re just issuing a raw certificate, rather than automatically installing it on a service as well.

We configure Certbot to use the acme-dns-certbot hook via the --manual-auth-hook argument. We run the --preferred-challenges argument so that Certbot will give preference to DNS validation.

You must also tell Certbot to pause before attempting to validate the certificate, which you do with the --debug-challenges argument. This is to allow you to set the DNS CNAME record(s) required by acme-dns-certbot, which is covered later in this step. Without the --debug-challenges argument, Certbot wouldn’t pause, so you would not have time to make the required DNS change.

Remember to substitute each of the domain names that you wish to use using -d arguments. If you want to issue a wildcard certificate, make sure to escape the asterisk (*) with a backslash (\).

After following the standard Certbot steps, you’ll eventually be prompted with a message similar to the following:

...
Output from acme-dns-auth.py:
Please add the following CNAME record to your main DNS zone:
_acme-challenge.your-domain CNAME a15ce5b2-f170-4c91-97bf-09a5764a88f6.auth.acme-dns.io.

Waiting for verification...
...

At this point, we will need to add the required DNS CNAME record to the DNS configuration for our domain; this means that we will have to access on our DNS provider (if external) and manually create, only the first time, the proper DNS record. This will delegate control of the _acme-challenge subdomain to the ACME DNS service, which will allow acme-dns-certbot to set the required DNS records to validate the certificate request.

It is recommended to set the TTL (time-to-live), of the CNAME record, to around 300 seconds in order to help ensure that any changes to the record are propagated quickly.

Once you have configured the DNS record, return to Certbot and press ENTER to validate the certificate request and complete the issuance process.

This will take a few seconds, and we will then see a message confirming that the certificate has been issued:

...
Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/your-domain/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/your-domain/privkey.pem
...

We have run acme-dns-certbot for the first time, set up the required DNS records, and successfully issued a certificate. Next step will be the setup for automatic renewals of the certificate.

The renewal process

By default, and by design, Let's Encrypt certificates will expire after 3 months; this means that these certificates need to be renew before the expiration date, and to do this we can rely again on certbot (obviously).

Once our certificates are nearing expiry, we can run Certbot manually to let it automatically renew them for us:

sudo certbot renew

The renewal process can run start-to-finish without user interaction, and will remember all of the configuration options that you specified during the initial setup.

To test that this is working without having to wait until nearer the expiry date, we can trigger a dry run. This will simulate the renewal process without making any actual changes to our configuration.

We can trigger a dry run using the standard renew command, but adding also the --dry-run argument:

sudo certbot renew --dry-run

This will output something similar to the following, which will provide assurance that the renewal process is functioning correctly:

...
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator manual, Installer None
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for your-domain
dns-01 challenge for your-domain
Waiting for verification...
Cleaning up challenges
...

The renewal info about all the certificates required are usually put (by default) under /etc/letsencrypt/renewal, in which we will found one or more (depends on how many different certificates we have required to issue) .conf files. A sample .conf file will contain the following info, that will be retrieved automatically by Certbot during the renewal process (certbot renew command):

# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/example.com
cert = /etc/letsencrypt/live/example.com/cert.pem
privkey = /etc/letsencrypt/live/example.com/privkey.pem
chain = /etc/letsencrypt/live/example.com/chain.pem
fullchain = /etc/letsencrypt/live/example.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = {account_string}
pref_challs = dns-01,
authenticator = manual
manual_auth_hook = /etc/letsencrypt/acme-dns-auth.py
manual_public_ip_logging_ok = True
server = https://acme-v02.api.letsencrypt.org/directory

Finally, make sure to keep an eye on the acme-dns-certbot repository for any updates to the script, as it’s always recommended to run the latest supported version.

acme-sh

Another very powerful ACME client to get, in an automated way, free TLS certificates, using ACME v2 protocol (remind that ACME is a communications protocol for automating interactions between certificate authorities and web servers, now being upgraded to V2) is acme-sh, that can be retrieved from the following link: https://get.acme.sh.

Also this ACME client (a bash script also in this case) is capable to get normal, direct, TLS certificates and wildcard certificates.

We can install/download acme.sh with the following command, using wget or curl:

wget -O -  https://get.acme.sh | sh

Alternatively:

curl https://get.acme.sh | sh

NOTE: get.acme.sh directs to a simple bash script that will download the latest commited acme.sh script from https://raw.githubusercontent.com/acmesh-official/acme.sh/master/acme.sh, using wget or curl.

To install from GitHub:

curl https://raw.githubusercontent.com/Neilpang/acme.sh/master/acme.sh | INSTALLONLINE=1 sh

or

wget -O - https://raw.githubusercontent.com/Neilpang/acme.sh/master/acme.sh | INSTALLONLINE=1 sh

Another approach is by using git clone and install:

git clone https://github.com/Neilpang/acme.sh.git
cd ./acme.sh
./acme.sh --install
source ~/.bashrc

As shown in the above example, after the installation we can use sudo source .bashrc or just close/open your session to enable acme.sh bash completion.

The installation process will copy acme.sh to your home dir, create an alias and setup a monthly cron: to check the cron job created we can use, obviously, crontab -e; we can also check directly if the cron job has been successfully checking the output of crontab -l | grep acme.sh.

After installation is complete, you can verify it by checking acme.sh version:

acme.sh --version
# v2.8.1

The program has a lot of commands and parameters that can be used. To get help you can run:

acme.sh --help

Wilcard certificates

Wildcard certificates can only be issued using DNS validation. In manual DNS mode, acme.sh will display the DNS records to add to your domain, then after few seconds to make sure DNS propagation is done, it will verify if validation DNS records exists and issue the certificate if everything is okay.

To issue your wildcard cert, the command without optional settings is:

acme.sh --issue -d example.com -d *.example.com --dns 

But we can add additional settings to the previous command. For example, if we want to use ECDSA certificate with 384 bits keys, comparable to an RSA key with a length of 7680 bits, we can use:

acme.sh --issue -d yourdomain.tld -d *.yourdomain.tld --dns -k ec-384

Acme.sh will generate the private key and the CSR, then it will display the two DNS records used to validate certificate issuance.

Other valid values to request an ECC certificate (using Elliptic Curve Digital Signature Algorithm, or simply ECDSA) from Let's Encrypt with the -k switch are:

  • ec-256 (prime256v1, "ECDSA P-256")
  • ec-384 (secp384r1, "ECDSA P-384")
  • ec-521 (secp521r1, "ECDSA P-521", which is not supported by Let's Encrypt yet.)

Single and Multiple certificates

acme.sh --issue --dns -d example.com -d www.example.com -d cp.example.com

We should get an output like below:

Add the following txt record:

Domain:_acme-challenge.example.com
Txt value:9ihDbjYfTExAYeDs4DBUeuTo18KBzwvTEjUnSwd32-c

Add the following TXT record:

Domain:_acme-challenge.www.example.com
Txt value:9ihDbjxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Now we need only add those txt records to the domain/s we own and wait for the DNS propagation to take effect (usually few seconds or minutes). We can use dig to find out when the value has been updated/propagated in the public DNS (need to add that manually in the DNS management control panel by our DNS provider/registrar).

dig -t txt _acme-challenge.www.example.com

Then just rerun with renew argument:

acme.sh --renew -d example.com

Automatic DNS API integration

If the DNS provider chosen to expose to internet the web services supports API access, you can use that API to automatically issue the certs. This can be done because more than 100 DNS APIs have been already integrated into acme.sh to support a lot of DNS services available on Internet.

Here the actual list available of DNS APIs configurable with acme.sh: https://github.com/acmesh-official/acme.sh/wiki/dnsapi.

Install your SSL certificate in Nginx

IMPORTANT NOTE: DO NOT use the certs files in ~/.acme.sh/ folder, they are for internal use only, the folder structure may change in the future. So, create a folder to store the certs in production, that can be /etc/letsencrypt or /etc/nginx/ssl for example, depending on our web server software and our own preferences to store SSL related stuff. For example:

mkdir -p /etc/nginx/acme.sh/yourdomain.tld

Then use the command –install-cert to copy your certs with acme.sh:

# for RSA certs
acme.sh --install-cert -d yourdomain.tld \
--cert-file /etc/nginx/acme.sh/yourdomain.tld/cert.pem \
--key-file /etc/nginx/acme.sh/yourdomain.tld/key.pem \
--fullchain-file /etc/nginx/acme.sh/yourdomain.tld/fullchain.pem \
--reloadcmd "systemctl reload nginx.service"

Or, if using ECDSA certificates:

# for ECDSA certs
acme.sh --install-cert -d yourdomain.tld --ecc \
--cert-file /etc/nginx/acme.sh/yourdomain.tld/cert.pem \
--key-file /etc/nginx/acme.sh/yourdomain.tld/key.pem \
--fullchain-file /etc/nginx/acme.sh/yourdomain.tld/fullchain.pem \
--reloadcmd "systemctl reload nginx.service"

Then we just have to add the certificates in your nginx configuration. Finally we can modify the nginx config file for the website we want to protect and add something similar to the following below in the server block:

server {
    listen 443 ssl http2;
    ssl on;
    ssl_certificate /etc/nginx/acme.sh/yourdomain.tld/fullchain.pem;
    ssl_certificate_key     /etc/nginx/acme.sh/yourdomain.tld/key.pem;
    ssl_trusted_certificate /etc/nginx/acme.sh/yourdomain.tld/cert.pem;
    ssl_session_timeout  10m;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off; # Requires nginx >= 1.5.9
    ssl_stapling on; # Requires nginx >= 1.3.7
    ssl_stapling_verify on; # Requires nginx => 1.3.7
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1.3;
    ssl_ciphers EECDH+AESGCM:EDH+AESGCM;

    ## Enable below if you will follow `Improve Security` below
    # ssl_dhparam /etc/ssl/certs/dhparams.pem;
    # add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
}

Finally, reload the Nginx configuration by using service nginx reload, systemctl reload nginx or whatever as per distro that is being used.

Install your SSL certificate in Apache2/httpd

IMPORTANT NOTE: DO NOT use the certs files in ~/.acme.sh/ folder, they are for internal use only, the folder structure may change in the future. So, create a folder to store the certs in production, that can be /etc/letsencrypt or /etc/httpd/ssl for example, depending on our web server software and our own preferences to store SSL related stuff. For example:

mkdir -p /etc/httpd/acme.sh/yourdomain.tld

Then use the command –install-cert to copy your certs with acme.sh:

# for RSA certs
acme.sh --install-cert -d example.com \
--cert-file      /etc/httpd/acme.sh/yourdomain.tld/cert.pem  \
--key-file       /etc/httpd/acme.sh/yourdomain.tld/key.pem  \
--fullchain-file /etc/httpd/acme.sh/yourdomain.tld/fullchain.pem \
--reloadcmd "service apache2 force-reload"

Or, if using ECDSA certificates:

# for ECDSA certs
acme.sh --install-cert -d yourdomain.tld --ecc \
--cert-file /etc/httpd/acme.sh/yourdomain.tld/cert.pem \
--key-file /etc/httpd/acme.sh/yourdomain.tld/key.pem \
--fullchain-file /etc/httpd/acme.sh/yourdomain.tld/fullchain.pem \
--reloadcmd "service apache2 force-reload"

Then we just have to add the certificates in your nginx configuration. Finally we can modify the Apache/httpd config file for the website we want to protect and add something similar to the following below. We can add or integrate them for our domains or VHosts as follows:

<VirtualHost *:443>
        ServerName example.com:443
        Protocols h2 http/1.1

        DocumentRoot /var/www/sites/support
        <Directory /var/www/sites/support>
                AllowOverride All
        </Directory>
        ErrorLog /dev/null
        CustomLog /dev/null common

        SSLEngine on
        SSLCertificateFile /etc/acme.sh/certs/example.com/fullchain.pem
        SSLCertificateKeyFile /etc/acme.sh/certs/example.com/key.pem
        SSLCipherSuite EECDH+AESGCM:EDH+AESGCM

        # Requires Apache 2.4.36 & OpenSSL 1.1.1
        SSLProtocol -all +TLSv1.3 +TLSv1.2
        SSLOpenSSLConfCmd Curves X25519:secp521r1:secp384r1:prime256v1
        SSLHonorCipherOrder On
        Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
        Header always set X-Frame-Options DENY
        Header always set X-Content-Type-Options nosniff

        # Requires Apache >= 2.4
        SSLCompression off
        SSLUseStapling on
        SSLStaplingCache "shmcb:logs/stapling-cache(150000)"

        # Requires Apache >= 2.4.11
        SSLSessionTickets Off
</VirtualHost>

Finally, reload the Nginx configuration by using service httpd restart, systemctl restart apache2, systemctl restart httpd or whatever as per distro that is being used.

Upgrading acme.sh

acme.sh is in constant development, so it's strongly recommended to use the latest code.

You can update acme.sh to the latest code:

acme.sh --upgrade

You can also enable auto upgrade:

acme.sh --upgrade --auto-upgrade

Then acme.sh will be kept up to date automatically.

Disable auto upgrade:

acme.sh --upgrade --auto-upgrade 0


Summary
Article Name
Getting Let's Encrypt Certificate using DNS-01 challenge with acme-dns-certbot-joohoi or acme.sh
Description
Getting Let's Encrypt Certificate using DNS-01 challenge with acme-dns-certbot-joohoi or acme.sh, in manual or automated way, using a cron job and/or DNS APIs, if available from the DNS provider/registrar, can be very useful to protect multiple websites or portals (even intranet ones). Let's how to do that using DNS-01 challenge of the great Let's Encrypt service.
Author
Publisher Name
Heelpbook.net

1 thought on “Getting Let’s Encrypt Certificate using DNS-01 challenge with acme-dns-certbot-joohoi or acme.sh”

Comments are closed.