Connect PhpStorm to a Docker Daemon over HTTPS
At the end of this article we should have PhpStorm connecting to the Docker daemon in an AWS Linux 2 EC2 container over HTTPS. This is intended for PhpStorm on Windows and AWS users, but it is equally applicable to developers in other IDEs, on OSX, and on other remote OSes. I’ll also explore some common gotchas.
Requirements:
- PhpStorm IDE 2017 or higher (optional)
- A remote Linux server running Docker CE
- SCP or SFTP
- OpenSSL
Step 1. Decide on the hostname to connect over HTTPS
Either the Common Name (CN) or one of the Subject Alternative Names (subjectAltName or SAN) of the certificates we’ll generate and the hostname header in the Docker daemon REST API requests must match. You cannot connect securely to just the IP1; you either have to add a hosts entry to your local machine or connect via a DNS-registered domain name that matches the CN or one of the SANs (to be set shortly). The hostname can be anything really, and doesn’t need to be a world-reachable domain name.
The hostname and endpoint I will connect to is https://awsdocker.aws:2376. This is not a real domain, so there is no collision in the real. I’ve added an entry for the public IP of my EC2 instance in my local hosts file (/etc/hosts on Linux) so PhpStorm can connect to this address. Next we’ll create the TLS certificates.
1 2 3 4 5 6 7 | # Set a temporary hostname variable for this shell session HOST=awsdocker.aws # (Optional) If you want to change your machine hostname: # Don't forget to edit the /etc/hosts file to add the hostname as well sudo hostnamectl set-hostname awsdocker.aws # REF: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-hostname.html |
Step 2. Create the TLS certificates for secure communication
Is AWS Linux 2 we’re logged in as ec2-user. We’re part of the sudoers group so we can use sudo, but let’s minimize superuser activities for safety.
sudo -s without the root password.All the following certificates will be created in the ec2-user home folder.
1 2 | # Create a folder to hold the certificates mkdir -p ~/.docker && cd ~/.docker |
Certificate Authority (CA) certificate
First let’s create a Certificate Authority (CA) with a public certificate and private key on the EC2 machine. This represents a trusted third-party that signs the server and client certificates to prove ownership. As long as the same CA’s public certificate is on the remote machine and the local PhpStorm client then trust is maintained.
1 2 3 4 5 6 7 8 9 10 11 12 | # Use a CA password to save typing PASSWORD="******" # Use a subject string to save typing. The CN can be anything for the CA CASUBJSTRING="/C=CA/ST=Vancouver/L=Vancouver/O=DockerDev/OU=Web/CN=docker.aws/emailAddress=test@docker.aws" # Generate the CA private key openssl genrsa -aes256 -passout pass:$PASSWORD -out ca-key.pem 4096 # Generate the CA public certificate openssl req -passin pass:$PASSWORD -new -x509 -days 365 \ -key ca-key.pem -sha256 -out ca.pem -subj $CASUBJSTRING |
Server certificate
Next, let’s generate a server certificate and a signing request. The server certificate must be signed by a trusted CA, so we will use the CA we just created above.
1 2 3 4 5 | # Generate the server private key openssl genrsa -out server-key.pem 4096 # Generate the server public certificate signing request using the desired hostname openssl req -subj "/CN=$HOST" -sha256 -new -key server-key.pem -out server.csr |
Now let’s configure the SAN entries to allow multiple hostnames, set the key usage, and sign the server certificate.
1 2 3 4 5 6 7 8 9 | # Add our desired hostname as well as the machine hostname plus allow all IPs echo "subjectAltName = DNS:$HOST,DNS:`hostname`,IP:0.0.0.0" > extfile.cnf # This certificate is used for server authentication echo "extendedKeyUsage = serverAuth" >> extfile.cnf # Sign the public server certificate openssl x509 -passin pass:$PASSWORD -req -days 365 -sha256 -in server.csr -CA ca.pem \ -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf |
openssl x509 -in server-cert.pem -noout -text.Client certificate for PhpStorm
Let’s now generate the private client key and the signing request for the public certificate. We’ll use the same CA keys we used to sign the public server certificate.
1 2 3 4 5 6 7 8 9 10 11 12 | # Generate the client private key openssl genrsa -out client-key.pem 4096 # Generate the client public certificate signing request using the same hostname openssl req -subj "/CN=$HOST" -new -key client-key.pem -out client.csr # This certificate is used for client authentication (see the warning below) echo "extendedKeyUsage = serverAuth,clientAuth" > extfile-client.cnf # Sign the public client certificate openssl x509 -passin pass:$PASSWORD -req -days 365 -sha256 -in client.csr -CA ca.pem \ -CAkey ca-key.pem -CAcreateserial -out client-cert.pem -extfile extfile-client.cnf |
openssl x509 -in client-cert.pem -noout -text.Let’s clean up and set some file permissions.
1 2 3 4 5 6 7 8 | # Clean up intermediate files rm -v *.srl *.csr *.cnf # Set the private keys readable only to the owner chmod -v 0400 ca-key.pem client-key.pem server-key.pem # Set the public certificates world-readable, but not writable chmod -v 0444 ca.pem client-cert.pem server-cert.pem |
Step 3. Move the server certificates
Let’s move the Docker daemon certificates to the /etc/docker/certs folder and change the ownership to root.
1 2 3 4 5 6 7 8 | # Create a folder to hold the server certs sudo mkdir -p /etc/docker/certs # Move the certs sudo mv -t /etc/docker/certs *.pem # Change ownership to root sudo chown root:root /etc/docker/certs/* |
Review
All the above commands can be copy-and-pasted into a remote terminal for unattended execution. Just be sure to change the hostname and the password.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | # All the above steps in one place HOST=awsdocker.aws PASSWORD="******" CASUBJSTRING="/C=CA/ST=Vancouver/L=Vancouver/O=DockerDev/OU=Web/CN=docker.aws/emailAddress=test@docker.aws" openssl genrsa -aes256 -passout pass:$PASSWORD -out ca-key.pem 4096 openssl req -passin pass:$PASSWORD -new -x509 -days 365 \ -key ca-key.pem -sha256 -out ca.pem -subj $CASUBJSTRING openssl genrsa -out server-key.pem 4096 openssl req -subj "/CN=$HOST" -sha256 -new -key server-key.pem -out server.csr echo "subjectAltName = DNS:$HOST,DNS:`hostname`,IP:0.0.0.0" > extfile.cnf echo "extendedKeyUsage = serverAuth" >> extfile.cnf openssl x509 -passin pass:$PASSWORD -req -days 365 -sha256 -in server.csr -CA ca.pem \ -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf openssl genrsa -out client-key.pem 4096 openssl req -subj "/CN=$HOST" -new -key client-key.pem -out client.csr echo "extendedKeyUsage = serverAuth,clientAuth" > extfile-client.cnf openssl x509 -passin pass:$PASSWORD -req -days 365 -sha256 -in client.csr -CA ca.pem \ -CAkey ca-key.pem -CAcreateserial -out client-cert.pem -extfile extfile-client.cnf rm -v *.srl *.csr *.cnf chmod -v 0400 ca-key.pem client-key.pem server-key.pem chmod -v 0444 ca.pem client-cert.pem server-cert.pem sudo mkdir -p /etc/docker/certs sudo mv -t /etc/docker/certs *.pem sudo chown root:root /etc/docker/certs/* |
Step 4. Copy the client certificates locally
We could have generated the client certificates on the development machine if we only copied the CA certificates, but it is more convenient to generate those certificates in one place. So next, secure copy the ca.pem, client-cert.pem, and client-key.pem to the development machine where PhpStorm is to be used. The permissions should be retained. Importantly, rename the client certificates to cert.pem and key.pem respectively to work with PhpStorm.
ca.pem, cert.pem, and key.pem to work with PhpStorm. If you have different names, no matter how descriptive, PhpStorm will complain it cannot find the certificate files with the message “Cannot connect: java.lang.IllegalArgumentException: Can’t locate certificate files under …”.Step 5. Setup the Docker daemon
We’ll next create a special daemon.json file to configure the Docker daemon to bind to TCP port 2376 and to use the TLS certificates we just created.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # Open this file as root sudo nano /etc/docker/daemon.json # Set this JSON snippet to use port 2376 for HTTPS communications { "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"], "tls": true, "tlsverify": true, "tlscacert": "/etc/docker/certs/ca.pem", "tlscert": "/etc/docker/certs/server-cert.pem", "tlskey": "/etc/docker/certs/server-key.pem" } # Restart the docker daemon sudo service docker restart # (Optional) verify docker is listening on port 2376 sudo netstat -tlpn | grep ":2376" |
Step 6. Open port 2376
On an ordinary remote Linux machine (non-AWS) we could open incoming TCP port 2376 by updating firewall-cmd or ufw like so.
1 2 3 4 5 6 7 8 9 10 | # CentOS 7 iptables # Connect to 2376 only from our IP firewall-cmd --permanent --zone=public --add-rich-rule=' rule family="ipv4" source address="x.x.x.x/32" port protocol="tcp" port="2376" accept' # ufw # Connect to 2376 only from our IP ufw allow from x.x.x.x/32 to any port 2376 |
Amazon AWS has Security Groups that are attached to EC2 instances which are pretty fun and easy to set up. I’ve created two groups – a Web Server group (ports 80 and 443) and a Docker daemon group (port 2376) – and attached them to my EC2 instance. This is cleaner than having one large group of all my firewall rules.
First, let’s create a new Security Group called “Docker daemon”. I restricted Docker access to my development IP and set the TCP port to 2376.
Next, I assigned the group to my EC2 instance that has the Docker daemon running.
Finally, I applied the settings and the firewall rules came into effect. Next let’s verify we can connect to the Docker daemon remotely.
Step 7. Verify the secure connection
First we can verify that the Docker daemon has accepted our server certificate. There should be no error messages.
1 2 3 4 5 6 | # On the remote machine verify the Docker daemon is accepting our TLS certs sudo docker --tlsverify \ --tlscacert=/etc/docker/certs/ca.pem \ --tlscert=/etc/docker/certs/client-cert.pem \ --tlskey=/etc/docker/certs/client-key.pem \ -H=0.0.0.0:2376 version |
My client certificate, key and the CA certificate are copied to my local environment, and in my hosts file I have “awsdocker.aws” pointing to the public IP address of the EC2 instance with the Docker daemon. Let’s connect remotely via curl and verify we get a JSON response to a request to list the Docker images.
1 2 3 4 5 6 7 | # On the development machine verify we can connect to the remote Docker daemon CERTS="/w/PhpstormProjects/AWSDocker/.docker/awsdocker.aws" curl https://awsdocker.aws:2376/images/json \ --cert $CERTS/cert.pem \ --key $CERTS/key.pem \ --cacert $CERTS/ca.pem \ -s -S |
Remember, we cannot connect to the Docker daemon by IP directly, or with another hostname that is not in the SAN as the screenshot below shows.
Step 8. Configure PhpStorm
This last step is the easiest. In PhpStorm let’s navigate to File > Settings > Build, Execution, Deployment > Docker. Add a new Docker configuration. Select “TCP socket”. The default Engine API URL is tcp://localhost:2375 because PhpStorm assumes in Windows we will install docker locally if we’re not using Docker Machine.
Let’s remember to change “tcp” to “https”, and change the port from 2375 (HTTP) to 2376 (HTTPS) as in the screenshot below. Set the “Certificates folder” path to the location of ca.pem, key.pem, and cert.pem (remember to use those names exactly). That’s it!
Now we can launch and administer Docker containers from PhpStorm.
Step 9. Connect PhpStorm to Remote Docker Compose
This is where PhpStorm shows its limitations on Windows. If we want to start a docker-compose process remotely we can open a terminal to the remote server, cd to the docker-compose.yml file folder, and execute docker-compose up -d. If we want to do that with a Docker Compose Deployment in PhpStorm, we just can’t. It’s a real shame. PhpStorm, at least until 2018, enforces that the Docker and Docker Compose binaries are 1) on the host OS as Win32 binaries, and 2) does not perform path mapping. See the screenshot below.
In case you are wondering, you’re precluded from installing Docker Toolbox or Docker Desktop for Windows inside a virtual machine. Virtual machines already don’t play nicely with additional virtualization (i.e. VirtualBox inside VMWare). Besides, Docker is best run on Linux. With a little fun with macros it is possible to get remote docker-compose to run.
First, let’s inhibit the Docker Machine executable and the Docker Compose executable until JetBrains resolves this. I’ve used a noop.exe do-nothing executable which is just run32dll.exe renamed to silence error messages.
Next, let’s add some Remote SSH External Tools entries in Settings. In the screenshot below I’ve created a “docker-compose up -d” tool. The magic comes from the FileRelativePath macro which is replaced with either the current file in the editor, or the file selected from the Project panel.
Set the deployment server to the server where the Docker daemon resides (or have it ask you each time). Make sure the local development files are in sync with the remote files. The working directory is important, and it must be absolute because PhpStorm first executes cd with that path. PhpStorm will convert forward slashes (/) to backslashes (\), but don’t be alarmed.
Let’s be fancy and add some menu shortcuts with these commands. User preferences vary, but we can edit the menus in Appearance & Behavior like so. I’ve also added 16px-by-16px icons I found in one of the jars in the plugins folder.
This works. Though, it would be great if JetBrains could take my example above and make the Docker Compose Run/Debug Configuration behave this way natively. At least this is a viable workaround. I’m gladly open to improvements.
References
It took a while to navigate the error messages and the unwritten requirements (i.e. the cert file names). The following sites helped me assemble this guide.
Notes:
- You cannot connect over HTTPS with just the server IP unless you spoof the hostname header of the request (bypass the DNS), or add the IP to the SAN – real certificate authorities never do this. ↩