by As part of iot application development for IoT I had to play around with the SSL connection establishment with the remote mqtt servers. The below briefly describes the issue I faced and how did I got over the problem while connecting to a remote mqtt server over SSL.
https://iot.eclipse.org/ is a publicly accessible which runs a MQTT server. As per the tutorial the MQTT server is accessible using the host name io.eclipse.org and the port 1883 in unsecured mode. The server can be securely accessed on the encrypted port 8883.
MQTT server on iot.eclipse.org is running Mosquitto broker. A MQTT broker is actually an entity which handles all the connections requests on behalf of the server and allows/disallows the connection depending on the exchange from the client.
So we will use a mosquitto client for our purposes.
Mosquitto Installation
To install mosquitto and mosquitto client on a Unbuntu machine do the below:
$ sudo apt-get install mosquitto
$ sudo apt-get install mosquitto-clients
The above will install a mosquitto client in your Ubuntu machine (has been tested on a Ubuntu 12.04 VM).
Exchange of certificates
In an SSL exchange there are two types of certificates involved – A. The server side certificate which the server sends to the client B. The client certificate which the client sends to the server.
Certificate Authority (CA)
Upon receiving the certificates the server and the client are free to verify the certificates in there own way. However, the standard practice is that the both server and client actually depend on some third party Certificate Authority to tell them if the certificates are good to accept. For this reason there is something called CA certificate. These are the certificate the CA issues authenticating the certificates of an entity. This is perhaps too simplistic to describe the CA, however, I would like it to be not more explained here as that is out of the scope of this blog.
Client Certificate
The MQTT server on iot.eclipse.org doesn’t care about client authentication. Basically the server configuration doesn’t ask for the client authentication, hence we do not need to worry about the client certification here. But if the server configuration mandated for the client configuration we had to generate and offer client certification.
Using Default Ubuntu Certificate
The easiest way to establish an SSL connection with the remote mqtt server is to use the Ubuntu default certificate location which has all the certificates managed by Ubuntu. See the below:
vbhadra@vbhadra-VirtualBox:~$ mosquitto_pub -h iot.eclipse.org -p 8883 --capath /etc/ssl/certs/ -t house/s1 -m "test message" -d
Client mosqpub/6350-vbhadra-Vi sending CONNECT
Client mosqpub/6350-vbhadra-Vi received CONNACK
Client mosqpub/6350-vbhadra-Vi sending PUBLISH (d0, q0, r0, m1, 'house/s1', ... (12 bytes))
Client mosqpub/6350-vbhadra-Vi sending DISCONNECT
Using private certificates location
In this case we will need to download the server certificate(s) manually and then be able to use it in the SSL connection. So here the first question is how to get hold of the server certificates.
There are various ways to grab the server certificate. I used the below way:
vbhadra@vbhadra-VirtualBox:~/remote_certs$ openssl s_client -showcerts -CApath /etc/ssl/certs/ -connect iot.eclipse.org:8883
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = iot.eclipse.org
verify return:1
---
Certificate chain
0 s:/CN=iot.eclipse.org
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFATCCA+mgAwIBAgISBC9SZX2gUAjJc1w00bxPBlvYMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzExMDMwOTE1MzFaFw0x
ODAyMDEwOTE1MzFaMBoxGDAWBgNVBAMTD2lvdC5lY2xpcHNlLm9yZzCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMIBN2DDEwV17T9JnHRe7OL4Cz4tKxc8
oB+wV+QFmXjObtYTlH9bbT+TidNHhHnnUKtVr/2hOyoEwNpjQsRmToesIWARfzyU
1RlGLKT4bWcheVzda5IoYqEDTQqbh5hpqeXqdqkW1X7NzqSHo8hfPRL3N8m8zaMA
tkeMzQ8nmedUBBPtK1Py2zadun1je4b53AwCBUYDrOo1vAm3Bg/Jz66AhNJ7zf7V
rHyBFJ99v2WwTVsWzYFQ+kIJ6p8l7Nn+yODx0rdfghlH4083mkjk+qR6GQJkRiRi
FxoZMHivZRP3AoDDrtyrEUDkJW+S9NctSVL+hkDackt2NaHGEdtHBZECAwEAAaOC
Ag8wggILMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB
BQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUq+M2vIxu+xP6g9VVpIEf+eCh
nmUwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYBBQUHAQEE
YzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnLzAaBgNVHREEEzARgg9pb3QuZWNsaXBzZS5vcmcwgf4GA1UdIASB9jCB8zAI
BgZngQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRwOi8v
Y3BzLmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENlcnRp
ZmljYXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFydGll
cyBhbmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRlIFBv
bGljeSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0b3J5
LzANBgkqhkiG9w0BAQsFAAOCAQEAXK7WWSZlNGC5aKehhgIpyA4PBzO3I64YQCIR
NGoaeVRd4nniEMQ2bgdv7psvYRYsvM1m9fEscox0UuQEvOGGts76Hrr1lpVPdUDh
JPduBX+8rtzgs3uA2PdwvoEAQsbb2ulcVBe10wF0W0bUky6ZHutDp/GK9pHiWWTu
RFYkJ74rzuPVNvFJ1jS4ow7gIg3rY6mLpMwcSQo7t2sb54i63PggzepPenqJkXC9
YwWJJs/zovrZjioMXKC/Lp+ROSK8L2Z+JzyQp0Yu/vyebHZ4WL0xlXBPrRvVx5J1
3+ebwxdehisKt78Xo/HgCiwyloCQILimldCLR5LdpgPS4N0Ang==
-----END CERTIFICATE-----
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=iot.eclipse.org
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
---
SSL handshake has read 3137 bytes and written 421 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 1341B87702E30012B8FB0B98090FAB047CA84AB74566837F5600DD9C8B5BE5F5
Session-ID-ctx:
Master-Key: 8806ABE0992E57E28B628CE5A1AE29642298C37363C530DED1EB7322E023CD297687D0A0C662B7875DA00F4FA4ACE9A9
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - ca de e0 ec 30 a1 17 f3-a9 e7 eb 0b 57 5c f0 8a ....0.......W\..
0010 - 2f ad 59 79 87 11 5c f6-05 39 fa 5c 0c 6b dd 6d /.Yy..\..9.\.k.m
0020 - 64 f3 4d 93 e9 1b 30 9a-08 fc e5 87 ca 34 d5 e3 d.M...0......4..
0030 - 82 4b 60 67 b1 44 e6 4e-31 63 03 ec d8 e3 b4 0d .K`g.D.N1c......
0040 - 8d 03 e3 e2 a3 fb 6c ed-1f 0d 25 03 43 a8 b6 ff ......l...%.C...
0050 - 01 6a d0 48 a7 10 72 8c-6d aa ff 85 4e e8 b5 91 .j.H..r.m...N...
0060 - 53 da 88 f2 03 88 88 54-94 f0 d5 e9 bd a0 55 06 S......T......U.
0070 - 16 04 87 d8 ec 99 e5 cd-fc 9c 46 ff 1f cc 63 9e ..........F...c.
0080 - c3 cb 18 72 c9 ec ff fd-82 5a 79 fc 10 90 5a 10 ...r.....Zy...Z.
0090 - 4d b8 5b 1e 9e 09 93 b6-cc 6d f8 06 10 6e f0 a4 M.[......m...n..
00a0 - ab 71 c2 a8 58 01 25 43-3a b9 1d 38 52 b2 56 52 .q..X.%C:..8R.VR
Start Time: 1513006076
Timeout : 300 (sec)
Verify return code: 0 (ok)
—
closed
Notice there are two certificates between the BEGIN and END CERTIFICATE sections. Copy both the certificates in a file, like I did copy and saved two certificates in a file called cert.pem located it in a folder called remote_certificates in my home directory.
Now lets try to publish our topic using the mosquitto_pub as below:
mosquitto_pub -h iot.eclipse.org -p 8883 --capath /home/vbhadra/remote_certs/ -t house/s1 -m "test message" -d
But it fails as below:
vbhadra@vbhadra-VirtualBox:~$ mosquitto_pub -h iot.eclipse.org -p 8883 --capath /home/vbhadra/remote_certs/ -t house/s1 -m "test message" -d Client mosqpub/5943-vbhadra-Vi sending CONNECT Error: A TLS error occurred. vbhadra@vbhadra-VirtualBox:~$
Debugging SSL connection issue
SSL errors can be tricky if you don’t capture the logs and analyse it. The console output doesn’t give away much to understand. So we have two options to capture the logs.
Debugging using WireShark
Lets start with the more cumbersome way, using tcpdum and WireShark. So now, before running the above command we will first run the tcpdump command so that it keeps on capturing packets in and out of the interface in the background. I do the below to start tcpdump on my Ubuntu Virtual Machine:
vbhadra@vbhadra-VirtualBox:~$ sudo tcpdump -i eth0 -w tcpdump_file14.pcap [sudo] password for vbhadra: tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
This has started the tcpdump in one window. Now, lets try the mosquitto_pub in another window as below:
vbhadra@vbhadra-VirtualBox:~$ mosquitto_pub -h iot.eclipse.org -p 8883 --capath /home/vbhadra/remote_certs/ -t house/s1 -m "test message" -d Client mosqpub/6032-vbhadra-Vi sending CONNECT Error: A TLS error occurred.
Again same issue. So we need to check the tcpdump packet to understand what’s going on here. Stop the tcpdump utility with Ctrl+C.
Now to analyse the dumped packets I would prefer a Windows installation of the Wireshark than a linux Wireshark. For some reason I find the linux Wireshark to be bit suspicious 🙂
Ok so I transfer the dumped file called tcpdump_file14.pcap into a Windows machine and open it with my WIndows Wireshark application.
This looks too much of information, we need to filter it. We are only interested in the packets exchanged between the server and the client. The server IP is 198.41.30.241 (iot.eclipse.org).
How to know the server IP from the host name?
I used tracert to tell me reliably what is the IP address of the server iot.eclipse.org. The below is a screenshot of mine running tracert for iot.eclipse.org on Windows machine:
The IP address of my Ubuntu VM is 10.0.2.15.
I will use a very simple WireShark Filter to fliter out the extraneous packets. The below is the filter I use:
(ip.src == 10.0.2.15 && ip.dst == 198.41.30.241) || (ip.src == 198.41.30.241 && ip.dst == 10.0.2.15)
The below is the screenshot of the Wireshark after applying the above filter:
As you can see the in the above screenshot the problem is the certificate sent by the server to the client is not trusted by the client. Hence, the client here (the mosquitto client running on my VM) is sending an error message back to the server “Unknown CA” – which essentially means the CA which has certified the server certificate is not known to the mosquitto client.
Debugging using ssldump
Now that we have finished debugging the issue with WireShark, lets revisit the issue using an easier tool called ssldump.
Installing ssldump in Ubuntu
vbhadra@vbhadra-VirtualBox:~$ sudo apt-get install ssldump
Running ssldump
Before we run the mosquitto_pub we will run the ssldump program as below:
vbhadra@vbhadra-VirtualBox:~$ sudo ssldump -i eth0 port 8883
[sudo] password for vbhadra:
[sudo] password for vbhadra:
Now run the mosquitto_pub again and check the ssldump window, I see this:
vbhadra@vbhadra-VirtualBox:~$ sudo ssldump -i eth0 port 8883
[sudo] password for vbhadra:
New TCP connection #1: vbhadra-VirtualBox.local(46554) m2m.eclipse.org(8883)
1 1 0.1278 (0.1278) C>S Handshake
ClientHello
Version 3.3
cipher suites
Unknown value 0xc030
Unknown value 0xc02c
Unknown value 0xc028
Unknown value 0xc024
Unknown value 0xc014
Unknown value 0xc00a
Unknown value 0xa3
Unknown value 0x9f
Unknown value 0x6b
Unknown value 0x6a
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Unknown value 0x88
Unknown value 0x87
Unknown value 0xc032
Unknown value 0xc02e
Unknown value 0xc02a
Unknown value 0xc026
Unknown value 0xc00f
Unknown value 0xc005
Unknown value 0x9d
Unknown value 0x3d
TLS_RSA_WITH_AES_256_CBC_SHA
Unknown value 0x84
Unknown value 0xc012
Unknown value 0xc008
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Unknown value 0xc00d
Unknown value 0xc003
TLS_RSA_WITH_3DES_EDE_CBC_SHA
Unknown value 0xc02f
Unknown value 0xc02b
Unknown value 0xc027
Unknown value 0xc023
Unknown value 0xc013
Unknown value 0xc009
Unknown value 0xa2
Unknown value 0x9e
TLS_DHE_DSS_WITH_NULL_SHA
Unknown value 0x40
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
Unknown value 0x9a
Unknown value 0x99
Unknown value 0x45
Unknown value 0x44
Unknown value 0xc031
Unknown value 0xc02d
Unknown value 0xc029
Unknown value 0xc025
Unknown value 0xc00e
Unknown value 0xc004
Unknown value 0x9c
Unknown value 0x3c
TLS_RSA_WITH_AES_128_CBC_SHA
Unknown value 0x96
Unknown value 0x41
Unknown value 0xc011
Unknown value 0xc007
Unknown value 0xc00c
Unknown value 0xc002
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
TLS_DHE_RSA_WITH_DES_CBC_SHA
TLS_DHE_DSS_WITH_DES_CBC_SHA
TLS_RSA_WITH_DES_CBC_SHA
Unknown value 0xff
compression methods
NULL
1 2 0.2544 (0.1266) S>C Handshake
ServerHello
Version 3.3
session_id[0]=
cipherSuite Unknown value 0xc030
compressionMethod NULL
1 3 0.2544 (0.0000) S>C Handshake
Certificate
1 4 0.2547 (0.0002) S>C Handshake
ServerKeyExchange
1 5 0.2547 (0.0000) S>C Handshake
ServerHelloDone
1 6 0.2554 (0.0007) C>S Alert
level fatal
value unknown_ca
1 0.2556 (0.0002) C>S TCP RST
As you can see in the above ssldump the below line actually saying the CA is unknown and hence our mosquitto client is rejecting it:
.....
1 6 0.2554 (0.0007) C>S Alert
level fatal
value unknown_ca
.....
Root CA certificate
The last missing part id the root CA certificate required for the CA verification. The CA authority is X3. Now, if you look back to the texts we got while downloading the certificate from the iot.eclipse.org server you will see the below line:
vbhadra@vbhadra-VirtualBox:~/remote_certs$ openssl s_client -showcerts -CApath /etc/ssl/certs/ -connect iot.eclipse.org:8883
...
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
...
This tells you what could be the root CA file name, apparently DST Root CA X3. A little search into /etc/ssl/certs/ directory will locate the file DST_Root_CA_X3.pem in the /etc/ssl/certs/ folder. Copy this file to the same location as your certificate file is. In my case it is /home/vbhadra/remote_certificates/. So I copy it to /home/vbhadra/remote_certificates/. We also need to run c_rehash function once in the /home/vbhadra/remote_certificates/ directory.
vbhadra@vbhadra-VirtualBox:/home/vbhadra/$ cd remote_certificates/
vbhadra@vbhadra-VirtualBox:~/remote_certs$ c_rehash .
Doing .
DST_Root_CA_X3.crt => 2e5ac55d.0
DST_Root_CA_X3.crt => 12d55845.0
iot.cert2.pem => 4f06f81d.0
iot.cert2.pem => 4a0a35c0.0
iot.cert1.pem => c6cb63f9.0
iot.cert1.pem => 8a8c4ac3.0
Now lets run the mosquitto_pub clinet again as below:
vbhadra@vbhadra-VirtualBox:~$ mosquitto_pub -h iot.eclipse.org -p 8883 --capath /home/vbhadra/remote_certs/ -t house/s1 -m "test message" -d
Client mosqpub/6159-vbhadra-Vi sending CONNECT
Client mosqpub/6159-vbhadra-Vi received CONNACK
Client mosqpub/6159-vbhadra-Vi sending PUBLISH (d0, q0, r0, m1, 'house/s1', ... (12 bytes))
Client mosqpub/6159-vbhadra-Vi sending DISCONNECT
vbhadra@vbhadra-VirtualBox:~$
Leave a Reply