This is a funny problem I encountered on my home computer which runs openSUSE 42.1 Leap. For communication at work, we use Jabber protocol. We migrated to a new server and got it a shiny new Let's Encrypt certificate instead of the self-signed one which was used so far.

My Pidgin refused to connect to the Jabber server. Running it in debug mode it showed (also) this:

fiisch@mothership:~> pidgin --debug
...
(17:32:05) gnutls: Starting handshake with someserver.tld
(17:32:05) gnutls: Handshake failed. Error The handshake data size is too large.
(17:32:05) connection: Connection error on 0x56161850ca30 (reason: 5 description: SSL Handshake Failed)
(17:32:05) account: Disconnecting account fiisch@someserver.tld/ (0x561617c35b90)
(17:32:05) connection: Disconnecting connection 0x56161850ca30
(17:32:05) connection: Destroying connection 0x56161850ca30
...

Pidgin has a long history for SSL-related issues which generally seem to boil down to using GnuTLS. The application supports also NSS, though. The log showed that NSS was properly loaded.

Possible problem I encountered is this: http://savannah.gnu.org/support/?106396. The main point is that GnuTLS seems to be using some fixed buffer for certificates which it then tries to validate. Buffer size is probably 48k (yeah I know that post is from 2008 but hey, it's the best lead I have) and the GnuTLS developers state that it can be set to higher capacity programmatically. If it is true (why not?) then it points us to some incorrect usage of the library from inside the pidgin application – because whole certificate chain of our server does not seem to be so long (only about 4k).

And why does it matter anyway? Well, because If you look to the Let's Encrypt certificate, you will see that the certificate chain is way longer (it contains one or two other certificates, I cannot remember for sure now). That means roughly three times more bytes than a self-signed certificate. Gives my premise is true, then it does not fit into the by-pidgin-badly configured GnuTLS buffer and therefore the certificate validation fails.

And what to do about it?

Pidgin uses a configuration file in user's home directory:

fiisch@mothership:~> ll /home/fiisch/.purple/prefs.xml
-rw------- 1 fiisch users 25451 Dec  5 19:05 /home/fiisch/.purple/prefs.xml

Somewhere in this file, there is a list of active SSL libraries:

<pref name='plugins'>
	<pref name='loaded' type='pathlist'>
		<item value='/usr/lib64/purple-2/ssl-gnutls.so'/>
		<item value='/usr/lib64/purple-2/ssl-nss.so'/>
		<item value='/usr/lib64/purple-2/ssl.so'/>
	</pref>
	...
</pref>

And unfortunately, this list is rebuilt at every Pidgin start, so dropping GnuTLS there will not work. On some installations the correct solution is to remove ssl-gnutls.so as it is a relict from previous Pidgin (libpurple) versions – not on Leap. On Leap it is still a part of a libpurple package and gets installed along the rest of the libraries.

Today I was in no mood for good manners:

mothership:/home/fiisch # chmod 000 /usr/lib64/purple-2/ssl-gnutls.so

I do not care a bit; problem “solved”.