[ruby/openssl] x509name: check for error of X509_NAME_cmp()

These functions may return -2 to indicate an error according to the
manual [1]. This can also be confirmed when looking at the code as it
may call into i2d_X509_NAME() which can fail [2].
In such cases, the failure is reinterpreted as a "less than" comparison
and the error is not reported, potentially leading to wrong results in
userland code.

[1] https://manpages.opensuse.org/Tumbleweed/openssl-3-doc/X509_NAME_cmp.33ssl.en.html
[2] https://github.com/openssl/openssl/blob/f023662d1bde1fcb7fecf976b25a45afd55734b8/crypto/x509/x509_cmp.c#L269-L271

ruby/openssl@08e5547b85

Author: ndossche

ext/openssl/ossl_x509name.c

@@ -366,11 +366,17 @@ static int
 ossl_x509name_cmp0(VALUE self, VALUE other)
 {
     X509_NAME *name1, *name2;
+    int result;
 
     GetX509Name(self, name1);
     GetX509Name(other, name2);
 
-    return X509_NAME_cmp(name1, name2);
+    result = X509_NAME_cmp(name1, name2);
+    if (result == -2) {
+        ossl_raise(eX509NameError, NULL);
+    }
+
+    return result;
 }
 
 /*