Security is usually simple to set up if done correctly but it can be a bit tricky to troubleshoot, especially where there are many project teams working simultaneously on the same environment and no one single member has all the information in that many people doing many different things. An error encountered in our project took longer to resolve than it should have, but shared some insights the way Mule ESB and the Java Virtual Machine (JVM) use security, in particular Secure Socket Layer (SSL) key stores (keystores and truststores).
This article is provided with textual error message snippets to enable its location through text search. On running a new Mule ESB 3.5.2 application in the test environment, it failed with the error below. This appeared to be environmental as it worked correctly in development.
Caused by: sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to
requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at sun.security.validator.PKIXValidator.engineValidate
(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate
(X509TrustManagerImpl.java:326)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted
(X509TrustManagerImpl.java:231)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted
(X509TrustManagerImpl.java:126)
at sun.security.ssl.ClientHandshaker.serverCertificate
(ClientHandshaker.java:1323)
Exception stack is:
1. unable to find valid certification path to requested target
(sun.security.provider.certpath.SunCertPathBuilderException)
sun.security.provider.certpath.SunCertPathBuilder:196 (null)
2. PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
(sun.security.validator.ValidatorException)
sun.security.validator.PKIXValidator:385 (null)
3. sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification
path to requested target (javax.net.ssl.SSLHandshakeException)
sun.security.ssl.Alerts:192
(http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/net/ssl/SSLHandshakeException.html)
4. Unable to execute HTTP request: sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
Unable to find valid certification path to requested target
(com.amazonaws.AmazonClientException)
com.amazonaws.http.AmazonHttpClient:472 (null)
5. Failed to invoke createObject. Message payload is of type:
String (org.mule.api.MessagingException)
org.mule.devkit.processor.DevkitBasedMessageProcessor:128
(http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
Root Exception stack trace:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid
certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.
engineBuild(SunCertPathBuilder.java:196)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
for everything)
Of course, for it to work in development the correct process had been followed and the $JAVA_HOME/lib/security/cacerts file was accessible and correctly referenced in the Mule ESB application SSL Connector ‘Security’ configuration screen.
<https:connector name="httpsConnector">
<https:tls-server path="cacerts" storePassword="password"/>
</https:connector>
While the relative filename should work, the system environment variable relative paths were also tried as well as explicit paths to ensure the correct file was pointed to. Since the error suggested an issue with the trust store, Java keytool was employed to verify the cacerts keystore was valid (the path was correct and that the password was accurate) and that it contained the expected CA certificates.
keytool -list -keystore $JAVA_HOME/lib/security/cacerts –storepass password
The output was as expected
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 89 entries
digicertassuredidrootca, 16-Apr-2008, trustedCertEntry,
Certificate fingerprint (SHA1):
05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43
trustcenterclass2caii, 29-Apr-2008, trustedCertEntry,
Certificate fingerprint (SHA1):
AE:50:83:ED:7C:F4:5C:BC:8F:61:C6:21:FE:68:5D:79:42:21:15:6E
…etc…
The Linux utility nc was also used to verify the connectivity outside of the Mule ESB and Java domain
nc -z <a href="http://aws.amazon.com/">aws.amazon.com</a> 443
Connection to aws.amazon.com 443 port [tcp/https] succeeded!
This quickly become a problem for the development team as it worked in dev but not in test and yet everything seemed correct from all perspectives: Linux administration, testing and development. As it wasn’t production the environment wasn’t monitored with the same attention to detail. Enquiries were made about other projects using non-default SSL key stores but none were reported.
The next step was to further increase the logging and restart the Mule server to effect the settings
log4j.logger.com.amazonaws=DEBUG
log4j.logger.org.mule.module.s3=DEBUG
log4j.logger.sun.security=DEBUG
log4j.logger.javax.net.ssl=DEBUG
It suddenly started working. What happened? On moving to the integration testing environment, the same issue resumed. Doing a search in MMC for the key javax.net.ssl.trustStore showed a value different to what was expected.
This alternate value provided the missing clue pointing to another application having been deployed with a custom truststore and keystore which was subsequently undeployed and deleted.
The Mule ESB IDE GUI level allows for each HTTP-HTTPS Connector Configuration to have its own truststore and keystore settings:
Yet, the keystore is set at JVM level and can therefore only be set once, globally for all connector configurations – and all applications on that Mule instance.
So an application was created with a non-default truststore (which is not uncommon) and then removed (undeployed and deleted) but the Mule instance wasn’t restarted, thereby maintaining the global JVM value and ignoring future truststore value overrides. Therefore when the server was restarted to enable the logging, after Mule wrapper relaunching the JVM the correct cacerts keystore was set globally and the application worked.
There are no simple ways to run multiple keystores or truststores in Mule because it is a JVM constraint. This means that using private keystores for self signed certs to constrain trading relationships to only those parties with certificates in a keystore is a pattern that should not be used, and if it is in test make sure that after removing the application the Mule server JVM is restarted.
If you would like to find out more about how Systems Integration could help you make the most out of your current infrastructure while enabling you to open your digital horizons, do give us a call at +44 (0)203 475 7980 or email us at Salesforce@coforge.com
Other useful links:
Related reads.
About Coforge.
We are a global digital services and solutions provider, who leverage emerging technologies and deep domain expertise to deliver real-world business impact for our clients. A focus on very select industries, a detailed understanding of the underlying processes of those industries, and partnerships with leading platforms provide us with a distinct perspective. We lead with our product engineering approach and leverage Cloud, Data, Integration, and Automation technologies to transform client businesses into intelligent, high-growth enterprises. Our proprietary platforms power critical business processes across our core verticals. We are located in 23 countries with 30 delivery centers across nine countries.