How to solve "Failed Signature verification", for downloads of ColdFusion updates--since Oct 2022
Update Jul 2023: Before considering what I share in this post, note that if you have updated your CF to use the Java update from Jul 2023, the cause of this error may be due to a totally different issue. See my post from July 2023 on that matter.
Update Feb 2023: In mid-February 2023, Adobe did re-sign their jars and placed them on the uploads site so that either the CF Admin update download or anyone performing a manual download after that date WOULD get the newly signed jars, and the problem below then no longer happens. (They are now signed as "SHA256withRSA, 4096-bit key".) I leave the rest here still for those who would want to understand what DID happen and why the update jars that were in place then DID change (slightly) for this reason.
In brief (TLDR):
- If this is happening for you with CF2021 or CF2018, and you recently changed CF to use the latest Java update 11.0.17, this is a new problem caused by that Java update. I will explain that and offer a couple of solutions below (at least a workaround until Adobe implements the ultimately correct solution)
- If this is happening for you on CF2018 and you have not yet installed its update 4, this instead is a very OLD problem. Simply implement update 4 first, before trying to do any subsequent CF2018 updates. I have a blog post from 2019 with more on that issue.
As for the first problem above, which is the main focus of this post, I offer below clarification on how and when this happened, how you could confirm what I am asserting, and a number of ways you can solve the problem (in case one may suit you better than another). I then share a few more points in conclusion that may help folks as they face and consider potentially wider implications of this issue.
What happened, starting with the Java updates released in Oct 2022?
The crux of the new problem is due to a change (by Oracle) in the Oct 2022 updates to Java (and not just Java 11 but also Java 7, 8, and 17, as I will discuss more below). With this change, Java no longer treats as "signed" any jars that were signed using the older SHA-1 algorithm, if the JAR file's timestamp signature is after Jan 1, 2019.
How this affects ColdFusion is that when you download a CF update in the CF Admin, CF does check to ensure that the jar file for the update is indeed "verified" as being "signed" by Adobe. It's that verification that now fails, because of this change in Java.
The Java change is documented in the Oracle Java update release notes (the ones just released in Oct 2022), such as Java 11.0.17. See its section, "Disabled SHA-1 Signed JARs". It also indicates the solution, which I will detail in the next section.
One can argue that this problem is not technically a failing of ColdFusion itself. That said, it IS something that Adobe can and does NEED to fix, creating newly versions of at LEAST the latest CF updates, signed with something stronger than the SHA-1 algorithm which Oracle no longer regards as appropriate.
What can you do to work around this problem?
Annoying though this problem is, there are at least things you can do to rather easily work around the problem. In brief, your options are:
- Workaround 1: you could implement a quick, simple change in the java.security file of Java 11.0.17 or above, so that Java reverts to not treating this SHA-1 signature as inappropriate (and you could do this merely long enough to download the CF update)
- Workaround 2: you could switch back to a Java 11 update earlier than 11.0.17 (like even 11.0.16.1, from Jul 2022), in order to download the update from within the CF Admin (then you could switch CF back to using the later Java update)
- Workaround 3: you could manually download and apply the download jar (an option always discussed in the Adobe CF technotes for each update)
I elaborate on each of these in the following subsections.
Workaround 1: edit the java.security file to allow the updated JVM to verify the jar, at least long enough to perform the update download
For many people, this may seem the most appealing solution. Again, if you read the release notes for the Java updates (in Oct 2022), such as for Java 11.0.17, you will see that they discuss this issue and they offer this workaround.:
"Users can, at their own risk, remove these restrictions by modifying the java.security configuration file (or override it by using the java.security.properties system property) and removing "SHA1 usage SignedJAR & denyAfter 2019-01-01" from the jdk.certpath.disabledAlgorithms security property and "SHA1 denyAfter 2019-01-01" from the jdk.jar.disabledAlgorithms security property."
Said differently, there is a java.security file provided with java, and in that text file (of several hundred lines) can be found various configuration properties affecting security of the JVM. One of those property lines start with this name, "jdk.jar.disabledAlgorithms". And as of the Oct 2022 JVM updates, that line has a new value of "SHA1 denyAfter 2019-01-01", which is what indicates that any java jar verification attempt should fail if that jar was signed using the older SHA-1 algorithm. More specifically, it applies only if the timestamp for that signature is dated after Jan 1 2019.
(For now, the jvm technote indicates that "Any JAR signed with SHA-1 algorithms and timestamped prior to January 01, 2019 will not be restricted." It indicates further that "this exception may be removed in a future JDK release", meaning that in the future we may find that it WOULD apply also to jars signed with timestamps even BEFORE 2019.)
So the simple solution (really, it is fairly simple) to implement that workaround is that you can:
- Find the file java.security, which can be found (in Java 11) at the JVM's conf\security folder (so, for instance, on C:\Program Files\Java\jdk-11.0.17\conf\security\java.security)
- I would recommend that you make a backup of the file, so that you can revert to it if something goes amiss
- Edit the file and modify the line for jdk.jar.disabledAlgorithms to remove the value SHA1 denyAfter 2019-01-01 from the line, as well as the comma that followed the argument before it
- Note that while the Java update release notes refer also to a related jdk.certpath.disabledAlgorithms property, I did not find it necessary to modify THAT property, nor the cf update download verification to work.
- Save the file
- (I would recommend you keep the file open, until you confirm that CF restarts in the next step, in case you need to revert some mistake in your change)
- Restart CF. Assuming it comes up, you can proceed to try to download the update
- Enter the CF Admin and download the update (in the "Package Manager" page)
- After the download (whether before or after applying the update), you could undo the change to revert to the more "protective" new JVM behavior (until next need to update CF)
- I would recommend that if you DO revert the change, you should restart CF to make sure your reversion is correct. Otherwise, a later CF restart may fail if you make a mistake
- Be prepared to repeat this process when a new JVM update comes out--next planned for Jan 2023--if by then Adobe has not updated their download jars
Yes, that's a "number of steps" (and the other "workarounds" I offer below involve fewer ones). You can decide for yourself which workaround to use, at least until Adobe addresses this.
Note that the configuration line in question here appears on multiple lines (at least as of 11.0.17), as in the following:
DSA keySize < 1024, include jdk.disabled.namedCurves, \
SHA1 denyAfter 2019-01-01
You could comment out that last line by putting a pound sign (##) in front of it and then don't miss that you must remove the ", \" values at the end of the line above it, as in:
DSA keySize < 1024, include jdk.disabled.namedCurves
# SHA1 denyAfter 2019-01-01
If you decide to revert the change, be sure you change the values back to what is shown in the first example above.
Note As indicated in the Oracle release notes, they feel that their change (as of the Oct 2022 JVM updates) is one that you SHOULD want in place. Think of it like a doctor recommending a treatment. You ignore their recommendation at your risk, but you have that right to decide for yourself. Some will perform this change (my "workaround 1") only long enough to do the CF download and allow its verification. Others will do it and leave it in place, such as if they think that no other aspect of their CF application (or anything using the same, modified JVM) needs to worry about doing jar verificaton.
If this seems daunting, and you would like help, I can assist directly, remotely, and quickly via my troubleshooting consulting.
But again, this is just one of three workarounds I offer here.
Workaround 2: Change CF to use a Java 11 update from before Oct 2022, at least long enough to perform the update download
If you may be reluctant to do the first workaround, involving editing that java.security file, then another "simple" option is that you can simply change your CF instance to run with a Java version that is earlier than the Oct 2022 JVM update, which for the sake of this article is again 11.0.17 (since as of this writing, both CF2021 and 2018 support only Java 11, not earlier versions nor even later versions--for now).
All you need to do is change CF to use an earlier Java update, then restart CF, do the download in the CF Admin. (In workaround 3, I will propose how you can just skip using the CF Admin to download the update. Some people have reason to prefer using the CF Admin to do the download and install of CF updates.)
After doing the update, you could again change CF back to using 11.0.17 if you prefer: if you do that, consider my admonition in the previous workaround, where I suggest that you be sure to restart CF after you change it back to using the newer JVM. Otherwise, if you have made any mistake, you won't until you or someone else may restart CF, only to find it won't start because of your mistake.
I will not detail the process of changing CF to run as a different JVM version. Since I am writing to people who would have updated their CF to be using a Java version updated in Oct 22, I am assuming you or a colleague already know how to update your JVM, so again just change it back to something earlier, like even just 11.0.16.1 (which came out in July), or simply whatever Java version you were using before you updated it to this newer Java update.
I will say at least that I have a talk I did on the topic in mid-2022, "Updating the Java underlying ColdFusion", and I offer there the slides and a video recording. Or again I can help with direct assistance via my troubleshooting consulting.
Workaround 3: download the CF update manually, which therefore skips the jar verification process in the CF Admin download
Finally, recall that this problem is all predicated on a failure of the CF Admin update download feature to be able to verify the downloaded jar (due to the jvm update changing its behavior).
So yet another solution is to just download the jar manually. A link to it is offered within the update technote, such as that for CF2021 update 5, which offers a link to the cf2021 update 5 hotfix jar.
If you have internet access on the server, you can download it on the server and run the java -jar command against that jar (either running the commandline "as admin" on Windows, or running it as root or via sudo on Linux). The update technotes and CF docs discuss performing such a manual update.
Note, however, that if you do NOT have internet access on the server (where you want to update CF), with CF2021 it is NOT enough to just run that java -jar command. Instead, it's necessary to download also a zip of the CF2021 "packages", what's referred to as a zip of the "repository" (of "bundles"). Again, this is discussed in the technotes for updates to CF2021. (The concepts of "packages" and their being offered in "bundles" are new to CF2021 and above: they do not apply to CF2018 and earlier, so the previous paragraph's suggestion of just running the java -jar command against the hotfix jar will suffice for those on earlier CF versions.)
(I plan to do another blog post with more on this problem of how CF2021 changes things, about running that java -jar against the hotfix jar, if the server does NOT have internet access. I'll add a link here when I do that.)
So again, you have 3 choices for how to solve this problem introduced with the Oct 2022 Java updates, with respect to its inability to download and verify the hotfix jar. Consider the 3 choices above and decide which suits you best. (Or again I can help with direct assistance via my troubleshooting consulting.)
Some other issues to consider, related to this jar verification security matter
While I could leave things at the points I have made above, there are some other related points that readers might benefit to hear about. In the interest of time, I am going to post this as is, with all the above, and I will come back tomorrow (I hope) to expand on these points. Check out the headings to follow, in case any may catch your attention.
What Adobe needs to do
Again, this section no longer applies, since Adobe updated the jars finally in mid-February 2023.
As for ultimately solving this problem, Adobe needs to re-sign the update jars with a stronger algorithm than sha-1. I hope they will do that soon (it should not require a new update to CF itself, but just their dropping in corrected versions of the existing update jars.)
You can help get their attention by adding a vote to a bug report where this issue was raised the other day by someone I was helping. That was before I determined these "workarounds", so the focus of the bug report is on the "failed signature verification" error. Once I post this blog entry, I will add a link to it in the comment there, and I will encourage Adobe to fix the problem by updating the signature algorithm on their update jars.
If I learn that they do, I will try to update this post to clarify that. (Perhaps the blog post will still help folks using other than CF, who may hit similar jar signature verification errors after these Oct 2022 JVM updates.)
How you can confirm the error in CF's update.log
If it's of any interest, I'll note that one can find an indication of this problem in the "update log" within CF's logs folders (such as [cf]\cfusion\logs), which may show a link like this:
How this issue will need to be handled with each subsequent Java update, until Adobe corrects their jar signature
As I hinted at above, while it's great that we can implement my "workaround 1" change in our newly updated jvm, the fact is that change will help us only until we update a NEW jvm update, since Oracle can be expected to include this same change/security restriction in subsequent java updates.
Similarly, this will apply if/when Adobe may update CF to support Java 17 (as this Oct 2022 change applied to that JVM version as well). Again, hopefully Adobe will update their jar signature algorithm before then.
How you can verify the state of a jar's signature
In case you may wonder whether a hotfix jar IS or IS NOT signed with this old SHA-1 algorithm (perhaps if you're checking things in the future, after some time has passed since my writing this), it IS posible to detect that. The JDK versions of the JVM (not th JRE versions) include in their bin folder a tool called jarsigner. If we run that tool against the Adobe update jars (currently), adding as well the args -verify, it will report whether the jar will be treated as "signed" or not. Here first is an example of code you could run:
As for the messages, if you run a version of the jarsigner that's included in a JDK that's been updated since Oct 2022 (like 11.0.17), then the output will show:
Re-run jarsigner with the -verbose option for more details.
If instead you run it with a version of the jarsigner included in an OLDER JDK, then the output will show:
Warning:
The SHA-1 timestamp digest algorithm is considered a security risk. This algorithm will be disabled in a future update.
Re-run with the -verbose and -certs options for more details.
How this issue actually applies to the latest updates to more than just Java 11
Before wrapping up, I want to point out this issue does indeed apply to more than just the latest (Oct 2022) updates to Java 11. There were updates at the same time (and with the same change) to java 8 update 351(and even Java 7 update 361), but note that neither CF2021 and 2018 (the currently supported/updated versions of CF) support java 8 or earlier.
Then again, I will note that change regarding jar signatures was ALSO folded into the Oct 2022 update of Java 17 (17.0.5), but at least as the time I write this post, Adobe still has not updated CF2021 or 2018 to support Java 17.
Finally, though older version of CF did support Java 8 or earlier, note also how the JVM update release notes indicate how "Any JAR signed with SHA-1 algorithms and timestamped prior to January 01, 2019 will not be restricted.", and most CF updates for such earlier CF versions would be from before then. Even so, the release notes do also indicate that "This exception may be removed in a future JDK release", so it IS possible that in the future, some yet-NEWER JVM version WILL regard even such OLDER jar files as being unsigned, in which cases users of such OLDER CF versions (who update their JVM to such newer updates to whatever JVM their CF supports) may run into this issue.
Adobe bug report you can vote on
One more time, if you may want to add a vote to the bug ticket created reporting this problem, where I encourage Adobe in a comment to resolve this problem by recreating new update jars with a more secure signature, see the Adobe tracker ticket CF-4215607.
Conclusion
Until then, I hope that this post will help folks to at least better understand what's going on here. I know it was a lot of words. Perhaps some would connect the dots readily based solely on the brief verbiage in the JVM update release notes. I draw things out far more here, for those who may prefer to have those dots connected for them. And note that I DID start with the TLDR version of things at the opening here, trying to find a balance of too few vs too many words. :-)
For more content like this from Charlie Arehart:Need more help with problems?
- Signup to get his blog posts by email:
- Follow his blog RSS feed
- View the rest of his blog posts
- View his blog posts on the Adobe CF portal
- If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
- See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
as so often in the past, you helped me a lot with your post.
I updated the JRE of a ColdFusion 2016 instance (which I only use to run scheduled tasks because managing scheduled tasks under ACF is more comfortable than under Lucee :-) from jre8u311 to jre8u361. Unfortunately, none of the many scheduled tasks were executed afterwards and "SSL errors" and "ERROR Connection Failure" were reported.
Although your post refers to the ColdFusion update function in the admin interface, I tried your "workaround 1" and it solved my problem. All scheduled tasks are executed correctly again.
So once again I want to say a big thank you to you.
One may see errors referring to a ClassNotFoundException for "org/bouncycastle/asn1/DEREncodable" or perhaps "org.bouncycastle.asn1.DEREncodable". There may still other forms of the error message, or the tag may just fail and you might not readily see any error. And there are still other tags that have an encrypt or encryption attribute, which may be affected, such as CFPDF.
And the workarounds 1 and 2 that I share above could work: downgrade the JVM (not so wise) or use the JVM arg that allows you to tell the JVM to NOT treat as a failure the fact that CF needs to verify such jar that are not signed with modern algorithms (which also has its issues).
But in fact there may be a better solution for this specific issue: there's a fix from Adobe for this, which has at least worked for some people. See the bug report at https://tracker.adob..., and ask Adobe for the fix they're offering (which is NOT included in the March 2023 CF updates that came out this week).
Again, if that fix somehow doesn't work for you, I believe you'll find the root cause issue to be due to this jvm change, so the other solutions above could/should work. I'd welcome hearing from any who it this issue and how they may solve it.
Well now I have learned that if one updates the JVM used by the underlying CF add-on service (by default, what's pointed to by the file jetty.lax in the cfusion/jetty folder, and its line lax.nl.current.vm), then you will get errors reporting that the PDF can't be generated.
As discussed above in my "workaround 1" (https://www.carehart...), you can modify the java.security file and remove the value of "SHA1 denyAfter 2019-01-01" from the jdk.jar.disabledAlgorithms security property. Again, you must be careful about such changes. See above for more.
Then save that file, and restart the add-on service, and you may find you need to restart CF also (or use the trick I offer to reset the PDF service in the CF Admin here: https://community.ad...)
Then see if your PDF works. It's now clear to me that I ought to create a new post, as many people suffering such problems would never consider a post with this title as having their solution. :-)