[Looking for Charlie's main web site?]

Breaking change in CF2021, new date format mask of D may be serious problem for old code

Note: This blog post is from 2020. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.
Wow. Beware of this subtle breaking change in CF2021, something discovered since its release (was not documented as one of the "new" things, nor was it documented at all in the beta). It can be a devastating problem that might lurk for days or weeks, corrupting your data.

In this post, I discuss the problem, as well as two solutions you can choose between: a) finding and changing the incompatible code, or (new since Dec 2020) b) implementing a new JVM arg that Adobe offered to revert this behavior back to how things worked before CF2021. You must do one or the other of these things, if you are affected by this issue. The update does not REVERT the behavior.

Updates since the original Nov 2020 post here

[Update 2, Mar 2021: Adobe now offers the option to use the new JVM argument (discussed in "update 1" below) as of CF2021 Update 1. But you do STILL have to implement the JVM arg--or change your code. Here is the JVM argument:

-Dcoldfusion.datemask.useDasdayofmonth=true

Again, if you are on update 1 or above, while you need not obtain the special fix jar file discussed below, you do still need to add the jvm arg above. The update does NOT alone revert the behavior introduced in the initial release of CF2021. For more details, read on.]

[Update 1, Dec 2020: the week after I posted this, Adobe created an optional fix for this. See my Dec 2020 post, where they have offered both a special hotfix jar (now included in update 1 and above) and a jvm argument that you can use, if you want to revert CF2021 to treating the "D" datemask format as it had in prior releases. Read on for an explanation of the problem, to appreciate the available solution. In time I'll do a separate post with more on the update.]

TLDR; a simple demo of the problem

Consider this fragment, which could exist in similar form in millions of CFML templates: 

dateformat("11-24-20","MM-DD-YY")

See anything wrong? Probably not. It will indeed "work fine" in CF2018 and before, producing 11-24-2020, as most would expect.

But that same code in CF2021 will produces instead 11-329-2020, which virtually no one would expect! Because D now means "day in year". It's a Java-standard datemask, but until now CF didn't complain if you used D. It treated it like d.

And the disaster is that you may have old code that's worked like this for years, but now in CF2021 that code will produce these WRONG RESULTS, which is bad enough if it's just a date shown on a page; but it's catastrophic if that date is then used in code to make decisions, to perform calculations, to store in databases, to pass to other systems, and so on.

And of course it applies if you use 2 D's, as in MM/DD/YYYY, or any variation like DD-MM-YY, and so on. It also applies to lsdateformat, and any functions or tags that take a date mask, such as dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, parsedatetime.

So yeah, this seems a huge deal!

What changed, why is this a problem, and what can you or Adobe do? what about Lucee?

What's changed is that a new dateformat mask of D was added (to mean "day in year"). Of course, the lower-case d mask has long existed and meant of course "day in month", as in dateformat(now),"m/d/yyyy". 

The problem is that it turns out CF was loose in its enforcement of this casing before, so you got the same "day in the month" with a "d" OR a "D", such as M/D/YYYY.  Until CF2021, that is. So what now?

For more, read on where I discuss what you can do, what Adobe might want to consider, and how the same "loose casing" of masks happens in Lucee as well (but Lucee has no D for "day in year", so does not have the "same" problem, yet). I also explain how this D for "day in year" is indeed a Java-standard mask value.

Running example

Here's a running example, first at cffiddle.org, where you can change the version to run it either as CF2021, CF2018, or CF2016.

And as an update, since first posting this, I had offered a link at the bottom to run the same code at trycf.com, to see it running on Lucee. But as I had noted then, that site didn't offer CF2021, but I have learned later today that it's being added.

How were we to know?

Again, this was not mentioned in the 500-page beta release notes, nor was it on the what's new page, both of which I discussed in my blog post on, "The many new and improved features in CF2021".

I only found this issue yesterday myself while working with a client, helping him find and understand what was going on, and how things were unexpectedly different.

And while the new mask value of D is indeed documented in the CFML Dateformat reference, it's NOT (currently) indicated there as being something new. I did a comparison with old versions of the docs to discern that it was indeed new. (There's also a new M and Y value, but I have not been able to tell how they differ in their results from the older m and y.)

The Java standard dateformat also supports "D" vs "d"

FWIW, I will note also that the use of the D for "day in year" is not something Adobe just "created out of thin air". It's been in the Java dateformat for years, both per the Java 5 docs (2004 timeframe). 

Again, the problem is not so much Adobe supporting such a "standard" dateformat mask--it's that until now, CF ignored the case and so folks have current apps that use "D" but mean (and found that CF acted as if they meant) merely "d". A HUGE difference (which perhaps Java never suffered if back in earlier versions they had not blithely allowed a "D" where only "d" was defined, if there was such a time).

What can you do?

So, that's the problem. Now you know. But what can you do at this point? And what might Adobe want to consider?

First, if you are concerned with or have been hit by this problem, please register a vote here, the ticket I encouraged my client to create, which he did today: https://tracker.adobe.com/#/view/CF-4210921 [Update: again, this is no longer needed. Adobe did solve this.]

Second, you could certainly TRY to search your codebase to find such formatting masks, so as to change any "D" references to "d" instead. [Update: see the next subsection, with some regex's to use in editors to find such masks.]

There are of course ways to use either editors or command line tools or search tools to search for any string in code (my favorite, for Windows, is File Locator Lite), and they can all be told to do a case-sensitive search.

But a challenge here is that there are so many mask variants you may have, such as M-D-Y, MM-DD-YY, MM/DD/YY, DD/MM/YY, and so on. It will be hard to pick a single string to be sure to find ALL such occurrences.

And you can't even try to limit it to references to dateformat, as you may also use dateformat, or still other CFML functions that use formatting masks, such as these (added after the original post): dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, and parsedatetime. See the comment to the post, below. And there are also tags that accepts a date mask, like cfinput or cfgridcolumn which both have a mask attribute.

Bottom line: I think it's going to be TOUGH for us to easily find, let alone rectify, any and all occurrences of such use of a capital D in a date mask. 

But forewarned is forearmed. Go forth and at least try.

Update: Some regex's to more easily find occurrences of this D in dateformat expressions

Since posting this blog entry, some kind community contributors have come to the rescue with regular expressions that you could use in any of various editors, to find and replace these troublesome dateformat usages. In particular, see comments below, first from Scott Steinbeck and then from Curtis Fraser, as well as the couple of comments before and after each that relate to the use of such regex's in different IDEs/editors. Thanks, guys!

What might Adobe want to consider?

Now, what might Adobe want to do about this?

Of course, it's probably too late to expect Adobe to change this behavior, such as to pick a NEW mask value to refer to "day in year"...but then since CF2021 is so new, perhaps they could.

Or perhaps they could add a switch (in the admin, app-level, or as a jvm arg) to allow turning OFF this new D mask, while perhaps also adding a new one. That way those who want to keep the "new" behavior can, but those who don't want to have so many compatibility issues (which may well be major business impacts) can choose to "keep the old way" for some period of time. (And yes, I realize a problem there is that many use such a crutch to then "do nothing"...but it's not like this is some security vulnerability if the "old way" was allowed to remain in perpetuity.)

[Update: Again, in December 2020 they did just that, adding a JVM arg (though you needed to obtain and implement a special fix jar as well). Then in Mar 2021 they incorporated that "fix" into CF2021 Update 1. You do still need to implement the JVM arg if you want to revert the behavior to allow D to act like d. For more, see my Dec 2020 post when the special fix was first released.]

What about Lucee?

For those who may wonder, I'll note that Lucee works the same as CF did BEFORE 2021--it DOES allow and treat "D" the same as "d".  And according to the Lucee docs for dateformat, it has no current mask for "day in year".  

I'll note also that it too makes no mention of how case is ignored (and none of the masks are capitalized).  So whether we can say "Lucee handles things better" seems an open question, but at least they can learn from this debacle and should NOT implement support for a "D", given how they also were loose in allowing a "D" to be left to mean the same as "d".

Finally, if it may interest anyone, here is a gist to run against Lucee the same code from the cffiddle above, using trycf.com.


So, quite a pickle.

What do you think of all this?

For more content like this from Charlie Arehart: Need more help with problems?
  • If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
  • See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
Related Blog Entries

Comments
The post here got discussed today on the wonderful Modernize Or Die podcast (https://cfmlnews.mod...). In the chat, Scott Steinbeck and I worked up a list of more tags and functions that use date masks. He found these functions:

dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, parsedatetime

And I found these tags: cfinput and its mask attribute, cfgridcolumn and its mask attribute

Of course, we can decry people using such UI tags, and it's not clear if their masks are affected by this change, but I'm noting them for the sake of completeness.

And Brad Wood wondered about a regex to help people find such mask use and Scott also started working on one, including to handle the variants above. He will hopefully post here if he comes up with one. Thanks, Scott!

Or if anyone else is up to the challenge, feel free. :-)
I first noticed this too when I installed MuraCMS 7.1 under CF2021...
Hi, Peter. Hope you'll let them know. They may be in a good position to resolve this in their code base, for the sake of all their users.
I have a feeling CF2018 might have done the same but can no longer confirm...
Peter, are you wondering if/asserting that this problem I raise here might have happened in CF2018? No, definitely not. I wouldn't have written the post as I did.

More to the point, if you look at the cffiddle.org link, you can actually RUN the code against CF2021 AND CF2018 AND CF2016, and the problem happens ONLY in CF2021.
As an update to my last comment, I find that somehow my post did NOT have the cffiddle link that I just indicated. To be clear, I HAD meant to offer the link, and had even referred to it in my post.

But I just noticed that somehow the link didn't make it in. I have corrected that, now under the new "Running Example" section near the top.
Hi. Thanks for this.
However, I believe there are some typos in your sample code.
This should be better:
https://cffiddle.org...
# Posted By GunterO | 11/25/20 11:22 AM
Gunter, those are not typos. The things you changed, like M to mm and Y to YY, etc were referring to the secondary observation made in my post, that Adobe had added also a new M and Y, and my observation (in the blog post and the code comment) that I wasn't seeing them doing anything different, even in CF2021.

But again, the crux of the article--and what's demonstrated in that code, whether changed as you have it or not--was about the D (or DD), and the surprising change in CF2021.

And more important is the matter that in prior releases, the D or DD was treated the same as d, and now no longer is.

But I appreciate that you were trying to help. Make sense now?
Thanks for your quick reply.
When I looked to the output, it made no real sense, because the "description" in front does not match the executed code.
dateformat("11-23-20","MM-DD-YY"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","MM-DD-YY"): 11-23-20
dateformat("11-23-20","MM-DD-YY"): 11-328-2020

That's why I changed it to :
dateformat("11-23-20","M-D-Y"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","m-d-y"): 11-23-2020
dateformat("11-23-20","M-D-Y"): 11-328-2020

Where the output matches the description in front.
But, I could be wrong of course.
# Posted By GunterO | 11/25/20 11:35 AM
Ah, got it. Sorry. Copy/paste error! When I created it, affecting the output site so I'd not noticed. I will correct it later today.  Thanks very much.
Thanks Charlie. The CF team is looking at it. We are exploring the possibility of a flag to define this behavior.
Wonderful to hear, Ashish, not only for considering the that but for responding directly. Thanks.
Hi Charlie,

As Ashish mentioned, to cater to the requirement of those users who would want the behavior of D to be similar to what was there in earlier versions, we will be introducing a new flag. We already have a fix ready and are currently testing it. We are also working on updating the documentation to call out this change.
# Posted By Vamsee | 11/27/20 8:49 AM
Again, so great to hear. Thanks.
Hi Charlie,

Thank you for flagging this. As discussed, we've provided a fix, the details of which are documented in the following.

1. https://helpx.adobe.... (also contains the hotfix location and installation details)

2. DateFormat- https://helpx.adobe.... (The history section has been updated)

3. New and changed functions in CF 2021- https://helpx.adobe....

Once again, a ton of thanks for your continuous help and support.

Thanks,
Saurav
# Posted By Saurav Ghosh | 12/2/20 2:36 AM
That's great news, Saurav. Thanks for sharing it. And I trust that you'll get a post out soon on it (on the forums or portal), though I may do another one also (on the portal), for those not seeing this.

A couple of things, though: while I highlighted the issue with D vs d, there would seem to be potential for issues with the other case-sensitive variants (Y vs y and M vs m). I would have thought the fix would have been to control case-sensitivity of masks rather than just about the D alone. Or do you have info from the engineers to suggest that's not an issue?

As for the DateFormat reference page, I see the history indicating this new JVM arg. I don't see it indicating that the D was added. I know some people are saying that this is not NEW in CF2021, but I really thought it was--or else we would have seen this problem sooner. Anyway, can you get someone to clarify WHEN it was added (and if it was removed and added back, perhaps), and get THAT indicated in the history? Same for Y and M.

Finally, you may want to get mention of this added to this doc:

https://helpx.adobe....
Fantastic and fast response from the Adobe CF team! <legends>
Quick check...
5974 lines w/ DateFormat
3648 of those with DD instead of dd.

Soo... thanks for being on top of this :-/
# Posted By Joseph Gooch | 12/6/20 5:17 PM
Hi, Joe. First, somehow your comment ended up in the post about the CF2021 new features. Given what you said, I assumed instead you were referring to this post on the Dateformat issue.

And did you happen to catch the comment from Adobe above, that they had addressed the problem with a new update? While all that code WOULD have failed, it need not if you apply the hotfix and JVM arg change.

I'll update this post to clarify that available fix and point to the Adobe comment, and then I plan to do a blog post also to help make the available resolution standout (and I will point to that here at the top of this post).
After looking at the possible arguments for all of the different functions I have narrowed it down to just dateformat and datetimeformat as well as their `LS` counterparts. The others don't fall victim to the same issue.

the regex pattern I have created is here:

((?<=l?s?(dateformat|datetimeformat)).+[DY]+[^\)]+)

This uses regex lookbehind to check for the existence of a D or Y in the dateformat/datetimeformat/lsdateformat/lsdatetimeformat functions.

should work both in script and tags

https://trycf.com/gi...
Scott, I appreciate the regex, but I feel like I'm missing something.

While your demonstration of the use of the java regex pattern matcher is interesting, and the 40k test cases compelling, can you help clarify how you'd see people using it to find and address the problem in their code? :-) That was your original intent, right?

It's just that I found issues with the more common ways I'd expect people might try to use it:

- when I used the regex (in your comment) in vsCode or CFBuilder, it matched all the dateformat examples above, whether using the upper or lowercase D (of course, using the setting in the Find to search on a regex). So that's curious.

- when I tried to use it in notepad++, its regex search flat said it was an invalid regex

- and even when I tried it in an refind function, it got "Malformed regular expression" and "Sequence (?<...) not recognized." Here's a gist of that:
https://trycf.com/gi... (which failed in all the CF versions or Lucee)

I tried the last example both to see if the CF regex engine would be different, and in case someone somehow might want to try to do such a scan of their code using CFML but maybe would not be allowed to do the createobject/cfobject to pull in the java regexer.

But of course it would seem the biggest concern is what I experienced in the editors. I tried to do a little digging to see if I could get it to work but I had to give up and wanted to get this comment out for you or others to consider.

Again, I do appreciate that you're wanting to help folks get a regex to help find code that would be troubled by this issue, especially if they can't use the new Adobe-provided JVM arg "fix".

Thanks, and looking forward to any thoughts.
Yes you can go ahead and edit my first comment. I think the plan was to implement it into the code checker.

I imagine the reason its failing in vscode and notepad++ would be due to parts of the regex expression not being recognized by their respective regex engine.

the tricky bit is the (?<... portion which is a 'lookbehind'). I can follow up with a compliant regex that would work for vs code, ill try notepad++ as well if it supports a different form of lookbehind. This part is required because otherwise, I can't confirm that the date mask is actually a part of the ColdFusion function dateformat.
Ok a little bit more testing, and here we go. A vscode/cfcodecheker/ possibly notepad++ (dont have it installed) compliant regex. I realized part of my need for the lookbehind was returning data vs just matching.

Here is the link https://trycf.com/gi...

Here is the regex:
l?s?(dateformat|datetimeformat)(.*?)[''"](.*?)[DY]{1,10}(.*?)[''"]

Pattern Breakdown:
- l?s?(dateformat|datetimeformat) --> find lsdateformat, lsdatetimeformat, dateformat, datetimeformat
- (.*?)[''"] --> lazy search until you find a single or double qoute
- (.*?)[DY]{1,10} --> lazy search until you can find a string containing between 1 - 10 characters of D and or Y
- (.*?)[''"] --> lazy search until you find a single or double qoute
FYI for VSCode you HAVE TO click the case sensitive "Aa" button in the search for it to use case sensitive
Hey, Scott, thanks for the regex and the explanation. It does indeed work, and in both VSCode and NotePad++. Hope that (and your code to search via CFML) may help those wanting to find and address this issue that way.

And as you say, if using either editor, we DO have to tick the option to make the find be case-sensitive. That may surprise some, if they figure that the point of the regex is that IT can itself control case-sensitivity in what it's matching. While that's true, the find features in the editors really won't pay attention to the case of what they find and match against (even if the regex has case-specificity) unless we do check that option.

Finally, sorry for the delay in responding to the comments here from a couple of weeks ago.
FYI: the regex that worked for me...
(date)(.*)\(.*(DD\/).*\)|(date)(.*)(date)(.*)\(.*(DD-).*\)|(date)(.*)\(.*(\/DD).*\)|(date)(.*)\(.*(-DD).*\)|(date)(.*)\(.*(D\/).*\)|(date)(.*)\(.*(D-).*\)|(date)(.*)\(.*(\/D).*\)|(date)(.*)\(.*(-D).*\)
# Posted By curtis fraser | 4/6/21 5:17 PM
Curtis, can you clarify for us all:
- what editors are you saying yours works in? And did you confirm it works in cfml itself?
- did you create that because for some reason the previous (simpler) one from Scott somehow did not serve you as well? How so?
- finally, it seems worth noting here that the bug fix mentioned above, from Adobe, is now rolled into Update 1 of cf2021, though that does not mean the error no longer happens. Rather, it means merely that we don't have to implement the special fix/jar before adding the jvm arg that reverts cf to the previous way of using D for dateformat.

Looking forward to your thoughts, since you did push on to find and share this alternative regex. Thanks.
Hi Charlie.

I used the regex in sublime, my current favorite editor.

The previous supplied regex definitely works to get any instance of dateformat or datetimeformat. You could also add in parsedatetime, dateadd, datepart if you wanted.

I created this regex to attempt to list any method that contained 'date', which in turn had capital D or DD as part of the mask.

previous regex would include items like DateFormat(defaultEnd,'dd')

after running my regex through a few systems I support, I realized that it was incomplete. The method text should be case insensitive, as some coders like to yell, others do not. This modified version seems to work to get them all.

(?i)date(?-i)(.*)\(.*(DD\/).*\)|(?i)date(?-i)(.*)(?i)date(?-i)(.*)\(.*(DD-).*\)|(?i)date(?-i)(.*)\(.*(\/DD).*\)|(?i)date(?-i)(.*)\(.*(-DD).*\)|(?i)date(?-i)(.*)\(.*(D\/).*\)|(?i)date(?-i)(.*)\(.*(D-).*\)|(?i)date(?-i)(.*)\(.*(\/D).*\)|(?i)date(?-i)(.*)\(.*(-D).*\)
# Posted By curtis fraser | 4/7/21 12:00 PM
OK, great. Thanks for the clarifications, and the code. :-)
I just got burned with this and replaced all instances of uppercase in my date masks. This is an excellent presentation of this problem. I am glad I didn't miss some obvious warning.
Thanks, Scott. And in case you missed it (or I add this for others subscribed to comments here also), Adobe did indeed provide a "fix" for this problem. I had created a new blog post in Dec 2020 when they did (and I updated this post to link to that), and that fix was incorporated into Update 1 of CF2021 in Mar 2021 (and I have updated both posts to mention that).

But as I clarify in both places: the "fix" Adobe offered is a JVM arg that one can add which changes the behavior BACK to letting the D mean the same as d. The fix does not ITSELF change the behavior back, but allows one to control it for a CF instance. And before update 1, one also had to get and implement a special hotfix jar file (discussed in my Dec 2020 post), but update 1 now incorporates the fix (and obviates the need of that jar).

That said, even as of Update 1, one still does need to enable that JVM arg to "revert the behavior" to how things worked before CF2021 (treating the D like a d). So this "breaking change" will remain in CF2021 going forward, it seems, unless one adds that special JVM arg.

If one does NOT want to (or cannot for any reason) rely on that jvm arg and prefers instead to find and replace all their occurrences of D in dateformat, some very good news is that previous comments here from Scott Steinbeck and Curtis Fraser offer regex's that could be used in editors/IDEs to find them. Perhaps you, Scott, were referring to them when you said you "replaced all instances of uppercase in my date masks".

I updated both my blog posts to offer a specific mention of and link to those comments, to help others use such a find/replace operation if they prefer that over relying on the new JVM arg. Again, thanks guys!
Hi Charlie - it gets worse! CF21 has another breaking (distorting) change in the DateDiff function.

In earlier versions of Coldfusion the mask "w" would return the number of weeks but now it returns the number of weekdays! If we want weeks we now need to use "ww".

https://helpx.adobe....

So we need to pick through our code for this one. Using "w" won't throw an error but it will give you unexpected results if you're expecting weeks your return will be out by a factor of about 7!

If Adobe wanted to introduce a weekdays mask they surely could have used "ww" for that and left the legacy "w" mask as was - seems rather inconsiderate to me, unless I've missed something.

Hope this helps someone.

Best wishes,

Nick
# Posted By Nick Way | 5/27/22 7:52 AM
* factor of 5, not 7
# Posted By Nick Way | 5/27/22 7:53 AM
Interesting indeed, Nick. But it seems it's not "new to CF2021". Note that the docs page you link to says:

"Note: The mask w as weekdays is only valid for the 2018 release of ColdFusion. For 2016 and earlier versions of ColdFusion, w returns the number of weeks."

It's kind of surprising to hear that no one seems to have noticed that for 4 years (that I've seen). Then again, most running it probably only switched to it (or later) in the years since, so such changes only become more widely known over time.

And FWIW, I'll confirm that I myself had not come across this change nor had I documented it for my "hidden gems in cf2018" talks back in 2018 and 2019.

So, at least it is documented. But I agree that such a change is significant and seems it should have been avoided. Thanks for sharing it.
As an update on this, Scott later offered a slightly different variant than his regex from Dec 2020 above. Here it is:

l?s?(dateformat|datetimeformat)(.*?)[''"](.*?)[DY]{1,10}(.*?)[''")]

Pattern Breakdown:
- l?s?(dateformat|datetimeformat) --> find lsdateformat, lsdatetimeformat, dateformat, datetimeformat
- (.*?)[''"] --> lazy search until you find a single or double quote
- (.*?)[DY]{1,10} --> lazy search until you can find a string containing between 1 - 10 characters of D and or Y
- (.*?)[''")] --> lazy search until you find a single or double quote or a ending parenthesis
Copyright ©2024 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting