Adobe recently released a performance brief claiming ColdFusion 9 is 40% faster than ColdFusion 8. While that number looks really good at first glance, Marc Ackermann was kind enough to point out that they were running ColdFusion 8 using the 1.6.0_04 version of Java, which has a known class loader bug.
While I don't doubt that ColdFusion 9 is faster than ColdFusion 8, I don't trust any of the numbers from the performance brief, especially a 700% improvement in CFC object creation.
I made this comment over on Ben's post, but I'll do it here. While the JVM does impact performance, Marc's comment, and yours, seems to imply that Adobe didn't do squat on the engineering side to help speed things up, which is simply not true.
ReplyDelete@Ray,
ReplyDeleteI wasn't trying to imply anything negative about their engineering efforts, so I'm sorry if it came out that way. I'm sure there were plenty of significant improvements made to help speed things up, and based on my experience, ColdFusion 9 does seem faster than ColdFusion 8. I just don't don't believe the improvements can be fully attributed to updates made to ColdFusion after having to deal with the 1.6.0_04 issues first-hand.
Why does that mean you don't trust the numbers? You're implying that they made them up or manipulated them somehow.
ReplyDeleteColdFusion ships with a version of Java that most people use and do not alter. The performance brief documents its exact methods and your attack is unfair especially as you are not providing any countering numbers merely speculation.
@Sam,
ReplyDeleteGood point. I should re-clarify myself a bit. I believe the numbers in the performance brief are accurate for comparing a default installation of ColdFusion 8 against a default installation of ColdFusion 9. However, it should be pointed out that an updated JVM is also partially responsible for the improvement gains that have been reported.
While you think it's unfair that I made this "attack" without any hard numbers to directly show the impact of an updated JVM, I think it's unfair of Adobe to make claims like "CFC object creation in ColdFusion 9 is 8 times faster than in ColdFusion 8" without immediately pointing out the differences in JVM. It's like saying Barry Bonds was a better home run hitter than Hank Aaron and Babe Ruth. Yeah technically Bonds hit more home runs, but you need to add an asterisk for steroid use in order to tell the full story.
If you handed the performance brief to a non-technical manager, they wouldn't question it for a second. They're definitely not going to take the time to read (or understand) the fine print.
With that being said, I'd be really intersted to see how ColdFusion 9 compares to ColdFusion 8 while running the same version of Java.
Only performance count I trust is how faster my applications is running.
ReplyDeleteI can't get a rid of impression when I make something to run 7+ times faster, under the same conditions, that it unambiguously means original code was "not good enough".
And, of course, that briefs like this one ARE made for marketing reason. Which is not bad. All companies that want to favor new version against previous one are doing that.
No matter how Adobe's engineers are good (and I have no doubt they are), number of 10000 frakkin' % is legendary :))))
All in all, faster or not, I am quite happy with new feature set of CF9.
I'm with you, Tony. At least part of those performance improvements have to be due to the JVM upgrade. (And anyone who read this as an "attack" on Adobe is way too sensitive if you ask me.)
ReplyDeleteI'll definitely buy that CF9 performs better than previous versions, but as far as exact metrics -- I'll only put my faith in an independent third party evaluation.
CF9 seems a lot faster to me... so who cares! ...so foodle to you and I will also add that cfc creation seems to be 'much' faster than 8... I only know this because a direct transplant of my framework code after upgrade from 8 to 9 ran twice as fast on startup... don't care why... just does (and it makes me happy)
ReplyDeleteand just in case the purists are reading... eval() works super quick too!
@Marko,
ReplyDeleteI love the new features as well. It's really hard going back to working on CF8 projects and not being able to leverage Hibernate and script-based CFCs.
@tonyg,
You said it better than I could. Thank you.
@Dawesi,
Depending on the JVM you were previously using, your framework might've ran twice as fast on ColdFusion 8 too if you had just upgraded the JVM. If you read the post by Sean Corfield that I linked to in the article, you'd see that his Model-Glue / ColdSpring / Transfer application went from loading in 30-60 seconds down to just a few seconds after upgrading his JVM.
CF9 is waaaayyyyyyyyyy faster than 8 on the last two JVMs. Do some ORM entityLoad tests, return a huge array of objects, and be prepared to pick your jaw off the floor.
ReplyDeleteI had done some initial comparison on this.
ReplyDeleteCF8 vs CF9 vs Railo 3.1
http://www.hemtalreja.com/?p=335
Enjoy!!!
-Hem
In principle, a good happen, support the views of the author
ReplyDelete