Open Malaysia

One of the main reasons people opt for text based formats is for human readability. XML is ideal for the encoding of information which retains the ability for humans to interpret the information via a simple text based viewer. These human-friendly benefits should NOT be undermined by any efforts to "optimize" it for a specific application to perform better.

Attributes need to be descriptive. Values need to have units. Naming conventions should be consistent. Abbreviations should be avoided. Namespaces should be clear. These are golden rules in ensuring that your format is as accessible as possible.

My previous post about how the Microsoft Office Open XML (MSOOXML) file format handles (or mis-handles) dates brought on quite abit of discussion. One of the reasons was because it was for "speed and optimisation issues"

This is an interesting thing. I remember way back last year, Alan Yates, from Microsoft, was complaining about ODF's performance against MSOOXML. Actually it was OpenOffice.org versus Microsoft Office. Or rather OpenOffice Calc versus Microsoft Office Excel in calculating a spreadsheet. What was curious was that he commented:

Alan Yates, the general manager of Microsoft's information worker strategy [responsible for the Microsoft Office line of products], told ZDNet UK on Wednesday. "The Open XML format is designed for performance. XML is fundamentally slower than binary formats so we have made sure that customers won't notice a big difference in performance."

So how do you make loading faster if XML is fundamentally slower? Perhaps you map it as close as possible to the binary format? You sacrifice on well known XML structures so it conforms better to the architecture of your product? Perhaps forget about the user friendly units and just spew out different machine friendly values? Otherwise you could just output binary bitfields if its too much trouble mapping it out to something useful?  Forget international developers, just give the attributes names which the internal Microsoft programmers use with all the different naming conventions.

Whenever someone pushes "performance" over other factors in their code, I am always skeptical. Speed over readability is almost always not a good thing. It hides future problems of maintainability and expandability. It restricts innovation. Performance is extremely important if you are programming a graphics intensive computer game and every 1% of improvement is important, but in the domain of office productivity however, this should not be so a priority.

There will be some hardcore number crunchers who would benefit from faster processing, but if they are waiting 25 minutes for a 'sheet to calculate 5.5million cells, I would suggest them re-thinking their strategy on relying on a spreadsheet for this sort of work. Contract a programmer to re-architect the solution and re-write it with proper tools using a RDBMS with a customized UI. The time this project saves would pay eventually for itself.

Let us take a more technical look at this issue. We hear that MSOOXML would rather not support ISO 8601 dates for the sake of performance. Instead MSOOXML opts for a confusing, limited, and un-user friendly number format.

I just found out that back in January 2007, Rick Jelliffe declared:

 "The use of numeric forms for storing dates is obviously the result of a conscious trade-off of the load/parse time of dates versus the precalculation benefits of numbers. It might not be the format the we would choose if we had any expectation that the numbers would be entered by hand or read unmediated by a formatter, but for spreadsheets (and scientific data) it is not a high priority requirement; instead, size and load/parse time issues certainly are important."

Additionally, hAl, added to the discussion at the Grokdoc EOOXML objections page:

"The ISO 8601 standard is extremly extensive and put a lot of extra implementation effort to fully implement which is why w3c implemnets the subset which is reused by OOXML."

ISO 8601 looks like parsing this representation "2007-06-22T02:02:02" versus a number representation like "39255.6071286466" is far harder to implement and longer to execute.

How much slower is parsing these "extensive" dates? There is only one way to find out, and that is to write a program to parse and compare these different dates.

So I fired up vi, and wrote a C program, and compiled it using  gcc to do just that. That said, it has been over 10 years since I wrote my last C program, so please be easy on the quality of my code.

This program basically creates millions of dates in both formats and holds it in memory. I have removed the I/O factor from this because reading from the disk would take approximately the same time for both encodings.
The program will then keep track of the duration it takes to parse through the numbers, and ultimately produces an average time taken. The more numbers to crunch, the better the average you'd get. So I ran it with 5,555,555 dates. Thats over 5 million dates.

yky@x1407:~/src/isodates$ ./iso 5555555
Count: 5555555
long: 1182449222  ISO 8601: 2007-06-21T18:07:02

Parsed 5555555 IntDates.
      vInt[5555554]:  '1238004772' -> 1238004772 -> 2009-03-25T18:12:52
      Total time: 1.030s
  Each: 185 ns x1.00

Parsed 5555555 ISODates.
      vISO[5555554]:  '2009-03-25T18:12:52' -> 2009-03-25T18:12:52
      Total time: 1.560s
  Each: 281 ns  x1.51

As you can see, to calculate over 5.5million dates, my 1.7GHz Centrino laptop (about 2 years out of date now) takes 1 second to parse the numeric data type. This means that it takes a mere 185 micro-seconds to process one date.

[Update: 070716: First report was calculated in micro seconds. Its off by a factor by 1000. Its actually nano-seconds] This needs emphasising: this is 185 nano-seconds. Or 0.185 micro-seconds. Or 0.000185 milli-seconds. Or 0.000000185 seconds. In a blink of an eye (1 second), over 5 million dates can be converted from strings to the native date datatype.

To parse an ISO 8601 date, it takes 281 nano-seconds. A cynical statistician would tell you that this is over 50% slower than processing a native date type. It may seem slow, but if we were to put this in perspective, the total time taken to process 5.5million dates is only 1.56 seconds!

This means that the added benefits of ISO 8601 dates (bug-free, humanly readable, with unlimited range) only cost us half a second extra for 5.5 million dates.

The proponents against human friendly encodings will then backpedal and ultimately change tact and a new argument would usually take this form; as Mr Jelliffe commented a few days later in January 2007 in the same thread:

As for the argument that speed does not matter, in general I agree, but that does not mean it never matters. The particular always overrides the general. In the case of office documents, we are talking about a few seconds difference, but this is multiplied by hundreds of millions of users. You are talking of billions or hundreds of billions of seconds of lost productivity. I think there is even a good case to be made that for office document formats, efficiency of loading/opening should be a prime requirement.

Perhaps then to save the hundreds of billions of lost productivity Operating System vendors should cut down on computer boot-up times. Don't load up that fancy 3D GUI! Don't run any annoying taskbar notifications! I believe that much "productivity seconds" could be saved in many other areas, but compromising on readability on dates for performance would reduce very little real time. 0.56 seconds is nothing compared to the 60 seconds "wasted" during the boot-up of unnecessary applications.

What is the price of waiting for the added benefits of readability, usability and flexibility? Also please remember that hardly anyone populates spreadsheets with over 5 million dates. Hence the 50% penalty is a price I am more than willing to pay for readability. Plus I am banking on Mr. Moore and his Law to subsidise the cost in the near future.

Now we come to something more pertinent.

If you can remember, I brought up some issues regarding the contradictory definitions of Percentage units in MSOOXML back in January 2007.   To recap, instead of having four different forms of encoding Percentages, I recommended to Ecma to adopt a more "standardised" use of units, which is similar to what all HTML and CSS programmers around the world are used to. The examples of the recommendation was to use something like "tableWidth = 80%" or "zoomFactor = 40%" or "shadingCoverage = 62.5%"

Personally, I thought it was a good solution. However, the J1N8530-22 Ecma Responses to Comments and Perceived Contradictions PDF released after the Fast Track comments were submitted, basically brushed aside all the careful deliberations, issues and recommendations not just by Malaysia, but by all other National Bodies.

Their dismissal of this particular recommendation read:

"In §2.15.1.95, "zoom (Magnification Setting)", §2.18.97, "ST_TblWidth (Table Width Units)", and §5.1.12.41, "ST_Percentage (Percentage)", the percentage values allowed have been limited to integers to permit efficient parsing and processing. (Use of the % symbol and allowing non-integer decimal values would introduce parsing complexity.)"

Read that again: "the percentage values allowed have been limited to integers to permit efficient parsing and processing." and "Use of the % symbol and allowing non-integer decimal values would introduce parsing complexity"

I just wonder how all the web browsers do it on a daily basis when evaluating the HTML and CSS table cell widths so efficiently?

The term "non-integer decimal values" sounds so "difficult". Non-integer decimal values in this case actually means "Floating point numbers." We are no longer in the days where math-coprocessors are a  rarity. It may have been an exotic feature on your CPU 15 years ago, but nowadays, its a commodity.

To test this claim by Ecma, I modified my program to also cater for floats. I then processed 5.5 million non-integers decimals with the % symbol appended and this is what I got:

yky@x1407:~/src/isodates$ ./iso 5555555
Parsed 5555555 Percentages.
      vPerc[5555554]:  '55.54%' -> 55.54 -> 55.54%
      Total time: 4.530s
  Each: 815 ns  x4.12

Yes, it took longer. It took over 4 times longer than a date. But note; dates in MSOOXML are actually formed as 39255.6071286466. And guess what? They are non-integer decimal values too! The only difference? Its a trailing "%" percentage sign in the string.

Because my C skills are not as sharp, to decode "55.54%", I just used the standard built in function:

sscanf( vPerc[i], "%f%%", &vpct );

This obviously is not the best way to parse a Percentage. Given a little bit more effort, a more efficient way can be found. It is hard to believe that parsing an ISO 8601 string with so many more elements (year, month, day, hour, min, sec) is faster than parsing a HTML percentage string, but writing the fastest parser is not the purpose of this exercise.

The purpose of this entry is to show that even if a "non-integer decimal value with a trailing percentage symbol" took 4 times longer than a "non-integer decimal value", the difference to read 5.5 million numbers is a mere 3 seconds. Surely we can trade at most 3 seconds for a readable and human friendly representation of your data?

This commentary would hopefully put to rest the myth that user friendly data-types in XML would cost significantly in performance when parsing through the files. We are fortunate nowadays to live in a world where multi-gigahertz computers are the norm. Faster technology has also spurred the popularity of XML where users do not have to worry about the cost vs benefits of human friendly representations of their data versus their applications performance.

If there are vendors, consultants contracted by vendors, or standards bodies controlled by vendors out there who state that the reason why their version of XML is so hard to read because it has been optimized for speed, then my little C program clearly demonstrates that encoding values in human readable forms cost very little in terms of performance and should not be traded away so cheaply.

yk.

[Update: 070716 - Thanks to Brian Jones' calculation I realised that my units stated was off by a factor of 1000. The units should be in nano-seconds instead of micro-seconds. So to calculate 24 x 10,000 dates in a spread sheet (total = 240,000 dates), it should take no longer than 0.07 seconds, and not 67seconds as per his calculations ]