Open source impeded by inconsistencies in the Office Open XML document format

When it comes to office documents, public administrations can choose from two ISO/IEC standards. Only one of these, ODF (ISO/IEC 26300), is vendor-neutral, open and reliable across a span of years and software versions, and supported by a variety of software products.

The later OOXML standard (ISO/IEC 29500), originally developed by a single proprietary software vendor, is implemented in three different versions (‘ECMA’, ‘Transitional’ and ‘Strict’) that are not compatible with each other. Although the ‘ECMA’ and ‘Transitional’ versions are outdated – ‘Transitional’ had only been accepted as a temporary solution to give the software vendor time to implement ‘Strict’ in its products – they both continue to be used in practice. This is because older versions of the vendor’s office suite (MS Office) cannot read or write OOXML Strict and are unlikely ever to gain such abilities.

This inconsistency causes a variety of problems for developers of free and open source office solutions, including:

Because of this, to date there are no free and open source solutions that fully support OOXML.

Problems with the document format also negatively affect public administrations. Archive experts warn that inconsistencies between the OOXML versions will cause problems with older documents, so that users may not be able to open or work with them. University studies show that even the proprietary vendor’s own products cannot deal with the different OOXML features properly.

Nobody is using only OOXML Strict

Michael Meeks, LibreOffice developers

Development of the OOXML standard began in 2004 at ECMA International (formerly the European Computer Manufacturer’s Association), a private, membership-based non-profit standards organization. Two years later, ECMA involved the international standardisation organisation ISO, which is composed of representatives from various national standards organisations. OOXML (also known as Office Open XML) became an industry standard alongside ODF, the Open Document Format used by a variety of products including LibreOffice, Apache OpenOffice and Google Docs.

OOXML’s original author, Microsoft, promises that its Office software fully supports OOXML as the ‘most-used document standard’. But there is a growing debate about the standard. Both its independence and the possibility of re-implementing it in other software products have become matters of intense discussion among free and open source software developers and public administrations.

Michael Meeks is one of the main developers of LibreOffice. Formerly at Suse, he now works for the LibreOffice support company Collabora. Meeks is convinced that OOXML (ISO 29500) is not being used ‘in the wild’, and says there will never be an OOXML standard independent of Microsoft. ODF, on the other hand, is a standard suitable for long-term document editing and archiving. ODF is supported by a variety of organisations, including many large ICT firms, whereas the only organisation claiming to support OOXML is its one and only vendor.

Furthermore, according to Meeks, it is extremely unlikely that any administration is really working with the ISO-compliant OOXML standard only. Organisations will have users working with older versions of the MS Office suite, he says, or they will be working on older documents. They will be exchanging documents with citizens, companies or administrations using office tools that do not fully support ISO 29500, and that will break the chain of OOXML Strict standard documents, often without informing users. Meeks continues: If a company or an administration uses different versions of MS Office, including old ones like MS Office 2007 or 2010, then documents exchanged through a chain of people will not stand up to the ISO Strict standard of OOXML as specified in ISO 29500.

For the typical daily use of a connected company or a municipality working with customers, citizens or partners, the inconsistencies around the OOXML standard are likely to cause problems sooner or later. Typically, a document might be created by someone using MS Office 2013, then edited and re-saved by a colleague with MS Office 2007 or 2010. Because the old versions of Microsoft Office do not comply with the latest, ‘Strict’ OOXML standard, metadata is very likely to get lost during such round-trips. And since the ‘Strict’ standard used by Microsoft is still neither fully documented nor open (it contains references to Microsoft websites, some of which no longer exist), data loss on conversion is a widespread and well-documented phenomenon. What makes this problem even worse is the absence of error messages, and a file extension (for example .docx for text documents) that does not show the user which format has been used to save the document.

Even though OOXML Strict is a standard which theoretically could be implemented in open source, its specification is so complex that we would have to re-implement huge amounts of detail from MS Office to fully comply with it, says Meeks. But an even bigger problem for developers like Meeks is support for OOXML in the wild, encompassing the different dialects and how these are supported in the different office suites. To me it remains unclear why anyone would choose to give a single, rather dominant vendor such a huge head start in the implementation of a document standard, when there are better, more open, document standards to mandate with wider implementation support, he says.

OASIS ODF TC member Svante Schubert, another of the veteran engineers from OpenOffice working on the interoperability of office suites, concurs: You can develop an OOXML-compliant implementation, covering all the mandatory OOXML features, but this implementation will very likely not be compatible with the majority of the OOXML documents that are out there.

This is due to the many ‘optional features’ hidden in the specification. Many of these have historical roots, and they create redundancies in OOXML, Schubert explains: From the many available examples, just pick tables. Why are there three different table formats in OOXML? There is one for Word, one for Excel and one for PowerPoint. The answer is: Because the three different departments were working separately on the format. Microsoft either bought or created these products separately, and during the specification phase they had to integrate these three different approaches in a single XML.

Developers and standardisation experts agree that standardisation should not work like this. Such examples show that the OOXML standard is driven by the needs of a software company more than by the procedures and ideals of normal industry standards, says Schubert. ODF, on the contrary, has only one specification for tables. This has been developed to ensure compatibility, and is used across all the different application components.

A report by the European Commission, part of Action 23 in the EC’s Digital Agenda, confirms this argument. The paper indicates that the OOXML format contains severe barriers to implementation. In its final report, the Guidelines for Public Procurement of ICT Goods and Services: SMART 2011/0044 declares: Whilst standards that are set through formal standard setting organisations go through a formal development process, they may still contain barriers to implementation by all interested parties. The report illustrates this point with the OOXML standard in particular (footnote 32): As an example, ISO standard (ISO/IEC 29500) for document formats. The technical specifications of this ISO standard include references to proprietary technology and brand names of specific products. Further, the specification of this ISO standard is not complete (i.e. the technical specification contains references to an external web site which refers to web pages on the vendor’s homepage that are currently not available.

With OOXML as a heterogeneous and ambiguous standard, and with Microsoft holding the threads and not updating old software versions, every software developer has to deal with a growing set of separate implementations, software versions and different OOXML ‘flavours’. This creates a complexity of problems, with each combination behaving slightly differently on operating systems ranging from Windows XP to Windows 8, with their various sub-versions, patch levels and service packs. Free software developers trying to fix office interoperability issues must not only grapple with the OOXML variations but also test their fixes over a wide variety of operating systems, Office versions, documents and implementations. This would not be necessary with a single, unambiguous and open ISO standard.

According to developers, such problems can be found in many places within OOXML. And it is not only the open source software movement that is suffering from the varying implementations and interpretations of ISO 29500. Even Microsoft has distinct problems with its document formats.

This is illustrated clearly, even for non-technical readers, by recent observations from Italo Vignoli, published on his blog Marketing FLOSS. Vignoli, a journalist from Milan, is a specialist in Open/LibreOffice and the director of the Document Foundation, the home of LibreOffice.

Vignoli shows how in five steps, all common spreadsheet operations, even Microsoft’s most recent Office version, Office 2013, does not properly support Open XML Strict, the declared document standard for all the vendor’s products. Creating a spreadsheet in MS Excel, adding some data and some metadata such as dates, calculations or formulas, and saving the document in the ISO-standardized OOXML Strict format can result in trouble when the document is re-opened.

Problems can include data loss and features wrongly represented, which might easily remain undiscovered because they do not produce error messages in Microsoft Office. In the case of Vignoli’s spreadsheet, for instance, all the user sees is some wrong dates, maybe some wrong XML tags – mostly things that are likely to go unnoticed, yet which could be disastrous to a public administration. Note, moreover, that this happened in Office 2013, which according to Microsoft has the best and most complete OOXML support of all the Office versions, and which is the only software product currently available that claims full compatibility with OOXML Strict.

The number of examples of errors like Vignoli’s proves that the problem lies much deeper than an ordinary software bug. In January 2014, the Hungarian developer Miklos Vajna presented a preview of the newest version of LibreOffice Writer (4.3). Writer is a complex text editor with many import and export functions – the LibreOffice equivalent of MS Word. IT administrators use it to migrate documents from one format to another, not only because of the quality of its import/export filters but also because of the ease with which these filters can be created and used.

One of the major feature improvements of Writer 4.3 will be the handling of shapes in files imported from MS Office documents using the docx import filter. This is no easy task, however, because Vajna has shown that Office itself interprets the same file differently: where Office 2010 shows a green triangle, Office 2007 has a red one. And this is just one example.

Microsoft’s own lack of consistency gave the LibreOffice developers a problem. As in so many other cases, again they had to resort to reverse engineering, yet at the end of the day they still had to choose whether to be compatible with the 2007 or the 2010 version of Office. Vajna admits: At least we’re now on a par with Word 2010, only to continue with a list of other features that do not survive the round-trip process of saving, exporting, loading, changing and exporting again. With bitter irony he explains the reverse-engineering work that open source developers have to do simply because the standard is not as open as it should be, and because different Microsoft products interpret it