XMHell: Handling 38GB of UTF-16 XML with Rust
As XML has gone from everybody&#x27;s darling to everybodys hate, it might be interesting to tell why we liked it so much in the past.  What was the problem it solved?<p>Encoding is the first one:  Most programs made assumptions about file encoding.  You got an american program and the user wanted parts of it translated&#x2F;configured in its own language?  Encoding problems were almost guaranteed, and you&#x27;ll have to live with losing some characters.  That&#x27;s in Western Europe, the rest of the world had it worse.  When XML came in, you had a good chance of getting UTF-8 or else one of the other UTF variants.  Finally.<p>Next up:  every program used its own config and data file formats.  Or multiple different formats for the same program.  You take the manual and learn a new one every 2 weeks.  Hooray, XML has only 1 set of rules to learn.<p>Escaping rules were different for each file format, or even incomplete or non-existent.  Feeding data form 1 program to another might work most of the time, and some day it breaks because someone, for the first time in 3 years, decided to start a sentence with a space.   Hooray, XML lets you escape any text you can think of, yes even PCDATA.<p>As a programmer, we stopped writing parsers for custom formats.  Hooray, take any of the shelf XML parser and you&#x27;re mostly done.  Byebye, yacc and flex.<p>The whole schema thing was a godsend.  You can validate the basic structure of your files before sending it off to a third party or putting it in a production machine.    There are tools to autocomplete or even generate test cases based on schemas.  Take that, custom formats from hell.<p>As a bonus, XML was big enough for an entire ecosystem of tooling to grow up around it, most of it cheap or free.  I can open an XML file in notepad++, click on &#x27;reformat&#x27;, have syntax highlighting, validation...   In the 90&#x27;s this was only possible for the most common of formats, and might cost 10 000 $ or more for an extremely limited, buggy tool delivered only by the vendor.<p>So XML was a very big step forward.  Things weren&#x27;t perfect, of course.  Spaces, entities, namespaces had enough pitfalls to keep you busy.  The fact that he takes an 38GB file and condenses it to 92MB speaks of it verbosity (Note how the author had  to sacrifice TAB&#x2F;CR&#x2F;LF to get there and invented a basic custom format, i.e. the 90&#x27;s attitude) .<p>Personally, I&#x27;d liked to see JSON extended with comments and dates, a bit better numeric support, and a default usage of schemas.  Then let TOML and YAML die in their respective little cradles.  It would have left us with exactly 1 format for data transfer and configuration that had XMLs advantages without its verbosity and enterprisyness.  But alas,  too late, that oportunity has passed.