<p dir="auto">Currently (pending <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="17611345" data-permission-text="Title is private" data-url="https://github.com/rust-lang/rust/issues/8287" data-hovercard-type="pull_request" data-hovercard-url="/rust-lang/rust/pull/8287/hovercard" href="https://github.com/rust-lang/rust/pull/8287">#8287</a>), libextra will have the following encodings</p>
<ul dir="auto">
<li>json</li>
<li>ebml</li>
<li>base64</li>
<li>hex</li>
</ul>
<p dir="auto">And in theory we should add these in the future:</p>
<ul dir="auto">
<li>xml</li>
<li>base32</li>
<li>ASN.1</li>
</ul>
<p dir="auto">(or at least those are the ones I could think of off the top of my head). Right now they're all placed directly inside the <code class="notranslate">libextra</code> folder, but they would probably benefit from a better organization scheme.</p>
<p dir="auto">Currently all of the "encodings" can generally fall into two categories. One is "bytes to strings and back" and the other is a "generic serialization format".</p>
<p dir="auto">Right now I believe that the "generic serialization format" is captured nicely via the <code class="notranslate">serialize::{Encoder, Decoder}</code> traits, and it unifies json/ebml. <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/omasanori/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/omasanori">@omasanori</a> has brought up concerns about the naming scheme, and my proposal below might alleviate these concerns.</p>
<p dir="auto">Otherwise, the "bytes to strings and back" type of encoding isn't very well unified at this point. There should be a trait (in possibly a new <code class="notranslate">extra::encoding</code> module) which unifies base64/base32/hex and whatever else we come across. In my opinion, the trait should look like:</p>
<div class="highlight highlight-source-rust notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="trait Text {
  fn encode(&amp;[u8]) -&gt; ~str;
  fn decode(&amp;str) -&gt; ~[u8];
}"><pre class="notranslate"><span class="pl-k">trait</span> <span class="pl-smi">Text</span> <span class="pl-kos">{</span>
  <span class="pl-k">fn</span> <span class="pl-en">encode</span><span class="pl-kos">(</span><span class="pl-c1">&amp;</span><span class="pl-kos">[</span><span class="pl-smi">u8</span><span class="pl-kos">]</span><span class="pl-kos">)</span> -&gt; ~<span class="pl-smi">str</span><span class="pl-kos">;</span>
  <span class="pl-k">fn</span> <span class="pl-en">decode</span><span class="pl-kos">(</span><span class="pl-c1">&amp;</span><span class="pl-smi">str</span><span class="pl-kos">)</span> -&gt; ~<span class="pl-kos">[</span><span class="pl-smi">u8</span><span class="pl-kos">]</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span></pre></div>
<p dir="auto">This makes it clear what the "encode" procedure is and what the "decode" procedure is. Additionally, it's explicitly clear that bytes live on <em>one</em> side, and strings live on the other, there's no intermingling between the two of them.</p>
<p dir="auto">Finally, I think we should move everything around into:</p>
<ul dir="auto">
<li><code class="notranslate">extra::encoding::base64</code></li>
<li><code class="notranslate">extra::encoding::base32</code></li>
<li><code class="notranslate">extra::encoding::hex</code></li>
<li><code class="notranslate">extra::encoding::json</code></li>
<li><code class="notranslate">extra::encoding::ebml</code></li>
</ul>
<p dir="auto">And then <code class="notranslate">encoding/mod.rs</code> would define the traits</p>
<ul dir="auto">
<li><code class="notranslate">Text</code> (is the name too short? if you reference it by <code class="notranslate">encoding::Text</code> I think it's clear but that's not always guaranteed)</li>
<li><code class="notranslate">Serializer</code> (<code class="notranslate">serialize::Encoder</code> today)</li>
<li><code class="notranslate">Deserializer</code> (<code class="notranslate">serialize::Decoder</code> today)</li>
</ul>
<p dir="auto">What do others think about this? If this looks good, I'd be more than happy to reorganize everything!</p>
<p dir="auto">So, I have the following</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="mod tokenizer {

    #[test]
    fn test_tokenize_one() {
        //Numbers
        let string = ~&quot;1&quot;;
        let string = ~&quot;321&quot;;

        //Strings
        let string = ~&quot;'This is a string'&quot;;

        //Operators
        let string = ~&quot;⍒&quot;;
        let string = ~&quot;÷&quot;;

        //Comments
        let string = ~&quot;⍝&quot;;
        let string = ~&quot;⍝ lol&quot;;

        //Function definition
        let string = ~&quot;∇Function&quot;;

        //Close function
        let string = ~&quot;∇&quot;;

        //Endlines
        let string = ~&quot;\n&quot;;

        //Variables
        let string = ~&quot;⍙var&quot;;
        let string = ~&quot;∆var&quot;;
        let string = ~&quot;var&quot;;
    }
}"><pre class="notranslate"><code class="notranslate">mod tokenizer {

    #[test]
    fn test_tokenize_one() {
        //Numbers
        let string = ~"1";
        let string = ~"321";

        //Strings
        let string = ~"'This is a string'";

        //Operators
        let string = ~"⍒";
        let string = ~"÷";

        //Comments
        let string = ~"⍝";
        let string = ~"⍝ lol";

        //Function definition
        let string = ~"∇Function";

        //Close function
        let string = ~"∇";

        //Endlines
        let string = ~"\n";

        //Variables
        let string = ~"⍙var";
        let string = ~"∆var";
        let string = ~"var";
    }
}
</code></pre></div>
<p dir="auto">And for some reason, on both my OSX build of Incoming, and my on my Ubuntu build of it, I get an error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="rust: task failed at 'Assertion bpos == mbc.pos || bpos.to_uint() &gt;= mbc.pos.to_uint() + mbc.bytes failed', /home/tony/Projects/rust/src/libsyntax/codemap.rs:500"><pre class="notranslate"><code class="notranslate">rust: task failed at 'Assertion bpos == mbc.pos || bpos.to_uint() &gt;= mbc.pos.to_uint() + mbc.bytes failed', /home/tony/Projects/rust/src/libsyntax/codemap.rs:500
</code></pre></div>
<p dir="auto">When I start using the variables (such as sticking Print statements after each definition of it) it compiles fine. Taking out the weird APL symbols that my interpreter is going to be testing also fixes it.</p>
<p dir="auto">Is there an issue with having too many unused instances of a variable? Or with trouble with (super obscure) Unicode glyphs?</p>