<h2 dir="auto">STR</h2>
<div class="highlight highlight-source-rust notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="extern {
    static error_message_count: u32; // anything will do
}
pub static BAZ: u32 = *&amp;error_message_count;
fn main() {}"><pre class="notranslate"><span class="pl-k">extern</span> <span class="pl-kos">{</span>
    <span class="pl-k">static</span> error_message_count<span class="pl-kos">:</span> <span class="pl-smi">u32</span><span class="pl-kos">;</span> <span class="pl-c">// anything will do</span>
<span class="pl-kos">}</span>
<span class="pl-k">pub</span> <span class="pl-k">static</span> <span class="pl-v">BAZ</span><span class="pl-kos">:</span> <span class="pl-smi">u32</span> = <span class="pl-c1">*</span><span class="pl-c1">&amp;</span>error_message_count<span class="pl-kos">;</span>
<span class="pl-k">fn</span> <span class="pl-en">main</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span><span class="pl-kos">}</span></pre></div>
<h2 dir="auto"></h2>
<h1 dir="auto">Native pointer vectors</h1>
<p dir="auto">This bug is a <em>request for comment</em>.  I am but a novice Rust developer, but I suspect that this would be a strong addition to the language.</p>
<p dir="auto">The exact mechanics were fleshed out further in this bug; read down for more on what actually was implemented.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">While working on bindings for FFTW (the Fastest Fourier Transform in the West), I ran into the hairy issue that it doesn't seem possible to have Rust hold vector pointers to the outside world.  What do I mean by this -- and why do we care?  Well, I'll provide (a distilled version of) the FFTW APIs in question, by way of example:</p>
<div class="highlight highlight-source-c notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="double *fftw_malloc(size_t sz);
fftw_plan_t *fftw_plan_fft(size_t n, double *in, double *out, int flags); /* morally */
void fftw_execute(fftw_plan_t *plan);"><pre class="notranslate"><span class="pl-smi">double</span> <span class="pl-c1">*</span><span class="pl-en">fftw_malloc</span>(<span class="pl-smi">size_t</span> <span class="pl-s1">sz</span>);
<span class="pl-smi">fftw_plan_t</span> <span class="pl-c1">*</span><span class="pl-en">fftw_plan_fft</span>(<span class="pl-smi">size_t</span> <span class="pl-s1">n</span>, <span class="pl-smi">double</span> <span class="pl-c1">*</span><span class="pl-s1">in</span>, <span class="pl-smi">double</span> <span class="pl-c1">*</span><span class="pl-s1">out</span>, <span class="pl-smi">int</span> <span class="pl-s1">flags</span>); <span class="pl-c">/* morally */</span>
<span class="pl-smi">void</span> <span class="pl-en">fftw_execute</span>(<span class="pl-smi">fftw_plan_t</span> <span class="pl-c1">*</span><span class="pl-s1">plan</span>);</pre></div>
<p dir="auto">The general course of action when using FFTW is to use <code class="notranslate">fftw_malloc</code> to get a chunk of space to store your inputs and outputs, then create a "plan" with <code class="notranslate">fftw_plan</code> (a potentially expensive operation, but if you intend to run many FFTs, ultimately time-saving), and then finally execute the plan with <code class="notranslate">fftw_execute</code> when you've populated your buffers.  You can <code class="notranslate">fftw_execute</code> the same plan many times (and, in fact, if you can, you should!) after refilling the buffers and taking data out.</p>
<p dir="auto">In FFTW-land, <code class="notranslate">fftw_malloc</code> is not strictly necessary as the only way to get memory, but it's a Damn Good Idea -- and as I'll discuss in a moment, not using it doesn't save us.  <code class="notranslate">fftw_malloc</code> goes out of its way to obtain memory that has "nice" properties; for instance, it tries hard to align things not just to <code class="notranslate">double</code> boundaries, but also to cache lines, depending on how much memory you ask for.  If it knows anything else interesting about your system, it takes that into account.  So, it's not fatal to not use it, if that's the only thing you can do ... but it sure does hurt.</p>
<p dir="auto">Currently, in Rust, there is no way to import a pointer from the outside world and use it mutably, nor is there a way to tell Rust that this pointer may be written to by an outside API.  The closest thing that Rust has is <code class="notranslate">vec::unsafe::from_buf</code>, which in turn calls <code class="notranslate">rustrt::vec_from_buf_shared</code>, but the first thing that <code class="notranslate">vec_from_buf_shared</code> does is <em>allocate a new space and copy the memory away</em>!  This makes it unsuitable for referencing both the <code class="notranslate">in</code> and the <code class="notranslate">out</code> pointers; changing a pointer that has been imported through this mechanism will cause changes not to get written back to the outside world, and executing a transform to (and overwriting the contents of) a pointer that has been imported through this mechanism will cause Rust to not see the changes that happen after the copy.</p>
<p dir="auto">In this case, we have another option, though.  We could create a vector inside Rust, and use <code class="notranslate">vec::unsafe::to_ptr</code> to create a pointer to it.  This will work, but it is dangerously broken (violates safety) in three ways.  In the first, Rust is not in control of the lifecycle of the external reference; Rust cannot know when the external reference no longer exists, and may prematurely garbage collect the vector.  This can happen, for instance, in the case in which a reference to the <code class="notranslate">out</code> pointer is still live, and a reference to the plan is still live, but the reference to the <code class="notranslate">in</code> pointer is dead; calling <code class="notranslate">fftw_execute</code> on this <code class="notranslate">plan</code> will result in doing accesses to the dead <code class="notranslate">in</code> vector, which may have been garbage collected.</p>
<p dir="auto">In the second, Rust may reallocate memory out from under the external application.  The vector can be appended to, which may cause a reallocation to occur.  The external application's pointer will not be updated, and when it goes to access that pointer, that memory will be dead.  In the case of a "correct" usage, this will not occur (i.e., the programmer can be instructed not to do that), but this violates safety; at the very least, <code class="notranslate">+=</code> now becomes an <code class="notranslate">unsafe</code> operation.</p>
<p dir="auto">In the third, perhaps most compellingly, this locks Rust into a world in which it is not permissible to have a copying garbage collector.  Right now, Rust does ref-counting and garbage collection with free()... but this need not always be the case!  Another implementation of Rust could very conceivably experiment with garbage collection schemes for better performance.  If the behavior of pointers were already specified, it would be one story, but currently that behavior is not -- and I argue, for the better.</p>
<p dir="auto">In short, the language's existing capabilities for this are not sufficient to operate with at least one API.</p>
<h2 dir="auto">Use cases</h2>
<p dir="auto">The FFTW API is not the only system in which the current capabilities for referring to native memory are insufficient.  Consider also:</p>
<ul dir="auto">
<li>A framebuffer interface, in which an external resource has <code class="notranslate">mmap()</code>ed some memory.  Accesses must go to exactly those addresses; none else will do.</li>
<li>A command queue interface, in which a set of commands are provided for an external device (say, a graphics card) to DMA to and from chunks of memory.  (This is how modern graphics cards operate.)  Accesses must go to some prescribed address, usually managed through a remote resource manager (DRI, et al); no other address will do.</li>
</ul>
<p dir="auto">These are both examples of requiring a specific address... recall, however, that there are surely many applications that require some mutable chunk of memory, any mutable chunk at all!  Consider, for instance, the <code class="notranslate">ioctl()</code> interface on Linux, which has a similar "combination input/output buffer".</p>
<p dir="auto">The FFTW API is one of many cases that require mutable memory accessible to the native system.</p>
<h2 dir="auto">Proposed solution</h2>
<p dir="auto">I propose the addition of a vector qualifier, <code class="notranslate">[native T]</code>.  The type <code class="notranslate">[native T]</code> does not unify with <code class="notranslate">[T]</code>; it is mainly distinct from the normal vector, but that dereferencing indexes in it and iterating over it both work.</p>
<p dir="auto">The type <code class="notranslate">[native T]</code> has one introduction form:</p>
<div class="highlight highlight-source-rust notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="unsafe fn std::vec::unsafe::native&lt;T&gt;(ptr: *T, elts: uint) -&gt; [native T];"><pre class="notranslate"><span class="pl-k">unsafe</span> <span class="pl-k">fn</span> std<span class="pl-kos">::</span>vec<span class="pl-kos">::</span>unsafe<span class="pl-kos">::</span>native&lt;<span class="pl-v">T</span>&gt;<span class="pl-kos">(</span>ptr<span class="pl-kos">:</span> <span class="pl-c1">*</span><span class="pl-v">T</span><span class="pl-kos">,</span> elts<span class="pl-kos">:</span> uint<span class="pl-kos">)</span> -&gt; <span class="pl-kos">[</span>native <span class="pl-v">T</span><span class="pl-kos">]</span><span class="pl-kos">;</span></pre></div>
<p dir="auto">The following elimination forms of vectors function for native vectors:</p>
<ul dir="auto">
<li><code class="notranslate">v[a]</code> as an expression (with size checking)</li>
<li><code class="notranslate">v[a]</code> as an lvalue (with size checking)</li>
<li><code class="notranslate">for t: T in v</code> as a loop construct</li>
</ul>
<p dir="auto">Notably, the following form does not function:</p>
<ul dir="auto">
<li><code class="notranslate">v += vp</code> as an append</li>
</ul>
<p dir="auto">When a native vector goes out of scope, the native memory pointed to is not modified or otherwise operated upon.</p>
<p dir="auto">These are the basic rules for a native vector.</p>
<h2 dir="auto">Implementation</h2>
<p dir="auto">A native vector has the following internal representation:</p>
<p dir="auto"><code class="notranslate">type native_repr&lt;T&gt; = { len: uint, data: *T };</code></p>
<p dir="auto">It is distinct in the type system because it does not share a representation with a Rust vector.  This choice was made to avoid the performance cost of having to check at run time whether any given vector is a Rust vector or a native vector before accessing it.</p>
<p dir="auto">Translation is presumably very similar to Rust vectors.</p>
<p dir="auto">It could be the case that no <code class="notranslate">rustrt</code> support is needed for this, since the <code class="notranslate">native_repr</code> type can be constructed purely in Rust, and then can be <code class="notranslate">reinterpret_cast</code>ed into a native vector.</p>
<h2 dir="auto">Extensions</h2>
<p dir="auto">The above describes a basic semantics for a native vector.  It provides a semantics sufficient to behave safely, but missing are two potentially useful extensions.  These are optional, and certainly not required for a first pass implementation, but would make the native vector substantially more usable.</p>
<h3 dir="auto"><code class="notranslate">[native? T]</code> unification</h3>
<p dir="auto">There is currently a substantial vector library built up to operate on Rust vectors.  Just because some operations do not apply, it does not make sense to have to duplicate the ones that do.  For that, I propose the <code class="notranslate">[native? T]</code> qualifier, similar to <code class="notranslate">[mutable? T]</code>.  Presumably a separate code path would have to be emitted at translate time.  I do not know enough about the inner workings of <code class="notranslate">mutable?</code> to comment on how similar this might be, and how possible this might be given the existing Rust codebase.</p>
<h3 dir="auto">Built in destructors</h3>
<p dir="auto">It can be potentially useful to have Rust take over lifecycle management of memory, if the FFI binding builder is careful.  Classically, the mechanism by which one might do this is as such:</p>
<div class="highlight highlight-source-rust notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="type mem = {
  vec : [native f64],
  dtor : @mem_res
};

fn malloc(n : uint) -&gt; mem {
  let mem = fftw_native::fftw_malloc(n * 8u);
  ret { vec: unsafe { vec::unsafe::native(mem as *f64, n) }, dtor: @mem_res(mem) };
}

resource mem_res(mem: fftw_native::mem) {
  fftw_native::fftw_free(mem);
}"><pre class="notranslate"><span class="pl-k">type</span> <span class="pl-smi">mem</span> = <span class="pl-kos">{</span>
  <span class="pl-smi">vec</span> <span class="pl-kos">:</span> <span class="pl-kos">[</span><span class="pl-smi">native</span> f64<span class="pl-kos">]</span><span class="pl-kos">,</span>
  <span class="pl-smi">dtor</span> <span class="pl-kos">:</span> @<span class="pl-smi">mem_res</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span>

<span class="pl-k">fn</span> <span class="pl-en">malloc</span><span class="pl-kos">(</span><span class="pl-s1">n</span> <span class="pl-kos">:</span> <span class="pl-smi">uint</span><span class="pl-kos">)</span> -&gt; <span class="pl-smi">mem</span> <span class="pl-kos">{</span>
  <span class="pl-k">let</span> mem = fftw_native<span class="pl-kos">::</span><span class="pl-en">fftw_malloc</span><span class="pl-kos">(</span>n <span class="pl-c1">*</span> <span class="pl-c1">8</span>u<span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">ret</span> <span class="pl-kos">{</span> <span class="pl-c1">vec</span><span class="pl-kos">:</span> <span class="pl-k">unsafe</span> <span class="pl-kos">{</span> vec<span class="pl-kos">::</span>unsafe<span class="pl-kos">::</span><span class="pl-en">native</span><span class="pl-kos">(</span>mem <span class="pl-k">as</span> <span class="pl-c1">*</span><span class="pl-smi">f64</span><span class="pl-kos">,</span> n<span class="pl-kos">)</span> <span class="pl-kos">}</span><span class="pl-kos">,</span> <span class="pl-c1">dtor</span><span class="pl-kos">:</span> @<span class="pl-en">mem_res</span><span class="pl-kos">(</span>mem<span class="pl-kos">)</span> <span class="pl-kos">}</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span>

<span class="pl-en">resource</span> mem_res<span class="pl-kos">(</span>mem<span class="pl-kos">:</span> fftw_native<span class="pl-kos">::</span>mem<span class="pl-kos">)</span><span class="pl-kos"></span> <span class="pl-kos">{</span>
  fftw_native<span class="pl-kos">::</span><span class="pl-en">fftw_free</span><span class="pl-kos">(</span>mem<span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span></pre></div>
<p dir="auto">This has the unfortunate downside that a user of the API can extract the <code class="notranslate">vec</code> from the <code class="notranslate">type mem</code>, and let the <code class="notranslate">type mem</code> itself go out of scope (or otherwise become dead).  When the <code class="notranslate">type mem</code> itself becomes dead, the resource can immediately be freed, even though the native vector may still be live; this can violate safety.</p>
<p dir="auto">A native vector, then, may wish to have an introduction form that includes a resource pointer associated with it, for built-in cleanup.</p>
<h2 dir="auto">Conclusion</h2>
<p dir="auto">In this document, I describe a new form of vector called the native vector.  The native vector allows Rust to safely interact with external memory.  The introduction form is unsafe, so although it would permit inter-task shared memory communication, it does not do so in a particularly novel fashion.  This proposed solution addresses all of the mentioned use cases in what seems (to the untrained eye!) like an elegant, Rust-like fashion.</p>
<p dir="auto">Thoughts?</p>