<p dir="auto">I've been trying to find out the process of creating DataFrames in order to try to solve <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="8521212" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/2305" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/2305/hovercard" href="https://github.com/pandas-dev/pandas/issues/2305">#2305</a> with minimal memory use. I've made some tests that put in a IPython notebook: <a href="http://nbviewer.ipython.org/gist/tinproject/7d0e0de9475b16910fcf" rel="nofollow">http://nbviewer.ipython.org/gist/tinproject/7d0e0de9475b16910fcf</a></p>
<p dir="auto">Currently when reads from a generator pandas internally allocate the whole generator in memory within a list, using at least twice the memory needed. I suspect sometimes could be even more if there is a later type conversion needed.</p>
<p dir="auto">Also, as it can be viewed in the notebook, the input data collection it's completely readed many times before data it's loaded in the final data structure.</p>
<p dir="auto">To read from an iterator, without put the whole it in memory, it's needed to process the values that yield one by one, or at least in a one pass form.</p>
<p dir="auto">One it could read data from a iterator in one pass, it could read from every iterable collection only needing the count of elements to read. It includes collections with implicit lenght: sequences(list, tuple, str), sets, mappings, etc. and collections with unknown length: generators, iterators, etc. which size must be explicitly given.</p>
<p dir="auto">I think this could end in a great improve on performance and enhance the input data types accepted (In my tests I saw that the tuple collection it's not a valid type for DataFrame!)</p>
<p dir="auto">Relates to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="8521212" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/2305" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/2305/hovercard" href="https://github.com/pandas-dev/pandas/issues/2305">#2305</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="8188284" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/2193" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/2193/hovercard" href="https://github.com/pandas-dev/pandas/issues/2193">#2193</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="25370910" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/5898" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/5898/hovercard" href="https://github.com/pandas-dev/pandas/pull/5898">#5898</a></p>
<p dir="auto">I want to help to solve this, but it's a bis task, and I'm lost when comes to the Blocks part. Is there anywhere documented what structure have and how Blocks works?</p>
<p dir="auto">Hi,</p>
<p dir="auto"><code class="notranslate">Series.irow(i)</code> (and also <code class="notranslate">Series.iget(i)</code>), with <code class="notranslate">i</code> being an integer, returns more than one value on Series with non-unique indexes if the index at location <code class="notranslate">i</code> has duplicates. This is not the behavior I expected and it is also different from the behavior shown by <code class="notranslate">DataFrame.irow(i)</code>, since it only returns the row located at position <code class="notranslate">i</code> regardless of any existing index duplicates. Also the documentation for <code class="notranslate">Series.irow</code>/<code class="notranslate">Series.iget</code> specifies that an "int" parameter should return a "scalar" value, so I guess this might be a bug.</p>
<p dir="auto">Here is some sample code I've run using pandas '0.8.2.dev-f5a74d4':</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [52]: s = Series([0, 1, 2], index=[0, 1, 0])

In [53]: s.irow(0)  # here I was expecting 0
Out[53]: 
0    0
0    2

In [54]: s.irow(1)
Out[54]: 1

In [55]: s.irow(2)  # here I was expecting 2
Out[55]: 
0    0
0    2
"><pre class="notranslate"><code class="notranslate">In [52]: s = Series([0, 1, 2], index=[0, 1, 0])

In [53]: s.irow(0)  # here I was expecting 0
Out[53]: 
0    0
0    2

In [54]: s.irow(1)
Out[54]: 1

In [55]: s.irow(2)  # here I was expecting 2
Out[55]: 
0    0
0    2

</code></pre></div>
<p dir="auto">and its DataFrame counterpart:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [56]: df = DataFrame([[0], [1], [2]], index=[0, 1, 0])

In [57]: df.irow(0)
Out[57]: 
0    0
Name: 0

In [58]: df.irow(1)
Out[58]: 
0    1
Name: 1

In [59]: df.irow(2)
Out[59]: 
0    2
Name: 0"><pre class="notranslate"><code class="notranslate">In [56]: df = DataFrame([[0], [1], [2]], index=[0, 1, 0])

In [57]: df.irow(0)
Out[57]: 
0    0
Name: 0

In [58]: df.irow(1)
Out[58]: 
0    1
Name: 1

In [59]: df.irow(2)
Out[59]: 
0    2
Name: 0
</code></pre></div>
<p dir="auto">Thanks and regards.</p>