<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/person142/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/person142">@person142</a> responsed to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="293736807" data-permission-text="Title is private" data-url="https://github.com/scipy/scipy/issues/8354" data-hovercard-type="issue" data-hovercard-url="/scipy/scipy/issues/8354/hovercard" href="https://github.com/scipy/scipy/issues/8354">#8354</a> that his preference was for a <a href="https://github.com/scipy/scipy/issues/8354#issuecomment-362469371" data-hovercard-type="issue" data-hovercard-url="/scipy/scipy/issues/8354/hovercard">cython optimize api</a>.</p>
<blockquote>
<p dir="auto">This came up before: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="218287443" data-permission-text="Title is private" data-url="https://github.com/scipy/scipy/issues/7242" data-hovercard-type="issue" data-hovercard-url="/scipy/scipy/issues/7242/hovercard" href="https://github.com/scipy/scipy/issues/7242">#7242</a>.</p>
<p dir="auto">As you can see I was against the idea. Handling all reasonable halting conditions for a scalar solver is already a PITA, and I think the problem gets much worse for a vectorized scalar solver. IMO it is better to provide a Cython API to a scalar solver and let users handle looping over it.</p>
</blockquote>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="293809929" data-permission-text="Title is private" data-url="https://github.com/scipy/scipy/issues/8357" data-hovercard-type="pull_request" data-hovercard-url="/scipy/scipy/pull/8357/hovercard" href="https://github.com/scipy/scipy/pull/8357">#8357</a> only vectorized Newton methods. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="297653505" data-permission-text="Title is private" data-url="https://github.com/scipy/scipy/issues/8431" data-hovercard-type="pull_request" data-hovercard-url="/scipy/scipy/pull/8431/hovercard" href="https://github.com/scipy/scipy/pull/8431">#8431</a> allows users to use a "tight loop" in C with Brent, Ridder, and other root finders.</p>
<p dir="auto">We decided it was okay to <a href="https://github.com/scipy/scipy/pull/8431#issuecomment-366143028" data-hovercard-type="pull_request" data-hovercard-url="/scipy/scipy/pull/8431/hovercard">split up Newton</a></p>
<blockquote>
<blockquote>
<p dir="auto">Is it okay to split up Newton, Halley's, and secant into different calls in cython_optimize?</p>
</blockquote>
<p dir="auto">I am personally pro this (wish it had been done that way in optimize), but maybe others will feel more strongly that the APIs should match.</p>
</blockquote>
<p dir="auto">We discussed which callbacks to support and the fact that <code class="notranslate">cython_optimize</code> should be pure C so it can free the GIL in <a href="https://github.com/scipy/scipy/pull/8431#issuecomment-366148174" data-hovercard-type="pull_request" data-hovercard-url="/scipy/scipy/pull/8431/hovercard">this comment</a></p>
<p dir="auto">This <a href="https://github.com/scipy/scipy/pull/8431#issuecomment-417216245" data-hovercard-type="pull_request" data-hovercard-url="/scipy/scipy/pull/8431/hovercard">commit</a> has links to annotated html that shows that <code class="notranslate">zeros_struct</code> and <code class="notranslate">zeros_array</code> are pure C, no Python so they can release the GIL to call Cython <code class="notranslate">prange</code></p>
<p dir="auto">The Cython optimize API in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="297653505" data-permission-text="Title is private" data-url="https://github.com/scipy/scipy/issues/8431" data-hovercard-type="pull_request" data-hovercard-url="/scipy/scipy/pull/8431/hovercard" href="https://github.com/scipy/scipy/pull/8431">#8431</a> was discussed in <a href="https://mail.python.org/pipermail/scipy-dev/2018-February/022453.html" rel="nofollow">this SciPy-Dev post</a> and <a href="https://mail.python.org/pipermail/scipy-dev/2018-April/022762.html" rel="nofollow">this one too</a> when I specifically asked about callback signatures.</p>
<p dir="auto">My issue is about converting a large coo matrix (stored as npz) to csr format. The number of entries (including duplicates) in the coo matrix is 16,940,976,557. I tried converting it, but everytime it goes out of memory, despite the fact that I have 500 GB of memory in my machine.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import scipy.sparse as sp
coomat = sp.load_npz(input_file)
sp.save_npz(output_file, coomat.to_csr())"><pre class="notranslate"><code class="notranslate">import scipy.sparse as sp
coomat = sp.load_npz(input_file)
sp.save_npz(output_file, coomat.to_csr())
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Done loading coomat in 1138.3824996530311 seconds
Traceback (most recent call last):
  File &quot;coo_to_csr.py&quot;, line 16, in &lt;module&gt;
    sp.save_npz(output_file, coomat.tocsr())
  File &quot;/home/sidak/.conda/envs/myenv/lib/python3.6/site-packages/scipy/sparse/coo.py&quot;, line 392, in tocsr
    indices = np.empty_like(col, dtype=idx_dtype)
MemoryError"><pre class="notranslate"><code class="notranslate">Done loading coomat in 1138.3824996530311 seconds
Traceback (most recent call last):
  File "coo_to_csr.py", line 16, in &lt;module&gt;
    sp.save_npz(output_file, coomat.tocsr())
  File "/home/sidak/.conda/envs/myenv/lib/python3.6/site-packages/scipy/sparse/coo.py", line 392, in tocsr
    indices = np.empty_like(col, dtype=idx_dtype)
MemoryError
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
1.1.0 1.15.1 sys.version_info(major=3, minor=6, micro=6, releaselevel='final', serial=0)"><pre class="notranslate"><code class="notranslate">import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
1.1.0 1.15.1 sys.version_info(major=3, minor=6, micro=6, releaselevel='final', serial=0)
</code></pre></div>
<p dir="auto">The coo matrix gets loaded in memory, but converting to csr, it creates another copy and then goes out of memory. Is there some more efficient way of doing this? or if I can in some way split the matrix in parts and combine?</p>
<p dir="auto">It would be great if you can help me out on this. Thanks a lot! <g-emoji class="g-emoji" alias="smile" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f604.png">😄</g-emoji></p>