<p dir="auto">Sorry, long issue :-)</p>
<p dir="auto">In issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="37810421" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/7750" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/7750/hovercard" href="https://github.com/pandas-dev/pandas/issues/7750">#7750</a> there was some heated discussion about backwards compatibility and the fact we did or did not care enough about it. I don't want to start a new discussion on that topic again, rest assured (or not now :-)), but notwithstanding the at some point rather personal attacks in that discussion, I think we still should look at what we can learn from it.<br>
And what I learned is that we can do a <em>better</em> job at documenting <em>backward incompatible changes</em>.</p>
<ul dir="auto">
<li>We have no clear overview of these changes (and I am really speaking about 'changes for which users will have to change their code to keep it running/working correctly')</li>
<li>In the whatsnew docs, there is the "API changes" section, which you could assume is about such kind of changes. In practice however, we made it a mix of enhancements (which are of course also api changes in se, as they 'change (add to) the api', but that fact is not really relevant) and breaking api changes.
<ul dir="auto">
<li>eg the current api changes section: <a href="http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#api-changes" rel="nofollow">http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#api-changes</a>. Second point (addition of <code class="notranslate">level</code> keyword) is an enhancement, not a breaking api change, as with a lot of the other bullet points. For this reason, the 'real' breaking api changes disappear a bit between all the others.</li>
</ul>
</li>
<li>deprecations are in a seperate section,  but are also a kind of api changes</li>
</ul>
<hr>
<p dir="auto"><strong>What I propose</strong>:</p>
<ol dir="auto">
<li>a better stuctured whatsnew, with a more strict structure</li>
<li>limit the 'API changes' section to backwards incompatible changes, the rest are new features/enhancements (keep a clear distinction)</li>
<li>for each backwards incompatible change, very clearly explain: what was the previous behaviour? what is the new behaviour? How can I keep the old behaviour? What should I do to update my code to keep it working?</li>
</ol>
<p dir="auto">Some examples of other libraries: Scipy speaks about "<em>Backwards incompatible changes</em>" (<a href="http://docs.scipy.org/doc/scipy-0.14.0/reference/release.0.14.0.html" rel="nofollow">http://docs.scipy.org/doc/scipy-0.14.0/reference/release.0.14.0.html</a>), scikit-learn about "<em>API changes summary</em>" (<a href="http://scikit-learn.org/stable/whats_new.html" rel="nofollow">http://scikit-learn.org/stable/whats_new.html</a>), sqlalchemy about "<em>Behavioral changes</em>" (<a href="http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_09.html#behavioral-changes-orm-09" rel="nofollow">http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_09.html#behavioral-changes-orm-09</a>) and matlab about "<em>Compatibility considerations</em>" (<a href="http://www.mathworks.nl/help/matlab/release-notes.html" rel="nofollow">http://www.mathworks.nl/help/matlab/release-notes.html</a>).</p>
<p dir="auto">A possible structure (in the past, in pandas there was also a more rigid structure with new features/api changes):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="1. New features
     1.1 ... (subsections highlighting bigger new features)
     1.2 Categoricals
     1.3 Timedelta(Index)
     1.4 .dt accessor
     1.5 ....
2. Backwards incompatible API changes
     2.1 Breaking changes
     2.2 Deprecations
     2.3 Removal previously deprecated (prior version deprecations)
3. Other enhancements
4. Performance improvements
5. Bug fixes"><pre class="notranslate"><code class="notranslate">1. New features
     1.1 ... (subsections highlighting bigger new features)
     1.2 Categoricals
     1.3 Timedelta(Index)
     1.4 .dt accessor
     1.5 ....
2. Backwards incompatible API changes
     2.1 Breaking changes
     2.2 Deprecations
     2.3 Removal previously deprecated (prior version deprecations)
3. Other enhancements
4. Performance improvements
5. Bug fixes
</code></pre></div>
<p dir="auto">For comparison, the current table of contents for 0.15 whatsnew</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="API changes
Memory Usage
.dt accessor
Timezone API changes
Rolling/Expanding Moments API changes
Internal Refactoring
Categoricals in Series/DataFrame
TimedeltaIndex/Scalar
Prior Version Deprecations/Changes
Deprecations
Enhancements
Performance
Bug Fixes"><pre class="notranslate"><code class="notranslate">API changes
Memory Usage
.dt accessor
Timezone API changes
Rolling/Expanding Moments API changes
Internal Refactoring
Categoricals in Series/DataFrame
TimedeltaIndex/Scalar
Prior Version Deprecations/Changes
Deprecations
Enhancements
Performance
Bug Fixes
</code></pre></div>
<p dir="auto">Some points of dicussion:</p>
<ul dir="auto">
<li>'api changes' section before new features instead of after?</li>
<li>'other enhancements' as subsection in 'new features'</li>
<li>how to name the 'backwards incompatible API changes' (other options like behavioral changes, compatibility considerations, breaking API changes, ...)</li>
<li>and of course the general idea I describe in this issue</li>
</ul>
<p dir="auto">Thoughts?</p>
<p dir="auto">When you are working with a large array, it is not printed out in its entirety in the console, but when you have grouped them with <code class="notranslate">groupby('key')</code>, the groups are all printed out. Would it be possible to also restrict the output of <code class="notranslate">groupby('key').groups</code> to eg the first and last groups?</p>
<p dir="auto">I was working with a rather large dataframe (around 80000 rows), and first it took a long time to print it all, and second the console got stuck for a while (but that could also be an issue with spyder).<br>
I know it is not very useful to print out the groups, but I was curious to see what it would look like, but it was not a very good idea with such a large array.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; import pandas as pd
&gt;&gt;&gt; df = pd.DataFrame(np.random.randn(100000, 4), columns=list('abcd'))
&gt;&gt;&gt; df['g'] = np.random.randint(0, 100, 100000)
&gt;&gt;&gt; df.groupby('g').groups  # this repr should be truncated"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randn</span>(<span class="pl-c1">100000</span>, <span class="pl-c1">4</span>), <span class="pl-s1">columns</span><span class="pl-c1">=</span><span class="pl-en">list</span>(<span class="pl-s">'abcd'</span>))
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df</span>[<span class="pl-s">'g'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">0</span>, <span class="pl-c1">100</span>, <span class="pl-c1">100000</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df</span>.<span class="pl-en">groupby</span>(<span class="pl-s">'g'</span>).<span class="pl-s1">groups</span>  <span class="pl-c"># this repr should be truncated</span></pre></div>