<p dir="auto">I am upgrading Pandas from 0.8.1 to 0.10.1.dev-f7f7e13 . My environment is Window XP with below: Python: 2.7.3   Numpy: 1.6.2    MPL: 1.1.1      Pandas: 0.10.1.dev-f7f7e13.</p>
<p dir="auto">Then OK application on 0.8.1 now meets errors. I trace the root cause to filtering the duplicated index of Series. Detail in : <a href="http://stackoverflow.com/questions/14395678/how-to-drop-extra-copy-of-duplicate-index-of-pandas-series" rel="nofollow">http://stackoverflow.com/questions/14395678/how-to-drop-extra-copy-of-duplicate-index-of-pandas-series</a></p>
<p dir="auto">simply put: below snippet has two issues :</p>
<p dir="auto">import pandas as pd<br>
idx_tp = [('600809', '20061231'), ('600809', '20070331'), ('600809', '20070630'), ('600809', '20070331')]<br>
dt = ['demo','demo','demo','demo']<br>
idx = pd.MultiIndex.from_tuples(idx_tp,names = ['STK_ID','RPT_Date'])<br>
s = pd.Series(dt,index=idx)</p>
<h1 dir="auto">Issue 1: s[s.index.unique()] works well on 0.8.1 but not 0.10.1</h1>
<h1 dir="auto">Issue 2: s.groupby(s.index).first() will crash on my machine</h1>
<h4 dir="auto">Code:</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
&gt;&gt;&gt; pd.__version__
'0.20.1'
&gt;&gt;&gt; import platform
&gt;&gt;&gt; platform.platform()
'Windows-7-6.1.7601-SP1'
&gt;&gt;&gt; import pandas as pd
&gt;&gt;&gt; df = pd.read_csv(r'c:\tmp\中文.csv')
Traceback (most recent call last):
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py&quot;, line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File &quot;&lt;ipython-input-6-0cd6317422e5&gt;&quot;, line 1, in &lt;module&gt;
    df = pd.read_csv(r'c:\tmp\中文.csv')
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 655, in parser_f
    return _read(filepath_or_buffer, kwds)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 405, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 762, in __init__
    self._make_engine(self.engine)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 966, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 1582, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File &quot;pandas\_libs\parsers.pyx&quot;, line 394, in pandas._libs.parsers.TextReader.__cinit__ (pandas\_libs\parsers.c:4209)
  File &quot;pandas\_libs\parsers.pyx&quot;, line 712, in pandas._libs.parsers.TextReader._setup_parser_source (pandas\_libs\parsers.c:8895)
OSError: Initializing from file failed"><pre class="notranslate"><span class="pl-v">Python</span> <span class="pl-c1">3.6</span><span class="pl-c1">.0</span> <span class="pl-c1">|</span><span class="pl-v">Anaconda</span> <span class="pl-c1">4.3</span><span class="pl-c1">.1</span> (<span class="pl-c1">64</span><span class="pl-c1">-</span><span class="pl-s1">bit</span>)<span class="pl-c1">|</span> (<span class="pl-s1">default</span>, <span class="pl-v">Dec</span> <span class="pl-c1">23</span> <span class="pl-c1">2016</span>, <span class="pl-c1">11</span>:<span class="pl-c1">57</span>:<span class="pl-c1">41</span>) [<span class="pl-v">MSC</span> <span class="pl-s1">v</span><span class="pl-c1">.1900</span> <span class="pl-c1">64</span> <span class="pl-en">bit</span> (<span class="pl-v">AMD64</span>)] <span class="pl-s1">on</span> <span class="pl-s1">win32</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">pd</span>.<span class="pl-s1">__version__</span>
<span class="pl-s">'0.20.1'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">platform</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">platform</span>.<span class="pl-en">platform</span>()
<span class="pl-s">'Windows-7-6.1.7601-SP1'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-s">r'c:\tmp\中文.csv'</span>)
<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">2881</span>, <span class="pl-s1">in</span> <span class="pl-s1">run_code</span>
    <span class="pl-en">exec</span>(<span class="pl-s1">code_obj</span>, <span class="pl-s1">self</span>.<span class="pl-s1">user_global_ns</span>, <span class="pl-s1">self</span>.<span class="pl-s1">user_ns</span>)
  <span class="pl-v">File</span> <span class="pl-s">"&lt;ipython-input-6-0cd6317422e5&gt;"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1</span>, <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
    <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-s">r'c:\tmp\中文.csv'</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">655</span>, <span class="pl-s1">in</span> <span class="pl-s1">parser_f</span>
    <span class="pl-k">return</span> <span class="pl-en">_read</span>(<span class="pl-s1">filepath_or_buffer</span>, <span class="pl-s1">kwds</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">405</span>, <span class="pl-s1">in</span> <span class="pl-s1">_read</span>
    <span class="pl-s1">parser</span> <span class="pl-c1">=</span> <span class="pl-v">TextFileReader</span>(<span class="pl-s1">filepath_or_buffer</span>, <span class="pl-c1">**</span><span class="pl-s1">kwds</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">762</span>, <span class="pl-s1">in</span> <span class="pl-s1">__init__</span>
    <span class="pl-s1">self</span>.<span class="pl-en">_make_engine</span>(<span class="pl-s1">self</span>.<span class="pl-s1">engine</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">966</span>, <span class="pl-s1">in</span> <span class="pl-s1">_make_engine</span>
    <span class="pl-s1">self</span>.<span class="pl-s1">_engine</span> <span class="pl-c1">=</span> <span class="pl-v">CParserWrapper</span>(<span class="pl-s1">self</span>.<span class="pl-s1">f</span>, <span class="pl-c1">**</span><span class="pl-s1">self</span>.<span class="pl-s1">options</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1582</span>, <span class="pl-s1">in</span> <span class="pl-s1">__init__</span>
    <span class="pl-s1">self</span>.<span class="pl-s1">_reader</span> <span class="pl-c1">=</span> <span class="pl-s1">parsers</span>.<span class="pl-v">TextReader</span>(<span class="pl-s1">src</span>, <span class="pl-c1">**</span><span class="pl-s1">kwds</span>)
  <span class="pl-v">File</span> <span class="pl-s">"pandas\_libs\parsers.pyx"</span>, <span class="pl-s1">line</span> <span class="pl-c1">394</span>, <span class="pl-s1">in</span> <span class="pl-s1">pandas</span>.<span class="pl-s1">_libs</span>.<span class="pl-s1">parsers</span>.<span class="pl-v">TextReader</span>.<span class="pl-en">__cinit__</span> (<span class="pl-s1">pandas</span>\_<span class="pl-s1">libs</span>\p<span class="pl-s1">arsers</span>.<span class="pl-s1">c</span>:<span class="pl-c1">4209</span>)
  <span class="pl-v">File</span> <span class="pl-s">"pandas\_libs\parsers.pyx"</span>, <span class="pl-s1">line</span> <span class="pl-c1">712</span>, <span class="pl-s1">in</span> <span class="pl-s1">pandas</span>.<span class="pl-s1">_libs</span>.<span class="pl-s1">parsers</span>.<span class="pl-v">TextReader</span>.<span class="pl-en">_setup_parser_source</span> (<span class="pl-s1">pandas</span>\_<span class="pl-s1">libs</span>\p<span class="pl-s1">arsers</span>.<span class="pl-s1">c</span>:<span class="pl-c1">8895</span>)
<span class="pl-v">OSError</span>: <span class="pl-v">Initializing</span> <span class="pl-k">from</span> <span class="pl-s1">file</span> <span class="pl-s1">failed</span></pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">python 3.6 changed sys.getfilesystemencoding() to return "utf-8" instead of "mbcs"<br>
see <a href="https://www.python.org/dev/peps/pep-0529/" rel="nofollow">PEP 529</a>.</p>
<h3 dir="auto">How to fix</h3>
<p dir="auto">Here is the problem: <a href="https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/parsers.pyx#L689">parsers.pyx</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if isinstance(source, basestring):
     if not isinstance(source, bytes):
         source = source.encode(sys.getfilesystemencoding() or 'utf-8')"><pre class="notranslate"><span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">source</span>, <span class="pl-s1">basestring</span>):
     <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">source</span>, <span class="pl-s1">bytes</span>):
         <span class="pl-s1">source</span> <span class="pl-c1">=</span> <span class="pl-s1">source</span>.<span class="pl-en">encode</span>(<span class="pl-s1">sys</span>.<span class="pl-en">getfilesystemencoding</span>() <span class="pl-c1">or</span> <span class="pl-s">'utf-8'</span>)</pre></div>
<p dir="auto">the source parameter is our filename, and will be encoded to 'utf-8', not legacy 'mbcs' in python 3.6<br>
and finally passed to open() in <a href="https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/src/parser/io.c#L26">io.c:new_file_source</a><br>
thus interpreted as a mbcs string, so, the "File not found" exception is not suprised<br>
maybe this should be the responsiblity of cython for python 3.6 to handle these things by using unicode version of windows API,<br>
but for now, we just replace sys.getfilesystemencoding() to "mbcs"</p>