<p dir="auto">error:<br>
Traceback (most recent call last):<br>
File "D:\Python Program\Pworkspace\test.py", line 34, in <br>
pd.read_csv(path)<br>
File "C:\Users\wuqing\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f<br>
return _read(filepath_or_buffer, kwds)<br>
File "C:\Users\wuqing\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 405, in _read<br>
parser = TextFileReader(filepath_or_buffer, **kwds)<br>
File "C:\Users\wuqing\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 764, in <strong>init</strong><br>
self._make_engine(self.engine)<br>
File "C:\Users\wuqing\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 985, in _make_engine<br>
self._engine = CParserWrapper(self.f, **self.options)<br>
File "C:\Users\wuqing\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 1605, in <strong>init</strong><br>
self._reader = parsers.TextReader(src, **kwds)<br>
File "pandas_libs\parsers.pyx", line 394, in pandas._libs.parsers.TextReader.<strong>cinit</strong> (pandas_libs\parsers.c:4209)<br>
File "pandas_libs\parsers.pyx", line 712, in pandas._libs.parsers.TextReader._setup_parser_source (pandas_libs\parsers.c:8895)<br>
OSError: Initializing from file failed</p>
<p dir="auto">code:<br>
temp = pd.read_csv('测试.csv')</p>
<p dir="auto">the following code works:<br>
temp = pd.read_csv('tmp.csv')</p>
<p dir="auto">the following also works:<br>
temp = pd.read_csv('测试.csv', engine='python')</p>
<p dir="auto">如果文件名中有中文，并且不加engine时用的是engine='c'，这时则会报出如下错误：OSError: Initializing from file failed</p>
<p dir="auto">我看以前有类似问题出现，但是新版本仍然有这个bug，希望能帮忙解决</p>
<p dir="auto">This is to discuss pushing the <code class="notranslate">Categorical.categories</code> and<br>
<code class="notranslate">Categorical.ordered</code> information into the extension type <code class="notranslate">CategoricalDtype</code>.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pd.CategoricalDtype(categories, ordered=False)"><pre class="notranslate"><span class="pl-s1">pd</span>.<span class="pl-v">CategoricalDtype</span>(<span class="pl-s1">categories</span>, <span class="pl-s1">ordered</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)</pre></div>
<p dir="auto">Note that there is no <code class="notranslate">values</code> argument. This is a type constructor, that<br>
isn't attached to any specific <code class="notranslate">Categorical</code> instance.</p>
<h2 dir="auto">Why?</h2>
<p dir="auto">Several times now (<code class="notranslate">read_csv(..., dtype=...)</code>, <code class="notranslate">.astype(...)</code>, <code class="notranslate">Series([], dtype=...)</code>)<br>
we have places where we accept <code class="notranslate">dtype='category'</code> which takes the values<br>
in the method (the series, or column from the CSV, etc.)<br>
and hands it off to the <em>value</em> constructor, with no control over the<br>
<code class="notranslate">categories</code> and <code class="notranslate">ordered</code> arguments.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Categorical(values, categories=None, ordered=False)"><pre class="notranslate"><span class="pl-v">Categorical</span>(<span class="pl-s1">values</span>, <span class="pl-s1">categories</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">ordered</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)</pre></div>
<p dir="auto">The proposal here would add the <code class="notranslate">categories</code> and <code class="notranslate">ordered</code><br>
attributes / arguments to <code class="notranslate">CategoricalDtype</code> and provide a common API<br>
for specifying non-default parameters for the <code class="notranslate">Categorical</code> constructor<br>
in methods like <code class="notranslate">read_csv</code>, <code class="notranslate">astype</code>, etc.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="t = pd.CategoricalDtype(['low', 'med', 'high'], ordered=True)
pd.read_csv('foo.csv', dtype={'A': int, 'B': t)
pd.Series(['high', 'low', 'high'], dtype=t)

s = pd.Series(['high', 'low', 'high'])
s.astype(t)"><pre class="notranslate"><span class="pl-s1">t</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">CategoricalDtype</span>([<span class="pl-s">'low'</span>, <span class="pl-s">'med'</span>, <span class="pl-s">'high'</span>], <span class="pl-s1">ordered</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-s">'foo.csv'</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span>{<span class="pl-s">'A'</span>: <span class="pl-s1">int</span>, <span class="pl-s">'B'</span>: <span class="pl-s1">t</span>)
<span class="pl-s1">pd</span>.<span class="pl-v">Series</span>([<span class="pl-s">'high'</span>, <span class="pl-s">'low'</span>, <span class="pl-s">'high'</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">t</span>)

<span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>([<span class="pl-s">'high'</span>, <span class="pl-s">'low'</span>, <span class="pl-s">'high'</span>])
<span class="pl-s1">s</span>.<span class="pl-en">astype</span>(<span class="pl-s1">t</span>)</pre></div>
<p dir="auto">We would continue to accept <code class="notranslate">dtype='category'</code>.</p>
<p dir="auto">This becomes even more import when doing operations on larger than memory datasets with something like <code class="notranslate">dask</code> or even (<code class="notranslate">read_csv(..., chunksize=N)</code>). Right now you don't have an easy way to specify the <code class="notranslate">categories</code> or <code class="notranslate">ordered</code> for columns (assuming you know them ahead of time).</p>
<h2 dir="auto">Issues</h2>
<ol dir="auto">
<li><code class="notranslate">CategoricalDtype</code> currently isn't part of the public API. Which methods<br>
on it do we make public?</li>
<li>Equality semantics: For backwards compat, I think all instances<br>
of <code class="notranslate">CategoricalDtype</code> should compare equal with all others. You can use<br>
identity to check if two types are the same</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="t1 = pd.CategoricalDtype(['a', 'b'], ordered=True)
t2 = pd.CategoricalDtype(['a', 'b'], ordered=False)

t1 == t2  # True
t1 is t2  # False
t1 is t1  # True"><pre class="notranslate"><span class="pl-s1">t1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">CategoricalDtype</span>([<span class="pl-s">'a'</span>, <span class="pl-s">'b'</span>], <span class="pl-s1">ordered</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">t2</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">CategoricalDtype</span>([<span class="pl-s">'a'</span>, <span class="pl-s">'b'</span>], <span class="pl-s1">ordered</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)

<span class="pl-s1">t1</span> <span class="pl-c1">==</span> <span class="pl-s1">t2</span>  <span class="pl-c"># True</span>
<span class="pl-s1">t1</span> <span class="pl-c1">is</span> <span class="pl-s1">t2</span>  <span class="pl-c"># False</span>
<span class="pl-s1">t1</span> <span class="pl-c1">is</span> <span class="pl-s1">t1</span>  <span class="pl-c"># True</span></pre></div>
<ol start="3" dir="auto">
<li>Should the <code class="notranslate">categories</code> argument be required? Currently <code class="notranslate">dtype='category'</code><br>
says 1.) infer the categories based on the <em>values</em>, and 2.) it's unordered.<br>
Would <code class="notranslate">CategoricalDtype(None, ordered=False)</code> be allowed?</li>
<li>Strictness? If I say</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pd.Series(['a', 'b', 'c'], dtype=pd.CategoricalDtype(['a', 'b']))"><pre class="notranslate"><span class="pl-s1">pd</span>.<span class="pl-v">Series</span>([<span class="pl-s">'a'</span>, <span class="pl-s">'b'</span>, <span class="pl-s">'c'</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-v">CategoricalDtype</span>([<span class="pl-s">'a'</span>, <span class="pl-s">'b'</span>]))</pre></div>
<p dir="auto">What happens? I would probably expect a <code class="notranslate">TypeError</code> or <code class="notranslate">ValueError</code> as <code class="notranslate">c</code><br>
isn't "supposed" to be there. Or do we replace <code class="notranslate">'c'</code> with <code class="notranslate">NA</code>? Should<br>
<code class="notranslate">strict</code> be another parameter to <code class="notranslate">CategoricalDtype</code> (I don't think so).</p>
<p dir="auto">I'm willing to work on this over the next couple weeks.</p>
<p dir="auto">xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="190053261" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/14676" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/14676/hovercard" href="https://github.com/pandas-dev/pandas/issues/14676">#14676</a> (astype)<br>
xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="185449278" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/14503" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/14503/hovercard" href="https://github.com/pandas-dev/pandas/issues/14503">#14503</a> (read_csv)</p>