title: UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' when converting series object to unicode in pandas with utf-16
body: <p>I have a utf-16 csv file that I'm trying to load into Pandas.  By default the data comes in as an object datatype.  I plan to do some modeling with the caption column so I'd like to convert the column df['caption'] from an object to a unicode string.  Currently I'm running into the following error 'UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 6: ordinal not in range(128)' when doing 
  df['caption']=df['caption'].astype(unicode).</p>
  
  <p>I tried to solve this by using the encode and decode functions on the individual values in the df['caption'] column but I couldn't get it to work.</p>
  
  <p>I'm very new to pandas and unicode so I was wondering if there's some insight as to what I am doing wrong.</p>
  
  <p>Thanks in advance.</p>
  
  <p>Teresa</p>
  
  <p>Additional information is below:</p>
  
  <p>The traceback is as follows:</p>
  
  <pre><code>UnicodeEncodeError: Traceback (most recent call last)
  &lt;ipython-input-5-aad36f4acf38&gt; in &lt;module&gt;()
      10 print df['caption'].head(10)
      11 
  ---&gt; 12 df['caption']=df['caption'].astype(unicode)
  
  /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/generic.pyc in astype(self, dtype, copy, raise_on_error)
     2016 
     2017         mgr = self._data.astype(
  -&gt; 2018             dtype, copy=copy, raise_on_error=raise_on_error)
     2019         return self._constructor(mgr).__finalize__(self)
     2020 
  
  /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in astype(self, *args, **kwargs)
     2414 
     2415     def astype(self, *args, **kwargs):
  -&gt; 2416         return self.apply('astype', *args, **kwargs)
     2417 
     2418     def convert(self, *args, **kwargs):
  
  /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in apply(self, f, *args, **kwargs)
     2373 
     2374             else:
  -&gt; 2375                 applied = getattr(blk, f)(*args, **kwargs)
     2376 
     2377             if isinstance(applied, list):
  
  /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in astype(self, dtype, copy, raise_on_error, values)
      425     def astype(self, dtype, copy=False, raise_on_error=True, values=None):
      426         return self._astype(dtype, copy=copy, raise_on_error=raise_on_error,
  --&gt; 427                             values=values)
      428 
      429     def _astype(self, dtype, copy=False, raise_on_error=True, values=None,
  
  /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/internals.pyc in _astype(self, dtype, copy, raise_on_error, values, klass)
      442             # force the copy here
      443             if values is None:
  --&gt; 444                 values = com._astype_nansafe(self.values, dtype, copy=True)
      445             newb = make_block(values, self.items, self.ref_items,
      446                               ndim=self.ndim, placement=self._ref_locs,
  
  /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/common.pyc in _astype_nansafe(arr, dtype, copy)
    2222         return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape)
    2223     elif issubclass(dtype.type, compat.string_types):
     -&gt; 2224         return lib.astype_str(arr.ravel()).reshape(arr.shape)
    2225 
    2226     if copy:
  
     /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/lib.so in pandas.lib.astype_str (pandas/lib.c:12944)()
  
     /opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/lib.so in pandas.lib.astype_str (pandas/lib.c:12862)()
  
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 6: ordinal not in range(128)
  </code></pre>
  
  <p>My code is as follows:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  
  df = pd.read_csv('Chevrolet_4-7-2014_cvid_data.csv',encoding='utf-16',header=0,na_values=['N/A',''],names=['channel','link','title','posted','views','likes','dislikes','description','category','statdate','statviews','timewatched','averagetw','subsdriven','shares','caption'])
  print df.head(5)
  print df.dtypes
  
  
  print df['caption'].head(10)
  
  df['caption']=df['caption'].astype(unicode)
  </code></pre>
  
  <p>The data looks like the following:</p>
  
  <pre><code>channel                                        link  \
  0  Chevrolet  http://www.youtube.com/watch?v=dCayKZe6WvI   
  1  Chevrolet  http://www.youtube.com/watch?v=IRXK35dPXbE   
  2  Chevrolet  http://www.youtube.com/watch?v=XXdj4QMw748   
  3  Chevrolet  http://www.youtube.com/watch?v=_ger32ROs94   
  4  Chevrolet  http://www.youtube.com/watch?v=Chfm7Pou49k   
  5  Chevrolet  http://www.youtube.com/watch?v=ySmEJyQ94BI   
  
                                             title       posted   views  \
  0  Chevy Open House Event: From Our House to Your...  Apr  1 2014   73111   
  1  Truck Towing Capabilities: 2014 Silverado -- #...  Mar 26 2014   11934   
  2  Potholes at the Milford Proving Grounds: Tips ...  Mar 20 2014    8037   
  3  Diesel Trucks: Heavy Duty Strengths -- 2015 Si...  Mar 20 2014   12096   
  4  Captain America: All in a Day's Work -- 2014 T...  Mar 14 2014   93377   
  5  Media Blasting: Camaro Engineering -- 2014 Cam...  Mar 13 2014  109931   
  
     likes  dislikes                                        description  \
  0     43        13  In March over 100000 people visited our Chevy ...   
  1    183        56  Farmer Dewayne Kleman and General Motors engin...   
  2     58        10  Chevrolet vehicles are carefully designed to w...   
  3    210         6  Introducing the all-new 2015 Silverado HD. The...   
  4   1095        35  From saving the world to working on math homew...   
  
         category statdate  statviews timewatched averagetw  subsdriven  \
  0  Autos &amp; Vehicles      NaN        NaN         NaN       NaN         NaN   
  1  Autos &amp; Vehicles      NaN        NaN         NaN       NaN         NaN   
  2  Autos &amp; Vehicles      NaN        NaN         NaN       NaN         NaN   
  3  Autos &amp; Vehicles      NaN        NaN         NaN       NaN         NaN   
  4  Autos &amp; Vehicles      NaN        NaN         NaN       NaN         NaN   
  
     shares                                            caption  
  0     NaN   The Chevy Spring Open House Sale the perfect ...  
  1     NaN   0:03 A Man And His Truck And An Engineer / To...  
  2     NaN   0:02 Severe Bump road sign 0:07 Pothole Facil...  
  3     NaN   0:03 And there's no stronger Silverado than t...  
  4     NaN   0:03 Are you doing anything fun Saturday nigh...  
  5     NaN   0:05 Camaro Z/28 logo 0:07 Z/28 Bead Lock 0:0...  
  
  [5 rows x 16 columns]
  channel         object
  link            object
  title           object
  posted          object
  views           object
  likes            int64
  dislikes         int64
  description     object
  category        object
  statdate        object
  statviews      float64
  timewatched     object
  averagetw       object
  subsdriven     float64
  shares         float64
  caption         object
  
  dtype: object
  0     The Chevy Spring Open House Sale the perfect ...
  1     0:03 A Man And His Truck And An Engineer / To...
  2     0:02 Severe Bump road sign 0:07 Pothole Facil...
  3     0:03 And there's no stronger Silverado than t...
  4     0:03 Are you doing anything fun Saturday nigh...
  5     0:05 Camaro Z/28 logo 0:07 Z/28 Bead Lock 0:0...
  
  Name: caption, dtype: object
  </code></pre>