Stack Exchange
How to solve unicode error from exporting to csv file (python)
I'm trying to export text from web-scraping using python. However, the result shows this:
> UnicodeEncodeError Traceback (most recent call last) in () 71
> 'ranking_title': ranking_title, ---> 72 'ranking_category':
> ranking_category}) 73
> 
> ~\Anaconda3\lib\csv.py in writerow(self, rowdict) 154 def
> writerow(self, rowdict): --> 155 return
> self.writer.writerow(self._dict_to_list(rowdict)) 156
> 
> ~\Anaconda3\lib\encodings\cp1252.py in encode(self, input, final) 18
> def encode(self, input, final=False): ---> 19 return
> codecs.charmap_encode(input,self.errors,encoding_table)[0] 20
> 
> UnicodeEncodeError: 'charmap' codec can't encode characters in
> position 299-309: character maps to

What could be the possible mistake I did? I could export into the CSV file with the text I scraped once but when I tried on another day, it's error.
Ps. I've already imported Beautifulsoup and csv at the beginning of the workbook. 

Why are you using codepage 1252 instead of UTF-8?

@PM2Ring I do not know how the codepage 1252 may mess something up. I think it comes from default setting. How could I change into UTF-8?

It looks like your HTML contains Unicode chars that cp1252 doesn't support. The best way to preserve those chars is to use an encoding that is designed to handle Unicode, like UTF-8. On Linux, it's pretty easy to modify the environment to use UTF-8 as the default encoding in a terminal. I believe it's a little trickier on Windows, but there are answers here on Stack Overflow that describe how to do it.

I'll try, thank you so much.

There's one option shown here. There may be better ways, and if you always run your code from an IDE then it should give you a way to choose the encoding. But I don't have much contact with Windows, so I can't offer much help with stuff like this.

That's very kind of you. Thank you again.

Add a try-catch block for checking Unicode:
try:
    to_unicode = unicode
except NameError:
    to_unicode = str

# requests the URL
site = requests.get('specify URL')

# decodes the string using the codec registered for encoding.

data = site.content.decode('utf-8')

# use Beautiful Soup for scraping
Soup = BS(data, 'lxml')

In the end, while writing to a file ensure writing in Unicode :
file.write(to_unicode(data))

I hope this helps.

Thank you, but it's still result in the error. Is is correct that I put "try and except" at the above of the request of url and put "file.write" below the Writerow?
