Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.

I used pdf2text from PDFminer to reduce a PDF to text. Unfortunately it contains special characters. Let me show output from my console

>>>a=pdf_to_text("ap.pdf")

heres a sample of it, a little truncated

>>>a[5000:5500]
'f one architect. Decades ...... but to re\xef\xac\x82ect\none set of design ideas, than to have one that contains many\ngood but independent and uncoordinated ideas.\n1 Joshua Bloch, \xe2\x80\x9cHow to Design a Good API and Why It Matters\xe2\x80\x9d, G......=-3733'

I understood that I must encode it

>>>a[5000:5500].encode('utf-8')
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 237: ordinal not in range(128)

I searched around a bit and tried them, notably Replace special characters in python . The input comes from PDFminer, so its tough (AFAIK) to control that. What is the way to make proper plaintext from this output?

What am I doing wrong?

--A quick fix: change PDFminer's codec to ascii- but it's not a lasting solution--

--Abandoned the quick fix for the answer- changing the codec removes information --

--A relavent topic as mentioned by Maxim http://en.wikipedia.org/wiki/Windows-1251 --

share|improve this question

1 Answer 1

up vote 7 down vote accepted

This problem often occurs when non-ASCII text is stored in str objects. What you are trying to do is to encode in utf-8 a string already encoded in some encoding (because it contains characters with codes above 0x7f).

To encode such a string in utf-8 it has to be first decoded. Assuming that the original text encoding is cp1251 (replace it with your actual encoding), something like the following would do the trick:

u = s.decode('cp1251')  # decode from cp1251 byte (str) string to unicode string
s = u.encode('utf-8')   # re-encode unicode string to  utf-8 byte (str) string

Basically, the above snippet does what iconv --from-code=CP1251 --to-code=UTF-8 command does, i.e. it converts the string from one encoding to another.

Some useful links:

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.
{"cc_re__cc_re__cc_by_sa_3_0": ["        <li><a href=\"http://area51.stackexchange.com\" title=\"proposing new sites in the Stack Exchange network\">Area 51</a></li>\r\n                        <li><a href=\"http://careers.stackoverflow.com\">Stack Overflow Careers</a></li>\r\n                    \r\n                </ol>\r\n            </td>\r\n    </tr>\r\n</table>\r\n                </div>\r\n            </div>\r\n\r\n            <div id=\"copyright\">\r\n                site design / logo &#169; 2014 stack exchange inc; user contributions licensed under <a href=\"http://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a> \r\n                with <a href=\"http://blog.stackoverflow.com/2009/06/attribution-required/\" rel=\"license\">attribution required</a>\r\n            </div>\r\n            <div id=\"svnrev\">\r\n                rev 2014.8.27.1798\r\n            </div>\r\n            \r\n        </div>\r\n    </div>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Stack Overflow works best with JavaScript enabled<img src=\"http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif\" alt=\"\" class=\"dno\"></div>\r\n    </nos"], "copyright__copyright__copyright": [") string to unicode string\ns = u.encode('utf-8')   # re-encode unicode string to  utf-8 byte (str) string\n\nBasically, the above snippet does what iconv --from-code=CP1251 --to-code=UTF-8 command does, i.e. it converts the string from one encoding to another.\n\nSome useful links:\n\nshare|improve this answer\n\nYour Answer\n\n\u00a0\ndiscard\n\nBy posting your answer, you agree to the privacy policy and terms of service.\n\nNot the answer you're looking for? Browse other questions tagged or ask your own question."]}