ruby string encoding
So, I'm trying to do some screen scraping off of [a certain site](http://www.animenewsnetwork.com/) using nokogiri, but the site owners failed to specify the proper encoding of the page in a `<meta>` tag.  The upshot of this is that I'm trying to deal with strings that think they're utf-8, but really aren't.

(If you care, here are the files I was using to test this:

* main file: http://dpaste.de/nif5/
* ann.html: http://dpaste.de/YsLM/
* ann2.html: http://dpaste.de/Lofi/ 
* ann3.html: http://dpaste.de/R21j/ 
* a-p.html: http://dpaste.de/O9dy/ 
* output: http://dpaste.de/WdXc/

)

After doing a lot of searching around ([this SO question](http://stackoverflow.com/questions/1893064/clean-up-strange-encoding-in-ruby) was particularly useful), I found that calling `encode('iso-8859-1', 'utf-8')` on that test string "works", in that I get a proper © symbol.  The issue now is that there are other characters in some other strings I want that really do not work at being converted to latin encoding (`Shōta`, for instance, turns into `Sh�\x8Dta`).

Now, I'm probably going to bother the appropriate webmasters and try and get them to fix their damn encodings, but in the meantime, I'd like to be able to use the bytes that I've got.  I'm fairly certain that there is a way, but I just can't for the life of me figure out what it is.
ruby string encoding
===

So, I'm trying to do some screen scraping off of [a certain site](http://www.animenewsnetwork.com/) using nokogiri, but the site owners failed to specify the proper encoding of the page in a `<meta>` tag.  The upshot of this is that I'm trying to deal with strings that think they're utf-8, but really aren't.

(If you care, here are the files I was using to test this:

* main file: http://dpaste.de/nif5/
* ann.html: http://dpaste.de/YsLM/
* ann2.html: http://dpaste.de/Lofi/ 
* ann3.html: http://dpaste.de/R21j/ 
* a-p.html: http://dpaste.de/O9dy/ 
* output: http://dpaste.de/WdXc/

)

After doing a lot of searching around ([this SO question](http://stackoverflow.com/questions/1893064/clean-up-strange-encoding-in-ruby) was particularly useful), I found that calling `encode('iso-8859-1', 'utf-8')` on that test string "works", in that I get a proper © symbol.  The issue now is that there are other characters in some other strings I want that really do not work at being converted to latin encoding (`Shōta`, for instance, turns into `Sh�\x8Dta`).

Now, I'm probably going to bother the appropriate webmasters and try and get them to fix their damn encodings, but in the meantime, I'd like to be able to use the bytes that I've got.  I'm fairly certain that there is a way, but I just can't for the life of me figure out what it is.