title: PHP cp1252/windows-1252 conversion to UTF-8
body: <p>I'm in the process of trying to convert our database from latin1 to UTF-8. Unfortunately I can't do a massive single switchover as the application needs to stay online and we have 700GB of database to convert.</p>
  
  <p>So I'm trying to leverage a little mysql hack of converting tables to UTF-8 however not the data. I'd like the data to be read, converted, and replaced in real time. (A JIT conversion if you will)</p>
  
  <p>Our php app currently uses all of the defaults so it's connecting to mysql using the latin1 character set and it drops UTF-8 data encoded in latin1. When you view the data with latin1 the UTF-8 characters show up as expected. When you view the data with UTF-8 things get jumbled up.</p>
  
  <p>So I propose forcing the mysql character set to UTF-8 and then doing a just in time conversion of the data if necessary. Now, seeing as cp1252/windows-1252 is a subset of UTF-8 it's not so straight forward (as far as I can see) to detect the cp1252/windows-1252 encoding.</p>
  
  <p>I've written the following code that attempts to detect cp1252/windows-1252 encoding and convert as necessary. It should also detect properly encoded UTF-8 characters and do nothing.</p>
  
  <pre><code>$a = 'Cardâ˜ƒ'; //cp1252 encoded
  $a_test = '☃'.$a; //add known UTF8 character
  $c = mb_convert_encoding($a_test, 'cp1252', 'UTF-8');
  // attempt to detect known utf8 character after conversion
  if (mb_strpos($c, '☃') === false) {
      // not found, original string was not cp1252 encoded, so print
      var_dump($a);
  } else {
      // found, original string was cp1252 encoded, remove test character and print
      // This case runs
      $c = mb_strcut($c, 1);
      var_dump($c);
  }
  
  $a = 'COD☃'; //proper UTF8 encoded
  $a_test = '☃'.$a; //add known UTF8 character
  $c = mb_convert_encoding($a_test, 'cp1252', 'UTF-8');
  // attempt to detect known utf8 character after conversion
  if (mb_strpos($c, '☃') === false) {
      // not found, original string was not cp1252 encoded, so print
      // This case runs
      var_dump($a);
  } else {
      // found, original string was cp1252 encoded, remove test character and print
      $c = mb_strcut($c, 1);
      var_dump($c);
  }
  </code></pre>
  
  <p>The output of running this code is:</p>
  
  <pre><code>string 'Card☃' (length=7)
  string 'COD☃' (length=6)
  </code></pre>
  
  <p>I understand that running this on all strings coming out of the database will have a performance impact, yet to be measured, but if I can do a JIT conversion before switching everything completely it's worth it to me.</p>
  
  <p>Does anyone have any pointers on how to optimize this?</p>