Answers.Body: <p>I think this is going to be one of those impossible to answer questions because of so many unknown variables.</p>
  
  <p>MySQL could be screwing it up because it depends on 1) MySQL server configuration and 2) The charset and collation of the database, the tables, the connection strings, etc.</p>
  
  <p>The file system itself could be screwing it up because they all tend to have different treatments for character encodings (HFS, NTFS, FAT, EXT, etc.).</p>
  
  <p>Throw in the fact that you said this has been on various developer machines and touched by an unknown number of other programs, it makes it very difficult to diagnose.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-04-16T18:57:13.417Z
Answers.Text: Given this, and I've requested this several times both for the EE and Craft versions of Craft, can we please, please, please have a setting to convert  filenames to ascii, like slugs? :)
Answers.UserDisplayName: André Elvan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-04-16T19:02:10.723Z
Answers.Text: @AndréElvan Already on our internal list. :)
Answers.UserDisplayName: Brad Bell
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-04-17T08:36:24.450Z
Answers.Text: Brad: I totally understand :) I just wanted to collect experiences from other people!
Answers.UserDisplayName: gromgull
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-04-17T08:38:15.253Z
Answers.Text: I also take this to mean that "Craft does not intentionally apply unicode normalisation to filenames (or to anything)" - so it happened "somewhere else".
Answers.UserDisplayName: gromgull
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-04-17T15:57:16.063Z
Answers.Text: @gromgull Correct... Craft isn't doing any filename normalization currently.
Answers.UserDisplayName: Brad Bell
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-04-16T17:44:01.687Z
Answers.LastActivityDate: 2015-04-16T17:44:01.687Z
Answers.UserDisplayName: Brad Bell
Answers.Body: <p>This has now been solved by the addition of the <a href="https://craftcms.com/docs/config-settings#convertFilenamesToAscii" rel="nofollow"><code>convertFilenamesToAscii</code></a> config setting introduced in Craft 2.4.</p>
  
  <p>If you are working on a multilingual site, you might also find the <a href="https://craftcms.com/docs/config-settings#limitAutoSlugsToAscii" rel="nofollow"><code>limitAutoSlugsToAscii</code></a> config setting useful.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-06-08T19:22:37.643Z
Answers.LastActivityDate: 2016-06-08T19:22:37.643Z
Answers.UserDisplayName: Steve Rowling
Body: <p>We have a site deployed in Norway, and people have uploaded PDF-assets with Norwegian non-ascii characters in the filenames. </p>
  
  <p>Normally this works fine, the URLs are encoded like this and it works: </p>
  
  <p><a href="http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf" rel="nofollow">http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf</a></p>
  
  <p>("Årsberetning" - annual report)</p>
  
  <p>But one file didn't want to download, it only gave me 404. However, checking on the server the file was there, and the filename appeared correct. Also, re-scanning asset folders didn't remove it. </p>
  
  <p>BUT - I was not able to rename the file from within craft, nor download it. </p>
  
  <p>After much hair-pulling I established that the file on disk had the Å encoded as '\x41\xcc\x8a' whereas craft looked for a "pre-composed" Å ('\xc3\x85'). </p>
  
  <p>NOW the question - where did this break? :) </p>
  
  <p>Does Craft unicode-normalise filenames? 
  Does MySQL normalise the unicode it stores? </p>
  
  <p>The file has moved around various developer machines and staging servers before making it into production, it could be some other part of the tool-chain that broke it. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-04-16T18:55:51.750Z
Comments.Text: I'm having problems with this all the time with norwegian characters. For me, the problem normally happends when moving files between mac osx and various linux servers. As long as the files is uploaded to craft to a server, and stay on that server, it works fine.
Comments.UserDisplayName: André Elvan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-04-17T08:38:55.843Z
Comments.Text: André - yeah, that is my experience as well. On our linux staging server the file was uploaded, there it worked fine (but the file was stored twice, with both encodings...) - then it went through git onto a osx machine, and from there it was pushed live with rsync. It's actually a miracle that anything ever works at all :)
Comments.UserDisplayName: gromgull
Tags: <assets><filesystem>
Title: Asset filename unicode normalisation?
ThreadText: # Asset filename unicode normalisation?
  
  (Asked by: username_0 on 2015-04-16)
  
  We have a site deployed in Norway, and people have uploaded PDF-assets with Norwegian non-ascii characters in the filenames.
  
  Normally this works fine, the URLs are encoded like this and it works:
  
  <http://[oursite].no/assets/files/downloads/%C3%85rsberetning-2014_kladd.pdf>
  
  ("Årsberetning" - annual report)
  
  But one file didn't want to download, it only gave me 404. However, checking on the server the file was there, and the filename appeared correct. Also, re-scanning asset folders didn't remove it.
  
  BUT - I was not able to rename the file from within craft, nor download it.
  
  After much hair-pulling I established that the file on disk had the Å encoded as '\x41\xcc\x8a' whereas craft looked for a "pre-composed" Å ('\xc3\x85').
  
  NOW the question - where did this break? :)
  
  Does Craft unicode-normalise filenames?
  Does MySQL normalise the unicode it stores?
  
  The file has moved around various developer machines and staging servers before making it into production, it could be some other part of the tool-chain that broke it.
  
  **Tags:** assets, filesystem
  
  > **Comments:**
  > **username_1**: I'm having problems with this all the time with norwegian characters. For me, the problem normally happends when moving files between mac osx and various linux servers. As long as the files is uploaded to craft to a server, and stay on that server, it works fine.  
  > **username_0**: username_1 - yeah, that is my experience as well. On our linux staging server the file was uploaded, there it worked fine (but the file was stored twice, with both encodings...) - then it went through git onto a osx machine, and from there it was pushed live with rsync. It's actually a miracle that anything ever works at all :)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2015-04-16. Score: 3 | Accepted answer
  
  I think this is going to be one of those impossible to answer questions because of so many unknown variables.
  
  MySQL could be screwing it up because it depends on 1) MySQL server configuration and 2) The charset and collation of the database, the tables, the connection strings, etc.
  
  The file system itself could be screwing it up because they all tend to have different treatments for character encodings (HFS, NTFS, FAT, EXT, etc.).
  
  Throw in the fact that you said this has been on various developer machines and touched by an unknown number of other programs, it makes it very difficult to diagnose.
  
  > **Comments:**
  > **username_1**: Given this, and I've requested this several times both for the EE and Craft versions of Craft, can we please, please, please have a setting to convert  filenames to ascii, like slugs? :)  
  > **username_2**: @username_1 Already on our internal list. :)  
  > **username_0**: username_2: I totally understand :) I just wanted to collect experiences from other people!  
  > **username_0**: I also take this to mean that "Craft does not intentionally apply unicode normalisation to filenames (or to anything)" - so it happened "somewhere else".  
  > **username_2**: @username_0 Correct... Craft isn't doing any filename normalization currently.  
  
  ---
  
  ### Answer by username_3 on 2016-06-08. Score: 4
  
  This has now been solved by the addition of the [`convertFilenamesToAscii`](https://craftcms.com/docs/config-settings#convertFilenamesToAscii) config setting introduced in Craft 2.4.
  
  If you are working on a multilingual site, you might also find the [`limitAutoSlugsToAscii`](https://craftcms.com/docs/config-settings#limitAutoSlugsToAscii) config setting useful.
  
  
  ---
  
  