juerd waalboer bxaqoconvolutionnl 510 imminent found round tuit update documentation suggests reading files utf8 unsafe discussion bigger unicode documentation update mind maybe time 512 tuits sufficiently round patch changed utf8 examples reading files encodingutf8 added warnings unsafety utf8 changed semantics use open locale uses encodingutf8 instead utf8 left utf8 use writing output alone discussion please added perlunifaqs including one unicode semantics nonsvutf8 strings rephrased perlunifaqs rearranged perlunifaq internals related things grouped end document updated email address way 510 rc1 tests perfectly machine tests successful u229 s092 cu26383 cs2441 scripts1380 tests184452 heres patch diff ru perl5100rc1authors perl5100rc1updatedauthors perl5100rc1authors 20071117 162625000000000 0100 perl5100rc1updatedauthors 20071117 190644000000000 0100 4767 4767 joost van baal jt mcduffie juan gallego juerd waalboer juerd waalboer juerdnl juha laiho julian yip juna diff ru perl5100rc1libopenpm perl5100rc1updatedlibopenpm perl5100rc1libopenpm 20071117 162623000000000 0100 perl5100rc1updatedlibopenpm 20071117 191923000000000 0100 7911 797 unless defined locale_encoding warningswarniflayer figure encoding use last unless defined locale_encoding locale_encoding utf8i layer utf8 else layer encodinglocale_encoding layer encodinglocale_encoding std 1 else target layer layer name 1517 1477 use open io locale use open utf8 use open encodingutf8 use open locale use open encodingiso88597 1938 1898 equivalent use open utf8 use open io utf8 use open encodingutf8 use open io encodingutf8 2109 2066 many encodings several aliases see l details list supported locales note cutf8 perlio layer must always specified exactly like subject loose matching encoding names open given explicit list layers threearg syntax override list declared using pragma 22010 21310 cutf8 cencoding subpragmas converts standard filehandles stdin stdout stderr comply encoding selected inputoutput handles example input chosen cutf8 cstd mean stdin stdout stderr also cutf8 hand output chosen c encodingkoi8r cstd cause stdout stderr c clocale subpragma chosen cencodingutf8 cstd mean stdin stdout stderr also cencodingutf8 hand output chosen c encodingkoi8r cstd cause stdout stderr c clocale subpragma implicitly turns cstd logic clocale described full l diff ru perl5100rc1libperliopm perl5100rc1updatedlibperliopm perl5100rc1libperliopm 20071117 162623000000000 0100 perl5100rc1updatedlibperliopm 20071117 192153000000000 0100 1396 13910 closef note layer validate byte sequences reading input using cencodingutf8 instead bare cutf8 strongly recommended item bytes inverse cutf8 layer turns flag diff ru perl5100rc1podperlcheatpod perl5100rc1updatedpodperlcheatpod perl5100rc1podperlcheatpod 20071117 162626000000000 0100 perl5100rc1updatedpodperlcheatpod 20071117 192557000000000 0100 847 847 head1 author juerd waalboer help many perl monks juerd waalboer juerdnl help many perl monks head1 see also diff ru perl5100rc1podperlfuncpod perl5100rc1updatedpodperlfuncpod perl5100rc1podperlfuncpod 20071117 162625000000000 0100 perl5100rc1updatedpodperlfuncpod 20071117 191535000000000 0100 434510 434510 note depending status socket either 8bit bytes characters received default sockets operate bytes example socket changed using binmode operate cutf8 io layer see c pragma l io operate utf8 encoded unicode characters bytes similarly cencoding pragma case pretty much characters read binmode operate cencodingutf8 io layer see c pragma l io operate utf8 encoded unicode characters bytes similarly cencoding pragma case pretty much characters read item redo label x 47847 47847 otherwise note even filehandle set operate characters example using cutf8 open operate characters example using cencodingutf8 open layer tell return byte offsets character offsets implementing would render seek tell rather slow 497410 497410 note depending status socket either 8bit bytes characters sent default sockets operate bytes example socket changed using binmode operate cutf8 io layer see l c pragma l io operate utf8 encoded unicode characters bytes similarly cencoding pragma case pretty much characters sent binmode operate cencodingutf8 io layer see l c pragma l io operate utf8 encoded unicode characters bytes similarly cencoding pragma case pretty much characters sent item setpgrp pidpgrp x x 63289 63289 negative note even filehandle set operate characters example using cutf8 io layer tell return byte offsets character offsets implementing would render sysseek slow characters example using cencodingutf8 io layer tell return byte offsets character offsets implementing would render sysseek slow sysseek bypasses normal buffered io mixing reads c example c read c c 64559 64559 last read note even filehandle set operate characters example using cutf8 open layer tell return byte offsets character offsets would render seek tell rather slow operate characters example using cencodingutf8 open layer tell return byte offsets character offsets would render seek tell rather slow return value tell standard streams like stdin depends operating system may return 1 something else diff ru perl5100rc1podperlopentutpod perl5100rc1updatedpodperlopentutpod perl5100rc1podperlopentutpod 20071117 162626000000000 0100 perl5100rc1updatedpodperlopentutpod 20071117 190919000000000 0100 9177 9177 c c c c variants example openmy fh utf8 fn openmy fh crlf fn item diff ru perl5100rc1podperlrunpod perl5100rc1updatedpodperlrunpod perl5100rc1podperlrunpod 20071117 162626000000000 0100 perl5100rc1updatedpodperlrunpod 20071117 191120000000000 0100 11249 11249 pseudolayer turns flag layer tell perl output utf8 input regarded already utf8 form may useful perlio environment variable make utf8 default turn behaviour use cbytes layer already valid utf8 form check validity handled caution input generally cencodingutf8 best option reading utf8 encoded data item win32 xwin32 diff ru perl5100rc1podperlunicodepod perl5100rc1updatedpodperlunicodepod perl5100rc1podperlunicodepod 20071117 162625000000000 0100 perl5100rc1updatedpodperlunicodepod 20071117 191313000000000 0100 15237 15237 filehandle read write utf8 5007 binmode fh utf8 binmode fh encodingutf8 item diff ru perl5100rc1podperlunifaqpod perl5100rc1updatedpodperlunifaqpod perl5100rc1podperlunifaqpod 20071117 162626000000000 0100 perl5100rc1updatedpodperlunifaqpod 20071117 195009000000000 0100 27 27 perlunifaq perl unicode faq head1 description head1 q list questions answers unicode perl intended read l 166 1621 think unicode special magical didnt want disappoint decided call document unicode tutorial head2 character encodings perl support find character encodings perl supports run perl mencode le print encodeencodingsall head2 version perl use well upgrade recent certainly c581 newer tutorial faq based status quo c588 also check modules upgrade necessary example htmlentities requires version 132 function correctly even though changelog silent head2 binary data like images well apart bare c shouldnt treat specially 2720 429 appropriate encoding join binary strings see also dont encode head2 utf8 flag please unless youre hacking internals debugging weirdness dont think utf8 flag means probably shouldnt use c c_utf8_on c_utf8_off perls internal format happens utf8 unfortunately perl cant keep secret everyone knows source much confusion better pretend internal format unknown encoding always encode decode explicitly head2 decode encode whenever youre communicating anything external perl whenever youre communicating text anything external perl process like database text file socket another program even thing youre communicating also written perl 8823 927 binmode fh encodingutf8 database drivers dbi also automatically encode decode typically limited utf8 encoding cheat head2 cheat tell cheat well perls internal format utf8 skip encoding decoding step manipulate utf8 flag directly instead cencodingutf8 simply use cutf8 widely accepted good behavior youre writing dangerous reading causes internal inconsistency invalid byte sequences instead c c could use c_utf8_on c_utf8_off considered bad style especially c_utf8_on dangerous reason cutf8 shortcuts oneliners see cc l sometimes limited utf8 encoding head2 dont know encoding used 1466 13425 properly encode strings output none concern c dumped data always head2 regex character classes sometimes match ascii range head2 characters uppercase lowercase correctly seemed like good idea time keep semantics standard strings perl got unicode support might repaired future deal fact perl treats equal strings differently depending internal state affected c c c c cu cl cu cl cd cs cw cd cs cw c ci c force unicode semantics upgrade internal representation c change strings already upgraded detailed discussion see l cpan head2 determine string text string binary string cant use utf8 flag thats misuse makes well 1766 18345 open barfh encodingbar examplebartxt print barfh _ foofh head2 c c alternate syntaxes c c head2 wide character term used characters ordinal value greater 127 characters ordinal value greater 255 character occupying one byte depending context perl warning wide character caused character ordinal value greater 255 specified encoding layer perl tries fit things iso88591 backward compatibility reasons cant emits warning warnings enabled outputs utf8 encoded data instead avoid warning avoid different output encodings single stream always specify encoding explicitly example perlio layer binmode stdout encodingutf8 head1 internals head2 utf8 flag please unless youre hacking internals debugging weirdness dont think utf8 flag means probably shouldnt use c c_utf8_on c_utf8_off utf8 flag also called svutf8 internal flag indicates current internal representation utf8 without flag assumed iso88591 perl converts automatically one perls internal formats happens utf8 unfortunately perl cant keep secret everyone knows source much confusion better pretend internal format unknown encoding always encode decode explicitly head2 c pragma dont use makes sense deal bytes text string 18610 23236 c usually failed attempt something useful forget head2 c c head2 c pragma alternate syntaxes c c dont use unfortunately assumes programmers environment user use encoding use encoding source code stdin stdout program copied another machine source code change stdio environment might need nonascii characters source code make utf8 encoded file c need set encoding stdin stdout stderr example based users locale c head2 difference cencoding cutf8 utf8 one perls internal formats often skip encoding decoding step manipulate utf8 flag directly instead cencodingutf8 simply use cutf8 skips encoding step data already represented utf8 internally widely accepted good behavior youre writing dangerous reading causes internal inconsistency invalid byte sequences using cutf8 input sometimes result security breaches please use cencodingutf8 instead instead c c could use c_utf8_on c_utf8_off considered bad style especially c_utf8_on dangerous reason cutf8 shortcuts oneliners see cc l head2 whats difference c c 22324 2959 encoding certain string instead encode encoding want head2 character encodings perl support find character encodings perl supports run perl mencode le print encodeencodingsall head2 version perl use well upgrade recent certainly c581 newer tutorial faq based status quo c588 also check modules upgrade necessary example htmlentities requires version 132 function correctly even though changelog silent head1 author juerd waalboer juerd waalboer juerdnl head1 see also diff ru perl5100rc1podperluniintropod perl5100rc1updatedpodperluniintropod perl5100rc1podperluniintropod 20071117 162626000000000 0100 perl5100rc1updatedpodperluniintropod 20071117 184002000000000 0100 1677 1677 wide character print output utf8 use cutf8 output layer prepending output utf8 use cencoding cutf8 output layer prepending binmodestdout utf8 3177 3179 matching encoding names loose case matter many encodings several aliases note cutf8 layer must always specified exactly like subject loose matching encoding names loose matching encoding names also note cutf8 unsafe input accepts data without validating indeed valid utf8 see l cutf8 layer l l cencoding layer 3297 3317 unicode perls eyes specify appropriate layer opening files openmy fhutf8 anything openmy fhencodingutf8 anything line_of_unicode fh openmy fhencodingbig5 anything 3387 3407 io layers also specified flexibly c pragma see l look following example use open utf8 input output default layer utf8 use open encodingutf8 inputoutput default encoding utf8 open x file print x chr0x100 n close x 35811 3606 printf xn ord n print 0xc1 close also use cencoding layer openmy epicencodingiso88597iliadgreek line_of_unicode epic methods install transparent filter io stream converts data specified encoding read stream result always unicode 41113 40813 local read whole file 8bit characters close f open f utf8 file open f encodingutf8 file print f convert utf8 output close f run code twice contents f twice utf8 encoded c would avoided bug explicitly opening also f input utf8 utf8 encoded c would avoided bug explicitly opening also f input utf8 b cutf8 cencoding features work perl built new perlio feature default diff ru perl5100rc1podperlunitutpod perl5100rc1updatedpodperlunitutpod perl5100rc1podperlunitutpod 20071117 162625000000000 0100 perl5100rc1updatedpodperlunitutpod 20071117 190634000000000 0100 2017 2017 head1 author juerd waalboer juerd waalboer juerdnl head1 see also met vriendelijke groet kind regards korajn salutojn juerd waalboer perl hacker juerdnl convolution ict solutions consultancy fri 08 aug 2008 120602 0100 patch utf8 updates