davies alex awycctsptccom heres update previous patch extencodeunicodeunicodexs containing previous changes plus similar speed encode_xs program readswrites lot encodingutf16le filehandles see noticeable speed dan chance getting something similar encode release 510 many thanks alex unicodexsorig 20071010 120735294849100 0100 unicodexs 20071025 160357602427600 0100 1865 1884 define islosurrogatex 0xdc00 x x 0xdfff define invalid_ucs2x issurrogatex 0xffff x define perlio_bufsiz 1024 xxx value comes perlioencode_get_base avoid wasting much space result buffer static void shrink_buffersv result svlenresult 42 svcurresult char buf strlen datalen 1 svcurresult include nul byte strlen buflen perl_strlen_roundupdatalen newxbuf buflen char copysvpvxresult buf datalen char safefreesvpvxresult svpv_setresult buf svlen_setresult buflen static uv enc_unpackpthx_ u8 spu8 estrlen sizeu8 endian enc_unpackpthx_ u8 sp u8 e strlen size u8 endian u8 sp uv v 0 ssize e croakpartial character cchar endian croakpartial character cchar endian switchendian case n v v v 8 v v v 8 case n v v 8 v v 8 break v v 8 v v 8 break case v case v v v 8 endian v break v 16 v 24 break v v 8 endian v break v 16 v 24 break default croakunknown endian cchar endian break croakunknown endian cchar endian break sp return v void enc_packpthx_ sv resultstrlen sizeu8 endianuv value enc_packpthx_ sv result strlen size u8 endian uv value u8 u8 svgrowresultsvcurresultsize1 u8 u8 svpv_nolenresult switchendian case v case v svcurresult svcur_setresultsvcurresultsize size u8value 0xff value 8 break svcurresult svcur_setresultsvcurresultsize size u8value 0xff value 8 break case n case n svcur_setresultsvcurresultsize svcurresult size u8value 0xff value 8 break svcur_setresultsvcurresultsize svcurresult size u8value 0xff value 8 break default croakunknown endian cchar endian break croakunknown endian cchar endian break 94124 113162 iv check code u8 endian u8 svpv_nolenattrendian 6 int size svivattrsize 4 int ucs2 svtrueattrucs2 4 int renewed svtrueattrrenewed 7 sv result newsvpvn0 u8 endian u8 svpv_nolenattrendian 6 int size svivattrsize 4 int ucs2 1 needed event surrogate pairs sv result newsvpvn0 strlen usize size 0 size 1 protect rogue size0 strlen ulen strlen resultbuflen u8 resultbuf u8 u8 svpvbytestrulen u8 e u8 svendstr optimise common case called perlioencode_fill standard length buffer case result svs buffer used temporarily afford allocate maximum needed care unused space const bool temp_result ulen perlio_bufsiz st0 sv_2mortalresult svutf8_onresult endian ssize e uv bom endian size 4 n n bom enc_unpackathx_ sesizeendian bom bom_be bom bom16le endian v else bom bom32le endian v else croaksvfunrecognised bom uvxf hv_fetchhv svrvobjname40 bom uv bom endian size 4 n n bom enc_unpackathx_ sesizeendian bom bom_be bom bom16le endian v else bom bom32le endian v else croaksvfunrecognised bom uvxf hv_fetchhv svrvobjname40 bom 1 update endian next sequence renewed hv_storehv svrvobjendian6newsvpvchar endian10 update endian next sequence svtrueattrrenewed 7 hv_storehv svrvobjendian6newsvpvchar endian10 endif e ssize e uv ord enc_unpackathx_ sesizeendian u8 issurrogateord ucs2 size 4 check croaksvfno surrogates allowed uvxf hv_fetchhv svrvobjname40 ord ssize e skip next one well enc_unpackathx_ sesizeendian ord fbchar else uv lo ishisurrogateord check croaksvfmalformed hi surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ssize e partial character size back 1st half break exit loop lo enc_unpackathx_ sesizeendian islosurrogatelo check croaksvfmalformed lo surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ord 0x10000 ord 0xd800 10 lo 0xdc00 ord 0xfffe 0xfffe ord 0xfdd0 ord 0xfdef check croaksvfunicode character uvxf illegal hv_fetchhv svrvobjname40 ord else ord fbchar u8 svgrowresultsvcurresultutf8_maxlen1 uvuni_to_utf8_flagsdsvcurresult ord 0 svcur_setresultd u8 svpvxresult e unlikely happen fixedlength dankogai check encode_warn_on_err perl_warnerathx_ packwarnwarn_utf8svfpartial character hv_fetchhv svrvobjname40 temp_result resultbuflen 1 ulenusize utf8_maxlen else preallocate buffer minimum possible space required resultbuflen ulenusize utf8_maxlen 1 resultbuf u8 svgrowresult resultbuflen e ssize e uv ord enc_unpackathx_ sesizeendian u8 issurrogateord ucs2 1 ucs2 svtrueattrucs2 4 ucs2 size 4 check croaksvfno surrogates allowed uvxf hv_fetchhv svrvobjname40 ord ssize e skip next one well enc_unpackathx_ sesizeendian ord fbchar else uv lo ishisurrogateord check croaksvfmalformed hi surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ssize e partial character size back 1st half break exit loop lo enc_unpackathx_ sesizeendian islosurrogatelo check croaksvfmalformed lo surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ord 0x10000 ord 0xd800 10 lo 0xdc00 ord 0xfffe 0xfffe ord 0xfdd0 ord 0xfdef check croaksvfunicode character uvxf illegal hv_fetchhv svrvobjname40 ord else ord fbchar resultbuflen svcurresult utf8_maxlen 1 allocate 8mb minimum needed prevents allocating much rogue case large input consisting initially long sequence uft8byte unicode chars followed single utf8byte chars strlen remaining e susize strlen max_alloc remaining 810241024 strlen est_alloc remaining utf8_maxlen strlen newlen svlenresult minmax_alloc est_alloc est_alloc max_alloc max_alloc est_alloc resultbuf u8 svgrowresult newlen resultbuflen svlenresult uvuni_to_utf8_flagsresultbufsvcurresult ord 0 svcur_setresult u8 svpvxresult check check encode_leave_src e movessvpvxstresu8 svcur_setstres else svcur_setstr0 svendstr 0 unlikely happen fixedlength dankogai check encode_warn_on_err perl_warnerathx_ packwarnwarn_utf8svfpartial character hv_fetchhv svrvobjname40 check check encode_leave_src e movessvpvxstresu8 svcur_setstres else svcur_setstr0 svendstr 0 temp_result shrink_bufferresult xsreturn1 22275 27992 iv check code u8 endian u8 svpv_nolenattrendian 6 int size svivattrsize 4 int ucs2 svtrueattrucs2 4 int renewed svtrueattrrenewed 7 sv result newsvpvn0 u8 endian u8 svpv_nolenattrendian 6 const int size svivattrsize 4 int ucs2 1 needed invalid_ucs2 input const strlen usize size 0 size 1 sv result newsvpvn 0 strlen ulen u8 u8 svpvutf8utf8ulen u8 e u8 svendutf8 u8 u8 svpvutf8utf8 ulen const u8 e u8 svendutf8 optimise common case called perlioencode_flush standard length buffer case result svs buffer used temporarily afford allocate maximum needed care unused space const bool temp_result ulen perlio_bufsiz st0 sv_2mortalresult preallocate result buffer maximum possible size ie assume utf8 byte 1 character shrink results buffer necesary end svgrowresult ulen1 usize endian endian size 4 n n enc_packathx_ resultsizeendianbom_be endian size 4 n n enc_packathx_ resultsizeendianbom_be 1 update endian next sequence renewed hv_storehv svrvobjendian6newsvpvchar endian10 update endian next sequence svtrueattrrenewed 7 hv_storehv svrvobjendian6newsvpvchar endian10 endif e sutf8skips e strlen len uv ord utf8n_to_uvunis es len 0 len size 4 invalid_ucs2ord issurrogateord ucs2 check croaksvfcode point xuvxf high hv_fetchhv svrvobjname40ord enc_packathx_ resultsizeendianfbchar else uv hi ord 0x10000 10 0xd800 uv lo ord 0x10000 0x3ff 0xdc00 enc_packathx_ resultsizeendianhi enc_packathx_ resultsizeendianlo else supposed happen enc_packathx_ resultsizeendianfbchar strlen len uv ord utf8n_to_uvunis es len 0 len size 4 invalid_ucs2ord issurrogateord ucs2 1 ucs2 svtrueattrucs2 4 ucs2 check croaksvfcode point xuvxf high hv_fetchhv svrvobjname40ord enc_packathx_ resultsizeendianfbchar else uv hi ord 0x10000 10 0xd800 uv lo ord 0x10000 0x3ff 0xdc00 enc_packathx_ resultsizeendianhi enc_packathx_ resultsizeendianlo else supposed happen enc_packathx_ resultsizeendianfbchar else enc_packathx_ resultsizeendianord else enc_packathx_ resultsizeendianord e utf8 partial char happens often perlio since okay normal warn critical choose leave_src case die check encode_die_on_errencode_leave_src perl_croakathx_ svfpartial character allowed check 0x uvuf hv_fetchhv svrvobjname40 check check check encode_leave_src e movessvpvxutf8esu8 svcur_setutf8es utf8 partial char happens often perlio since okay normal warn critical choose leave_src case die check encode_die_on_errencode_leave_src perl_croakathx_ svfpartial character allowed check 0x uvuf hv_fetchhv svrvobjname40 check check check encode_leave_src e movessvpvxutf8esu8 svcur_setutf8es else svcur_setutf80 svendutf8 0 else svcur_setutf80 svendutf8 0 temp_result shrink_bufferresult xsreturn1 wed 06 aug 2008 224902 0400 patch faster utf1632 decode_xs encode_xs