davies alex awycctsptccom heres revised patch unicodexsv 23 also includes indentation fixes return back 22 unicodexsorig 20071010 120735294849100 0100 unicodexs 20071017 164701058716500 0100 2428 2428 u8 sp uv v 0 ssize e croakpartial character cchar endian croakpartial character cchar endian switchendian case n v v v 8 v v v 8 case n v v 8 v v 8 break v v 8 v v 8 break case v case v v v 8 endian v break v 16 v 24 break v v 8 endian v break v 16 v 24 break default croakunknown endian cchar endian break croakunknown endian cchar endian break sp return v 5825 5825 switchendian case v case v svcurresult svcur_setresultsvcurresultsize size u8value 0xff value 8 break svcurresult svcur_setresultsvcurresultsize size u8value 0xff value 8 break case n case n svcur_setresultsvcurresultsize svcurresult size u8value 0xff value 8 break svcur_setresultsvcurresultsize svcurresult size u8value 0xff value 8 break default croakunknown endian cchar endian break croakunknown endian cchar endian break 95123 95172 code u8 endian u8 svpv_nolenattrendian 6 int size svivattrsize 4 int ucs2 svtrueattrucs2 4 int renewed svtrueattrrenewed 7 int size svivattrsize 4 int ucs2 1 needed event surrogate pairs sv result newsvpvn0 strlen usize size 0 size 1 protect rogue size0 strlen ulen strlen resultbuflen u8 resultbuf u8 u8 svpvbytestrulen u8 e u8 svendstr optimise common case called perlioencode_fill 1024 byte buffer value comes perlioencode_get_base case result svs buffer used temporarily afford allocate maximum needed care unused space bool temp_result ulen 1024 st0 sv_2mortalresult svutf8_onresult endian ssize e uv bom endian size 4 n n bom enc_unpackathx_ sesizeendian uv bom endian size 4 n n bom enc_unpackathx_ sesizeendian bom bom_be bom bom16le endian v else bom bom32le endian v else croaksvfunrecognised bom uvxf bom bom16le endian v else bom bom32le endian v else croaksvfunrecognised bom uvxf hv_fetchhv svrvobjname40 bom bom 1 update endian next sequence renewed hv_storehv svrvobjendian6newsvpvchar endian10 update endian next sequence svtrueattrrenewed 7 hv_storehv svrvobjendian6newsvpvchar endian10 endif temp_result resultbuflen 1 ulenusize utf8_maxlen 14kb else preallocate buffer assuming 1 utf8 byte per unicode character resultbuflen ulenusize utf8_maxlen 1 resultbuf u8 svgrowresult resultbuflen e ssize e uv ord enc_unpackathx_ sesizeendian u8 issurrogateord ucs2 size 4 check croaksvfno surrogates allowed uvxf hv_fetchhv svrvobjname40 ord ssize e uv ord enc_unpackathx_ sesizeendian u8 issurrogateord ucs2 1 ucs2 svtrueattrucs2 4 ucs2 size 4 check croaksvfno surrogates allowed uvxf hv_fetchhv svrvobjname40 ord ssize e skip next one well enc_unpackathx_ sesizeendian ord fbchar else uv lo ishisurrogateord check croaksvfmalformed hi surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ssize e partial character size back 1st half break exit loop lo enc_unpackathx_ sesizeendian islosurrogatelo check croaksvfmalformed lo surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ord 0x10000 ord 0xd800 10 lo 0xdc00 ord 0xfffe 0xfffe ord 0xfdd0 ord 0xfdef check croaksvfunicode character uvxf illegal hv_fetchhv svrvobjname40 ord else ord fbchar u8 svgrowresultsvcurresultutf8_maxlen1 uvuni_to_utf8_flagsdsvcurresult ord 0 svcur_setresultd u8 svpvxresult enc_unpackathx_ sesizeendian ord fbchar else uv lo ishisurrogateord check croaksvfmalformed hi surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ssize e partial character size back 1st half break exit loop lo enc_unpackathx_ sesizeendian islosurrogatelo check croaksvfmalformed lo surrogate uvxf hv_fetchhv svrvobjname40 ord else ord fbchar else ord 0x10000 ord 0xd800 10 lo 0xdc00 ord 0xfffe 0xfffe ord 0xfdd0 ord 0xfdef check croaksvfunicode character uvxf illegal hv_fetchhv svrvobjname40 ord else ord fbchar resultbuflen svcurresult utf8_maxlen 1 allocate 8mb minimum needed prevents allocating much rogue case large input consisting initially long sequence uft8byte unicode chars followed single utf8byte chars strlen remaining e susize strlen max_alloc remaining 810241024 strlen est_alloc remaining utf8_maxlen strlen newlen svlenresult minmax_alloc est_alloc est_alloc max_alloc max_alloc est_alloc resultbuf u8 svgrowresult newlen resultbuflen svlenresult uvuni_to_utf8_flagsresultbufsvcurresult ord 0 svcur_setresult u8 svpvxresult e unlikely happen fixedlength dankogai check encode_warn_on_err perl_warnerathx_ packwarnwarn_utf8svfpartial character hv_fetchhv svrvobjname40 unlikely happen fixedlength dankogai check encode_warn_on_err perl_warnerathx_ packwarnwarn_utf8svfpartial character hv_fetchhv svrvobjname40 check check encode_leave_src e movessvpvxstresu8 svcur_setstres else svcur_setstr0 e movessvpvxstresu8 svcur_setstres else svcur_setstr0 svendstr 0 svendstr 0 avoid wasting much space result buffer temp_result svlenresult 42 svcurresult char buf strlen datalen 1 svcurresult include nul byte strlen buflen perl_strlen_roundupdatalen newxbuf buflen char copysvpvxresult buf datalen char safefreesvpvxresult svpv_setresult buf svlen_setresult buflen xsreturn1 23265 28164 u8 e u8 svendutf8 st0 sv_2mortalresult endian endian size 4 n n enc_packathx_ resultsizeendianbom_be endian size 4 n n enc_packathx_ resultsizeendianbom_be 1 update endian next sequence renewed hv_storehv svrvobjendian6newsvpvchar endian10 update endian next sequence renewed hv_storehv svrvobjendian6newsvpvchar endian10 endif e sutf8skips e strlen len uv ord utf8n_to_uvunis es len 0 strlen len uv ord utf8n_to_uvunis es len 0 len size 4 invalid_ucs2ord issurrogateord ucs2 check croaksvfcode point xuvxf high hv_fetchhv svrvobjname40ord enc_packathx_ resultsizeendianfbchar else uv hi ord 0x10000 10 0xd800 uv lo ord 0x10000 0x3ff 0xdc00 enc_packathx_ resultsizeendianhi enc_packathx_ resultsizeendianlo else supposed happen enc_packathx_ resultsizeendianfbchar else enc_packathx_ resultsizeendianord size 4 invalid_ucs2ord issurrogateord ucs2 check croaksvfcode point xuvxf high hv_fetchhv svrvobjname40ord enc_packathx_ resultsizeendianfbchar else uv hi ord 0x10000 10 0xd800 uv lo ord 0x10000 0x3ff 0xdc00 enc_packathx_ resultsizeendianhi enc_packathx_ resultsizeendianlo else supposed happen enc_packathx_ resultsizeendianfbchar else enc_packathx_ resultsizeendianord e utf8 partial char happens often perlio since okay normal warn critical choose leave_src case die check encode_die_on_errencode_leave_src perl_croakathx_ svfpartial character allowed check 0x uvuf hv_fetchhv svrvobjname40 check utf8 partial char happens often perlio since okay normal warn critical choose leave_src case die check encode_die_on_errencode_leave_src perl_croakathx_ svfpartial character allowed check 0x uvuf hv_fetchhv svrvobjname40 check check check encode_leave_src e movessvpvxutf8esu8 svcur_setutf8es else svcur_setutf80 e movessvpvxutf8esu8 svcur_setutf8es else svcur_setutf80 svendutf8 0 svendutf8 0 xsreturn1 also heres timing test script play use warnings use strict use timehires use encode 1 f utf16le_datatxt unless f f create utf16 open f rawperlioencodingutf16le f die write f n line x 80 n 110_000 print f line close f time long read utf16 via encoding layer open f rawperlioencodingutf16le f die open f n start timehirestime 110 seek f 0 0 die seek n print took timehirestime start n close f read utf16 buffer decode open f f die open f n binmode f read f utf16_data 10_000_000 die cant read n close f time long go thro explicit call decode start timehirestime 110 utf8 encodedecodeutf16le utf16_data print took timehirestime start n cheers alex wed 06 aug 2008 122929 0400 patch faster unicodexs decode_xs