davies alex awycctsptccom heres patch speeds reading unicode data via encodeunicodes decode_xs function test script simply reads utf16le data runs roughly half time works replacing previous grow result buffer character method allocating maximium possible length buffer begin patch uses maximum possible length c ulen 1 utf8_maxlen massively pessimistic ie smaller estimate possible nb method assumes decode_xs called arbitrary length input buffer ie arent potential going hit heavy handed utf8_maxlen multiplier requiring large allocation svgrow might fail previous code would however encodingxs always packages tobedecoded buffers 1024 byte chunks think ok heres patch also includes micro optimisation looking renewed ucs2 attributes needed typically unicodexsorig 20071010 120735294849100 0100 unicodexs 20071010 164245175942200 0100 9612 9613 u8 endian u8 svpv_nolenattrendian 6 int size svivattrsize 4 int ucs2 svtrueattrucs2 4 int renewed svtrueattrrenewed 7 int ucs2 1 needed event surrogate pairs sv result newsvpvn0 strlen ulen u8 u8 svpvbytestrulen u8 e u8 svendstr u8 root st0 sv_2mortalresult svutf8_onresult 12415 12522 1 update endian next sequence renewed svtrueattrrenewed 7 hv_storehv svrvobjendian6newsvpvchar endian10 endif preallocate temporary result buffer maximum possible size root u8 svgrowresult ulen 1 utf8_maxlen e ssize e uv ord enc_unpackathx_ sesizeendian u8 issurrogateord ucs2 1 ucs2 svtrueattrucs2 4 ucs2 size 4 check croaksvfno surrogates allowed uvxf 1918 1997 u8 svgrowresultsvcurresultutf8_maxlen1 uvuni_to_utf8_flagsdsvcurresult ord 0 uvuni_to_utf8_flagsrootsvcurresult ord 0 svcur_setresultd u8 svpvxresult e think similar technique might applicable speed encode_xs code aswell heres test script use timehires f utf16letxt create utf16 open f rawperlioencodingutf16le f die write f n 11000 print f foo x 20 n close f open f rawperlioencodingutf16le f die open f n start timehirestime 1200 seek f 0 0 die seek n end timehirestime print took end start n close f also noticed encodeunicodeunicodexs encodeencodexs seem indentations mangled last submission doesnt look deliberate might worth checking happened thanks time cheers alex wed 06 aug 2008 013831 0400 patch faster unicodexs decode_xs