Question: I want to get rid of all invalid characters; example `hexadecimal value 0x1A` from an XML file using sed.  

What is the regex and the command line?   

EDIT  

Added Perl tag hoping to get more responses. I prefer a one-liner solution.  

EDIT  

These are the valid XML characters  

```
x9 | xA | xD | [x20-xD7FF] | [xE000-xFFFD] | [x10000-x10FFFF]

```

Answer: 
The `tr` command would be simpler. So, try something like:

```
cat <filename> | tr -d '\032' > <newfilename>

```

Note that ascii character '0x1a' has the octal value '032', so we use that instead with `tr`. Not sure if `tr` likes hex.
There is actually a way to do this with sed, like so:

```
cat input_file | LANG=C sed -E \
   -e 's/.*/& /g' \
   -e 's/(('\
'[\x9\xa\xd\x20-\x7f]|'\
'[\xc0-\xdf][\x80-\xbf]|'\
'[\xe0-\xec][\x80-\xbf][\x80-\xbf]|'\
'[\xed][\x80-\x9f][\x80-\xbf]|'\
'[\xee-\xef][\x80-\xbf][\x80-\xbf]|'\
'[\xf0][\x80-\x8f][\x80-\xbf][\x80-\xbf]'\
')*)./\1?/g' \
   -e 's/(.*)\?/\1/g' \
   -e 's|]]>|]]>]]<![CDATA[>|g' > output_file

```

This works in four steps:

1. Add a single whitespace character to the end of every line.
2. Replace every sequence of legal characters followed by any character
with the same sequence of legal characters followed by a question mark
character (instead of the any).
Note that in a line of only legal characters, the '.' matches the last
character in the line, which is why we added a space in step 1.
3. Remove the last character in the line, which we expect to be a question mark.
4. Replace the string ']]>' with ']]>]]'.

The LANG=C env variable is set to prevent sed from doing charset conversion itself - it should treat every character as 8-bit ascii.