Question: OS: 14.04
sed: 4.2.4

I have multiple large files (2-4gb) that I want to perform some simple manipulations on. The entire file is in one line, which makes me wonder how to perform sed operations on it.

There are three things I want to do with each file:

1) Remove all `[` characters

2) Remove all `]` characters

3) Replace all occurrences of `},{` with `}{`.

So far I have tried `sed -e 's/},{/}{/g' file.json > file_new.json` with and without the `g` option, without any luck. I have also tried `sed -e 's/\[//g' file.json > file_new.json` without any luck. I only get a duplicate file.

Any ideas?

Answer: 
With gnu awk:

```
awk 'BEGIN{FS="},{";OFS="}{";RS="[][]";ORS=""}$1=$1' file

```

Perhaps faster with perl *(must be tested)*:

```
perl -0135 -pe 's/},{/}{/g;y/][//d' file

```

Where 135 stands for the character `]` in octal. The -0 option defines the record separator (instead of to be read line by line, the file is read by parts from and until each `]`)

The goal of these two scripts is to avoid to load the whole file in memory.

---

To store the result in a file:

You can redirect the output.

```
awk 'BEGIN{FS="},{";OFS="}{";RS="[][]";ORS=""}$1=$1' file > result

```

or

```
perl -0135 -pe 's/},{/}{/g;y/][//d' file > result

```

You can use command line options:

```
awk -i inplace -v INPLACE_SUFFIX=.bak 'BEGIN{FS="},{";OFS="}{";RS="[][]";ORS=""}$1=$1' file

```

or

```
perl -0135 -pi'*.bak' -e 's/},{/}{/g;y/][//d' file

```

*(these two commands create a backup of the original file adding the extension .bak, if you want to change the source file in place, remove `-v INPLACE_SUFFIX=.bak` for gawk, and `'*.bak'` for perl.)*
Try this to place a newline at the end of the file:

```
echo "" >> file
sed 'whatever' file

```

Many UNIX tools will simply not recognize a file with no ending newlines as a text file and so will not operate on them so that MAY be your problem. If that doesn't work then edit your question to include a concise, testable example of your file.