Question: OS: 14.04
sed: 4.2.4

I have multiple large files (2-4gb) that I want to perform some simple manipulations on. The entire file is in one line, which makes me wonder how to perform sed operations on it.

There are three things I want to do with each file:

1) Remove all `[` characters

2) Remove all `]` characters

3) Replace all occurrences of `},{` with `}{`.

So far I have tried `sed -e 's/},{/}{/g' file.json > file_new.json` with and without the `g` option, without any luck. I have also tried `sed -e 's/\[//g' file.json > file_new.json` without any luck. I only get a duplicate file.

Any ideas?

Answer: 
With gnu awk:

```
awk 'BEGIN{FS="},{";OFS="}{";RS="[][]";ORS=""}$1=$1' file

```

Perhaps faster with perl *(must be tested)*:

```
perl -0135 -pe 's/},{/}{/g;y/][//d' file

```

Where 135 stands for the character `]` in octal. The -0 option defines the record separator (instead of to be read line by line, the file is read by parts from and until each `]`)

The goal of these two scripts is to avoid to load the whole file in memory.

---

To store the result in a file:

You can redirect the output.

```
awk 'BEGIN{FS="},{";OFS="}{";RS="[][]";ORS=""}$1=$1' file > result

```

or

```
perl -0135 -pe 's/},{/}{/g;y/][//d' file > result

```

You can use command line options:

```
awk -i inplace -v INPLACE_SUFFIX=.bak 'BEGIN{FS="},{";OFS="}{";RS="[][]";ORS=""}$1=$1' file

```

or

```
perl -0135 -pi'*.bak' -e 's/},{/}{/g;y/][//d' file

```

*(these two commands create a backup of the original file adding the extension .bak, if you want to change the source file in place, remove `-v INPLACE_SUFFIX=.bak` for gawk, and `'*.bak'` for perl.)*
When I've got huge single-line files like that, for which the usual line-based tools won't work, I usually turn to: `tr`!

> 
> 1) Remove all [ characters  
> 
> 2) Remove all ] characters
> 

That's easy:

```
tr -d '[]' < file > strippedfile

```

(This might not work with a really, really old SysV version of tr, but it should be fine with any modern version.)

> 
> 3) Replace all occurrences of },{ with }{.
> 

That's trickier, because you care about context, so it's really a job for `sed`. One kludge I've used is to use `tr` to temporarily change some other character to a newline -- that is, to temporarily change the huge single-line file into a multi-line file -- then run `sed`, and finally change it back to a single-line file. Something like

```
tr '{' '\n' < file | sed 's/},$/}/' | tr '\n' '{' > newfile

```

This last works *only* if the original file contains *no* newlines. You could run through `tr -d '\n'` first to be sure.