I am working with a very large (~11GB) text file on a Linux system.  I am running it through a program which is checking the file for errors.  Once an error is found, I need to either fix the line or remove the line entirely.  And then repeat...
Eventually once I'm comfortable with the process, I'll automate it entirely.  For now however, let's assume I'm running this by hand.
What would be the fastest (in terms of execution time) way to remove a specific line from this large file?  I thought of doing it in Python...but would be open to other examples.  The line might be anywhere in the file.
If Python, assume the following interface:
def removeLine(filename, lineno):
Thanks,
-aj
I think there was a somewhat similar if not exactly the same type of question asked here. Reading (and writing) line by line is slow, but you can read a bigger chunk into memory at once, go through that line by line skipping lines you don't want, then writing this as a single chunk to a new file. Repeat until done. Finally replace the original file with the new file.
The thing to watch out for is when you read in a chunk, you need to deal with the last, potentially partial line you read, and prepend that into the next chunk you read.
Fastest Way to Delete a Line from Large File in Python