Question: I was asked this question in an interview recently. 

> 
> Given an input file, a regex and an output file. Read the input file, match each line to the regex and write the matching lines to an output file. 
> 
> 
> 

I came up with the rough scheme of using a BufferedReader chained to a FileReader (to optimize Reads from the disk). I used a similar scheme for writing. 

> 
> The interviewer then said that this process takes 3 seconds to read a line from the file, 1 second to compare the regex with the line and another 5 seconds to write back. so it takes a total of 9 seconds per line. How can we improve this? 
> 
> 
> 

I suggested reading the entire file at once, processing it and writing the entire output file at once. However, I was told that won't help (Writing 1 line = 5 seconds, writing 2 lines = 10 seconds) .

> 
> The interviewer further said that this is due to a hardware/ hard drive limitation. I was asked how I can improve my code to reduce the total seconds (currently 9 ) per line? 
> 
> 
> 

I could only think of buffered reading/ writing and could not find much on SO as well. Any thoughts ?

Answer: 
I think that the interviewer was looking for a solution that performs reading/regex checking in parallel with writing the output. If you set up a work queue that you fill asynchronously by reading and filtering, and put writing in a separate thread, then the combined process would take five seconds per line, starting with the second line.

The assumption here is that reading, parsing, and writing can happen independently of each other. In this case, you can be reading line 2 while line 1 is being written: you need only four seconds to read and apply your regex, and you've got a whole five seconds before the writer is ready for the second line. The writing remains your bottleneck, but the whole process gets sped up by some 44%, which isn't bad.
Well since the time for a read is fixed and the time for a write is fixed, the only option you have in this case is to change the nature of the regex bit.

You could write code to apply the regex test quickly without the overhead of all the clever things that regex can do.

If on the other hand the problem is that each IO request takes several seconds to be executed, but the limitation is not the actual drive, then have several readers reading simultaneously.