I try to read two one billion record files from a Unix server for file comparison.
I tried in python with paramiko package but it is very slow to connect and read the Unix files. That's why I chose Java.
In Java when I read the file I am facing memory issues and performance issues.
My requirement: first read all records from Unix server file1, then read second file records from Unix server and finally compare the two files.
As you are working in UNIX, I would advise you to sort the files and use the diff commandline tool: UNIX' commandline commands are quite powerful. Please show us an excerpt of the files (maybe you might need cut or awk scripts too).
How to handle two one billion record for file comparison