I need to compare two files of differing formats quickly and I'm not sure how to do it. I would very much appreciate it if someone could point me in the right direction.
I am working on CentOS 6 and I am most comfortable with Python (both Python 2 and Python 3 are available).

The problem
I am looking to compare the contents two large files (quickly). The files, unfortunately, differ in content; I will need to modify the contents of one before I can compare them. They are not particularly well-organized, so I can't move linearly down each and compare line-by-line.
Here's an example of the files:

File 1                    File 2
Job,Time                  Job,Start,End
0123,3-00:00:00           0123,2016-01-01T00:00:00,2016-01-04T00:00:00
1111,05:30:00             1111,2016-01-01T00:00:00,2016-01-01T05:30:00
0000,00:00:05             9090.abc,2016-01-01T12:00:00,2016-01-01T22:00:00
9090.abc,10:00:00         0000,2015-06-01T00:00:00,2015-06-01T00:00:05
...                       ...

I would like to compare the contents of lines with the same "Job" field, like so:

Job        File 1 Content   File 2 Content
0123       3-00:00:00       2016-01-01T00:00:00,2016-01-04T00:00:00
1111       05:30:00         2016-01-01T00:00:00,2016-01-01T05:30:00
0000       00:00:05         2015-06-01T00:00:00,2015-06-01T00:00:05
9090.abc   10:00:00         2016-01-01T12:00:00,2016-01-01T22:00:00
...        ...              ...

I will be performing calculations on the File 1 Content and File 2 Content and comparing the two (for each line).
What is the most efficient way of doing this (matching lines)?

The system currently in place loops through one file in its entirety for each line in the other (until a match is found). This process may take hours to complete, and the files are always growing. I am looking to make the process of comparing them as efficient as possible, but even marginal improvements in performance can have a drastic effect.
I appreciate any and all help.
Thank you!
I was trying to develop something where you'd split one of the files into smaller files (say 100,000 records each) and keep a pickled dictionary of each file that contains all Job_id as a key and its line as a value. In a sense, an index for each database and you could use a hash lookup on each subfile to determine whether you wanted to read its contents.
However, you say that the file grows continually and each Job_id is unique. So, I would bite the bullet and run your current analysis once. Have a line counter that records how many lines you analysed for each file and write to a file somewhere. Then in future, you can use linecache to know what line you want to start at for your next analysis in both file1 and file2; all previous lines have been processed so there's absolutely no point in scanning the whole content of that file again, just start where you ended in the previous analysis. 
If you run the analysis at sufficiently frequent intervals, who cares if it's O(n^2) since you're processing, say, 10 records at a time and appending it to your combined database. In other words, the first analysis takes a long time but each subsequent analysis gets quicker and eventually n should converge on 1 so it becomes irrelevant.
Comparing the contents of very large files efficiently