CareerCup

We could consider using two BloomFilters here, one for each file.
And since BloomFilters are not 100% accurate. We could compare the resulting BloomFilters of both the files if they match close to say something like ~97% (value depending on the efficiency of the HashFunction) then we could say they are identical.
Please correct me if I'm wrong.

Read 1st file and split into words .
put in a list and sort it .
read 2nd file do the above step.
iterate 1st list and remove from 2nd list

This is a little easy using pig...

1. Read file.
2. Extract words from file. Skip all escape characters.
3. Sort all words.
5. Make hash from sorted words (SHA2)
6. Compare hashes from 1st and 2nd files.

1. Remove all white spaces and new line characters.
2. Create object with these updated files.
3. Compare both objects.
Please kindly revert back to me if any concern.

Read first file.
Create a Hashtable of for a list of integers.
Key = word
list of positions in document.
Count words as table is built.
Read second file.
Increment counter for each match of word and position.
Print (MatchCount/WordCount)*100 for percentage of match.

1. split file, we can get every file 20,000,000
2. read file line by line, hash(line)/1000, according to hash number , every line write in to file1_[0,999], file2_[0,999]
3. now we just compare file1_0/file2_0, file1_1/file2_1 ... file1_999/file2_999

1. Have a file handler, (size_t) reminder & char *buffer(of size BUFF_SIZE) for each of the files.
2. Read file contents to buff1 & buff2
where, BUFF_SIZE1= BUFF_SIZE-reminder1
BUFF_SIZE1= BUFF_SIZE-reminder1
3. Remove newline from both the buffers and prefix last reminder chars (this may lead in buff length mismatch, so keep the extra chars of longer buff as reminder for next read).
3. Compare the formatted buffers
4. Go to step 2 with reminder1 & reminder2

This is your class homework. Sorry.