MD5 Hashing using parallel processing

Dec 14 2009 10:45 AM
I need to be able to process MD5 hash information for very large files (100+ GBytes).  This obviously takes some time.  As opposed to opening a filestream and sequentially reading and hashing the bytes, I want to use a random access file access method and open the file, in pieces and aggregate the hash information (i.e. server 1 opens 1st 1/2 of the file and computes hash info, while server 2, at the same time opens the 2nd 1/2 of the file and does the same. At completion time, one of the servers takes the output from the other and creates the final hash which represents the entire large file.  The intent is to spread this across all the available nodes in a SAN resulting in very fast hashing, for large files.  Using System.Security.Cryptography.MD5, is there an easy way to do this?

Answers (3)