Folders and files Name Name Last commit message
Last commit date
parent directory
View all files
MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat .
Map Reduce detail:
Split input file to some blocks.
Starts up 1 master, K workers which will pick M map tasks and R reduce tasks.
Map worker will read contents from input split, parse key/values pairs out of input data, and pair to user-defined Map functions.
Buffered the intermediate key/values pairs result in memory.
Periodically load to disk, partitioned into R regions by partitioning function and pass back the location of buffered paired to master.
Master will forwarding these location to reduce workers.
When reduce worker is notified by master about the location, by using RPC to read the buffered data from local disk of map workers.
After reading intermediate data, sort key (too large will store to disk).
The ouput of Reduce function is appended to a final output file.
After all worker completed, call user program.
fault tolerant
worker failure
master failure
semantics of presence of failure
Locality
to reduce network communication
Backup
to alleviate "straggler"
When MapReduce is close to completed, start a backup worker to the unfinished worker.
The task is marked as completed whenever the primary worker or the backup worker completed.
You can’t perform that action at this time.