We're updating the issue view to help you get more done. 

Consolidate shuffle files

Description

Right now on each machine, we create M * R temporary files for shuffle, where M = number of map tasks, R = number of reduce tasks.

This can be pretty high when there are lots of mappers and reducers (e.g. 1k map * 1k reduce = 1 million files for a single shuffle). The high number can cripple the file system and significantly slow the system down.

We should cut this number down to O(R) instead of O(M*R).

Environment

None

Status

Assignee

Jason Dai

Reporter

Reynold Xin

Labels

None

Test

None

Test

None

Fix versions

Priority

Major