hadoop: limit number of concurrent map / reduce tasks per job -
I want to submit a research work to a production cluster. As I do not need this job to finish quickly, and I do not want to delay production jobs, I want to limit the number of tasks that are executing for that job at any time. Can I do this in Hadoop 2?
To limit resources to Hadoop (reducing mappings / slots) proper scheduler can be used, you can create a new mailbiller pool by setting the maximum number of better mappers and the maximum reducer and job can be deposited in that newly built Meleshadular pool.
Comments
Post a Comment