difference is mainly about I/O transferring time, the network communication, and job scheduling time, etc. CPU Time This is a little harder to understand, because this time will not change if a job does not change. This is the time consumed by CPU, which is used for processing instructions of a computer program. It is only related to the number of instructions in a program, which means if we parallelize a program, the total execution time might be the third of the original, but the CPU time would not change because the amount of instructions does not change but are executed in parallel. Global Counters Hadoop framework does not have global variable sharing mechanism, it provides a global counter mechanism. From the name we can know that it is mainly for counting. We may need to keep track of some counters like how often a certain event has occurred during the job execution. Because of this feature of global counter, we can use it to share some global values during the whole job, such as how many documents or tweets in total, how many unique words in total. Global counters are reported back to the job tracker from the task tracker repeatedly like heart beating. Impact of Node Number For a cluster, the number of nodes is one of the very important aspects that affect the whole performance of execution. As we introduced in the time complexity section,
difference is mainly about I/O transferring time, the network communication, and job scheduling time, etc. CPU Time This is a little harder to understand, because this time will not change if a job does not change. This is the time consumed by CPU, which is used for processing instructions of a computer program. It is only related to the number of instructions in a program, which means if we parallelize a program, the total execution time might be the third of the original, but the CPU time would not change because the amount of instructions does not change but are executed in parallel. Global Counters Hadoop framework does not have global variable sharing mechanism, it provides a global counter mechanism. From the name we can know that it is mainly for counting. We may need to keep track of some counters like how often a certain event has occurred during the job execution. Because of this feature of global counter, we can use it to share some global values during the whole job, such as how many documents or tweets in total, how many unique words in total. Global counters are reported back to the job tracker from the task tracker repeatedly like heart beating. Impact of Node Number For a cluster, the number of nodes is one of the very important aspects that affect the whole performance of execution. As we introduced in the time complexity section,
ZhaoyuLi2014.docx