Network Traffic Analysis in Map Reduce for Bigdata Applications

EasyChair Preprint no. 8422

Abstract

Through the use of parallel map and reduce activities, the map-reduce programming methodology makes it easier to handle massive amounts of data in groups of items. While significant work has been done to boost the efficiency of map reduce tasks, this work ignores the network traffic created during the shuffle phase, which is vital to boosting efficiency in general. Historically, a hash function is used to partition intermediate data between reduction activities, which, however, are not traffic efficient due to the fact that the network topology and the size of the data associated with each is not considered key code. In this paper, we will examine how to reduce network traffic costs for a map reduction process by designing a new intermediate data partitioning scheme. Plus, together let's not forget the hassle of aggregator location, where each aggregator can reduce the combined traffic of multiple map activities. A set of assigned algorithms based primarily on decomposition is proposed to address the problem of large-scale optimization for large data programs, and a web set of rules is also designed to dynamically modify the partitioning and aggregation of data. In the end, the simulation results demonstrate that our suggestions can still significantly lower network traffic, both online and offline.

Keyphrases: BigData, Map Reduce, network traffic, web application

@Booklet{EasyChair:8422,
  author = {K Lakshmikanth and K N Prajwalsidhu and H N Rakshitha and A V Krishnamohan},
  title = {Network Traffic Analysis in Map Reduce for Bigdata Applications},
  howpublished = {EasyChair Preprint no. 8422},

  year = {EasyChair, 2022}}