Apache Hadoop: Manipulation and Transformation of Data Performance

1.1Hadoop Concepts 1.1.1HDFS

• The Design of HDFS
• Command line interface
• Hadoop File System

• Anatomy of a cluster
• Mater Node / Slave node
• Name Node / Data Node

• Map phase
• Reduce phase
• Shuffle

• Group-By with MapReduce
• Frequency distributions and sorting with MapReduce
• Plotting results (GNU Plot)
• Histograms with MapReduce
• Scatter plots with MapReduce
• Parsing complex datasets
• Counting with MapReduce and Combiners
• Build reports

• Document Cleaning
• Fuzzy string search
• Record linkage / data deduplication
• Transform and sort event dates
• Validate source reliability
• Trim Outliers

• Transforming logs
• Using Apache Pig to filter
• Using Apache Pig to sort
• Using Apache Pig to sessionize

• Joining data in the Mapper using MapReduce
• Joining data using Apache Pig replicated join
• Joining sorted data using Apache Pig merge join
• Joining skewed data using Apache Pig skewed join
• Using a map-side join in Apache Hive
• Using optimized full outer joins in Apache Hive
• Joining data using an external key value store

• Map
  • Investigating spikes in input data
  • Identifying map-side data skew problems
  • Map task throughput
  • Small files
  • Unsplittable files
• Reduce
  • Too few or too many reducers
  • Reduce-side data skew problems
  • Reduce tasks throughput
  • Slow shuffle and sort
• Competing jobs and scheduler throttling
• Stack dumps & unoptimized code
• Hardware failures
• CPU contention
• Tasks
  • Extracting and visualizing task execution times
  • Profiling your map and reduce tasks
• Avoid the reducer
• Filter and project
• Using the combiner
• Fast sorting with comparators
• Collecting skewed data
• Reduce skew mitigation