DesignPatterns.md

Design Patterns

Filtering Patterns

Eg. filtering data, sampling data, generating top N from list

• Filtering out posts on forum that matches certain pattern
• Top-N longest posts

Summarization Patterns

Eg. counting records, min/max, statistics, create index

• Inverted Index Mapper - count how many times a word is used on the forum
• Inverted Index Reducer - count how many times a word is used on the forum

Numerical Summarization

Eg. word/record count, mean, median, standard deviation

• Mean Mapper - is there any correlation between the day of the week and how much people spent on items
• Mean Reducer - is there any correlation between the day of the week and how much people spent on items

-> Use of combiners (like semi-reducers in MapReduce)

gedit ~/.bashrc

Structural Patterns

Eg. combining data sets, RDBMS to Hadoop (take advantage of hierarchical data)

1. Data sources linked by foreign keys

2. Data must be structured and row based

• Combine Dataset Mapper - what is the reputation of a post's author
• Combine Dataset Reducer - what is the reputation of a post's author