1. YARN
1.1. Applications are submitted to YARN resource manager
1.2. Coordinates the allocation of compute resources
1.3. Node managers (on each data node) are responsible for launching compute containers
1.4. Allows all kinds of apps to run on the cluster
2. MapReduce
2.1. Scale out architecture
2.2. Structured and unstructured
2.3. Runs on large clusters, commodity hardware
2.4. fault tolerant
2.5. Java/C#/Python....
2.6. Interoperates with Hive and Pig
2.7. Phases
2.7.1. Accepts KVP as input, outputs KVP
2.7.2. Input - place the data onto HDFS
2.7.3. Split - divide lines onto mapper instances
2.7.4. Map - transform input into intermediate output
2.7.5. Shuffle - sort and transfer map output to reducer
2.7.6. Reducer - aggregate values from shuffle
2.7.7. Output - aggregated list from the reduce phase
2.8. Joins
2.8.1. Map Side joins
2.8.1.1. performed on the mapper
2.8.1.2. faster but has constraints
2.8.1.2.1. sorted by the same key
2.8.1.2.2. equal number of partitions
2.8.1.2.3. all records of the same key have to be in the same partitions
2.8.2. Reduce side joins
2.8.2.1. Performed on the reducer
2.8.2.2. less efficient
2.8.2.2.1. because they have to be shuffled
2.8.3. Distributed cache
2.8.3.1. can be used in a map side join
2.8.3.2. efficient and may eliminate the need for areducer
2.9. Combiner
2.9.1. optional mini reducer
2.9.2. runs in memory. after map, before reducer
2.10. Partitioner
2.10.1. Defines how keys are assigned to reducers
2.10.2. Determines which reducer receives which KVP
2.11. Interfaces
2.11.1. Mapper Interface
2.11.1.1. KVP into intermediate KVP
2.11.2. Reducer Interface
2.11.2.1. Data aggregation
2.11.2.2. Processes them using Reduce into final KVPs
3. components
3.1. HQL statement is sent to the driver
3.2. compiler invoked by the driver
3.3. compiler translates the statement into DAG - Directed Acyclic Graph
3.4. Driver submits jobs to the execution engine
4. Pig
4.1. Scripting language for analysing large data sets
4.2. Similar to SQL and does not require Java
4.3. Can run on Hadoop 1 and 2 without changes
5. Name Node
5.1. not a single point of failure
6. Passive Name Node
6.1. Used for reads (possibly)
6.2. Name nodes per group (marketing, IT etc)
7. Data organization
7.1. Database
7.1.1. catalog of namespaces that separate tables
7.1.2. schema can evolve
7.2. Table
7.2.1. Table types
7.2.1.1. internal/managed
7.2.1.1.1. HDFS
7.2.1.1.2. Hive controls life cycle, data is deleted with table
7.2.1.2. external
7.2.1.2.1. stored outside of hive
7.2.1.2.2. data does not get deleted when table is deleted
7.3. Partition
7.3.1. A directory
7.3.2. Can reduce size of map stages, mappers, I/O, and time
7.4. Bucket
7.4.1. a file in a table directory
7.4.2. separates table data int more manageable parts
7.4.3. instead of creating lots of partitions
7.5. View
7.5.1. logical construct
7.5.2. treated like a table
7.6. Index
7.6.1. compaction
7.6.2. bitmap
7.7. Hive metastore
7.7.1. hive metadata
7.7.2. sql (mysql, postgres etc)
7.7.3. stores data about tables and table locations
7.7.3.1. partitions, schemas, table, columns