Spark

• Fast and expressive cluster computing system interoperable with MapReduce / Hadoop
• Improves performance (orders of magnitude faster)
  • In memory computing primitives
  • General computation graphs
• Improves usability (less code)
  • API in Scala, Java, Python
  • Interactive Shell

Stack

• Berkeley Data Analytics Stack (BDAS)

SPARK SQL	Spark Streaming	GraphX	MLIB
	(real-time)	graph	(Machine Learning)

Data Model

• Resilient Distributed Datasets (RDDs)
  • Distributed collections of objects that can be cached in memory across the cluster
  • Manipulated through parallel operators
  • Automatically recomputed on failure
• Program Interface
  • Functional API's in Scala, Java, Python
  • Interactive use from Scala Shell

Fault Tolerance

RDD's track lineage info to rebuild lost data.

file.map(lamba rec: (rec.type, 1))
	.reduceByKey(lambda x, y: x + y)
	.filter(lambda (type, count) : count > 100)

---

Data Model

• Semi-relational (relational with a few difference)
  • Each relation has to have at least one primary key
  • Several primary keys should be ordered
  • Relations have to be grouped into hierarchies and partitioned
• Adoption of semi-relational data model + synchronous replication on top of Big-table led to development of spanner
• Spanner turned into a global temporal multi-version large scale semi-relational data store
• SQL-based query language

Version Control

• Based on global timestamps
• Correspond to valid global serialization order
• T0 commits before T1 starts
  • T0's timestamp < T1's timestamp

True Time API

Methods

• TT.now() returns TTinterval (earliest, latest)
• TT.after(t) - returns true if time is after case

Instantaneous error bound $\epsilon$

• Define $\epsilon$ = (latest - earliest) / 2
• $\epsilon$ can be set based on the worst case of clock drift
• $\epsilon$ depends on the master time uncertainty and communication delay

---

Google TPU

• Map / Reduce and spark based machine learning frameworks
  • Very slow and energy inefficent at scale
• First Generation TPU
  • Classification stage implemented within hardware
• Architecture (1st Generation TPU)
  • Matrix Multiply Unit (MMU) (yellow)
  • Weight FIFO (blue) -> (MMU)
  • Unified Buffer (blue) -> (MMU)
  • MMU -> Accumulators