1. Starting Points
1.1. How do we define "big data"?
1.1.1. By the nature of the data...
1.1.1.1. Classic IBM Definition
1.1.1.2. Other Perspectives
1.1.1.2.1. Bandwidth of the Senses
1.1.1.2.2. Singularity Theory
1.1.2. By the nature of the solutions...
1.1.2.1. "Machine Learning"
1.1.2.1.1. Supervised Learning for Prediction
1.1.2.1.2. Supervised Learning for Classification
1.1.2.1.3. Unsupervised Learning for Clustering
1.1.2.2. "Autonomous Systems"
1.1.2.2.1. Autonomous Systems = Sensing + Learning + Acting
1.1.3. By the nature of the technology...
1.1.3.1. Big Data Technology Landscape
1.1.3.2. Distinctive Elements
1.1.3.2.1. "Wrangling" - the way data is moved and shaped
1.1.3.2.2. "Platforms" - the way data is stored, organised, managed, and accessed
1.1.3.2.3. "Data Science" - the way data is used to generate insights and signals
1.1.4. The reality (an opinion)...
1.1.4.1. The definition of "Big Data" is dependent on context
1.2. What are the objectives of this course?
1.2.1. "...Gain an overview of business applications of big data and analytics techniques..."
1.2.1.1. "Use Cases"
1.2.1.2. "Techniques"
1.2.2. "...Gain real-world insights into various applications of big data analytics and how it can be used to fuel better decision-making within an organisation/business..."
1.2.2.1. "Catalogue"
1.2.2.2. "Challenges"
1.2.3. "...Learn how big data and analytics techniques can create business value..."
1.2.3.1. "Business Case"
1.2.4. "...Understand how to manage big data and analytics projects and teams..."
1.2.4.1. "Technology"
1.2.4.2. "People"
1.2.4.3. "Process"
1.3. How are we going to approach the material?
1.3.1. A catalogue view of "Use Cases"
1.3.2. A thematic dissection of "Use Cases"
2. D. Techniques
2.1. Understanding what analytics can do...
2.1.1. The Analytics Curve
2.1.1.1. Perspective: Describe, Diagnose, Predict, Prescribe
2.1.2. Machine Learning Methods
2.1.2.1. Illustration: Machine Learning Family Tree
2.1.2.2. Principal Types
2.1.2.2.1. Supervised Learning
2.1.2.2.2. Unsupervised Learning
2.1.2.2.3. Reinforcement Learning
2.2. Tasks
2.2.1. Review the United Airlines case, what machine learning technique were they likely to be using?
2.2.2. Team Boards
2.2.2.1. Team #1
2.2.2.2. Team #2
2.2.2.3. Team #3
3. E. Technology
3.1. Technology is a critical enabler of big data solutions...
3.1.1. Big Data Technology Landscape
3.1.2. Principal Technology Challenges
3.1.2.1. ...moving massive volumes of high speed, high variety data from source to the "right place, at the right time, in the right shape"...
3.1.2.1.1. "Wrangling"
3.1.2.2. ...storing, architecting, and managing massive volumes of high variety data, whilst ensuring it is ready to use...
3.1.2.2.1. "Platforms"
3.1.2.3. ...driving insights from data assets in a form and on a schedule that is useful...
3.1.2.3.1. "Data Science"
3.1.3. Pipeline Architecture
3.1.3.1. Big data solutions typically take the form of a pipeline of common components...
3.1.3.1.1. Illustration: Iguazio
3.1.3.1.2. Perspective: Principal Pipeline Components
3.1.3.1.3. Cloud Implementations
3.1.4. The Apache Stack
3.1.4.1. Tools from the Apache Foundation are dominant in the big data world...
3.1.4.1.1. Illustration: Apache Pipeline
3.1.4.1.2. "Killer Apps" of the Apache Stack
3.1.4.1.3. Illustration: Wider Apache Big Data Projects
3.1.5. An Emerging Class
3.1.5.1. ...technologies responding to the choking of data availability resulting from privacy and data monetisation trends...
3.1.5.1.1. Context
3.1.5.1.2. Perspectives: Secure Sharing Platforms, Safe Spaces, and Sandboxes
3.1.5.1.3. Example: Google Ads Data Hub
3.1.6. Tasks
3.1.6.1. Validate Stacks
3.1.6.1.1. 1. Go to StackShare.io
3.1.6.1.2. 2. Browse Stacks
3.1.6.1.3. 3. Select a Big Data Candidate
3.1.6.1.4. 4. Consider Their Configuration
3.1.6.2. Build a stack in Crowdcraft.io
3.1.6.2.1. Target Stack
3.1.6.2.2. 1. Create a Free Account in Crowdcraft.io
3.1.6.2.3. 2. Create New Blueprint
3.1.6.2.4. 3. Compile Components
3.1.6.2.5. 4. Join Arrows from User to Source
3.1.6.3. Team Boards
3.1.6.3.1. Team #1
3.1.6.3.2. Team #2
3.1.6.3.3. Team #3
4. F. Process
4.1. What the human brings to the picture...
4.1.1. Perspective: Translation
4.1.2. Component Processes and Patterns
4.1.2.1. Aligning the technical solution to business objectives...
4.1.2.1.1. Illustration: Tibco CRISP-DM
4.1.2.2. Selecting the right data science technique...
4.1.2.2.1. Illustration: Machine Learning Model Selection
4.1.2.3. Managing an exploratory approach...
4.1.2.3.1. Illustration: OODA
4.1.2.4. Training, testing, and iterating machine learning models...
4.1.2.4.1. Illustration: Towards Data Science
4.1.2.4.2. Illustration: Automated Machine Learning
4.1.2.4.3. ...and adopting the learning pattern...
4.1.2.5. Positioning the learning process...
4.1.2.5.1. ...is the model going to learn and act "online", or will be trained in a sandbox and "batch" insights released?
4.1.2.6. Maintaining model quality...
4.1.2.6.1. Illustration: Overfitting Avoidance
4.1.2.6.2. Illustration: Balancing Bias
4.1.3. Hybrid Models
4.1.3.1. Illustration: Uber Michaelangelo
4.1.3.2. Illustration: Particle Data
4.1.4. Tasks
4.1.4.1. Review and discuss the processes implicit in the Grab Catwalk case...
4.1.4.1.1. Grab Catwalk
4.1.4.2. Team Boards
4.1.4.2.1. Team #1
4.1.4.2.2. Team #2
4.1.4.2.3. Team #3
5. G. People
5.1. Despite all the "tech", success and failure in big data still hinges on people...
5.1.1. Training Big Data Talent
5.1.1.1. ...the challenge of maintaining skills relevance...
5.1.1.1.1. Illustration: Stack "Fingerprints"
5.1.1.1.2. Illustration: Data Science Unicorns
5.1.1.1.3. Skills Fingerprints
5.1.2. Organising Big Data Talent
5.1.2.1. Blending Hubs and Markets
5.1.2.1.1. Perspective: Internal Markets
5.1.2.1.2. Perspective: CoE Capabilities
5.1.2.1.3. Illustration: CoE Positioning
5.1.2.1.4. Perspective: Specifying a CoE
5.1.2.2. Illustration: Uber Michaelangelo
5.1.2.2.1. "...Successfully scaling ML at a company like Uber requires getting much more than just the technology right—there are important considerations for organization and process design as well. In this section, we look at critical success factors across three pillars: organization, process, as well as technology..."
5.1.3. Tasks
5.1.3.1. Brainstorm the principal "Purple Challenges" an organisation aspiring to big data leadership might face...
5.1.3.2. Team Boards
5.1.3.2.1. Team #1
5.1.3.2.2. Team #2
5.1.3.2.3. Team #3
6. User Notes
6.1. Mindmeister
6.1.1. Single source for materials
6.1.2. Public link stays public
6.1.3. Take a personal copy if you want to keep!
6.2. Google Jamboards
6.2.1. For collaboration
6.2.2. Take a personal copy if you want to keep!
7. H. Challenges
7.1. Big data analytics initiatives routinely fail, why?...
7.1.1. Framing: McKinsey's 10 Flags
7.1.1.1. 1. The executive team doesn’t have a clear vision for its advanced-analytics programs
7.1.1.1.1. Illustration: Gartner Hype Cycle
7.1.1.1.2. Illustration: Growth of the Dataverse
7.1.1.2. 2. No one has determined the value that the initial use cases can deliver in the first year
7.1.1.2.1. Illustration: McKinsey Prioritisation
7.1.1.3. 3. There’s no analytics strategy beyond a few use cases
7.1.1.3.1. Illustration: Play to Win Choice Cascade
7.1.1.4. 4. Analytics roles - present and future - are poorly defined
7.1.1.4.1. Illustration: Data Science Unicorns
7.1.1.4.2. Illustration: Uber Michaelangelo
7.1.1.5. 5. The organization lacks analytics translators
7.1.1.5.1. Illustration: Purple People
7.1.1.6. 6. Analytics capabilities are isolated from the business, resulting in an ineffective analytics organization structure
7.1.1.6.1. Illustration: Internal Markets
7.1.1.6.2. Illustration: CoE Positioning
7.1.1.7. 7. Costly data-cleansing efforts are started en masse
7.1.1.7.1. Illustration: Tibco CRISP-DM
7.1.1.8. 8. Analytics platforms aren’t built to purpose
7.1.1.8.1. Illustration: Apache Pipeline
7.1.1.9. 9. Nobody knows the quantitative impact that analytics is providing
7.1.1.9.1. Recommended: Case Making Map
7.1.1.10. 10. No one is hyperfocused on identifying potential ethical, social, and regulatory implications of analytics initiatives
7.1.1.10.1. Illustration: Singapore MAS "FEAT"
7.1.1.10.2. Illustration: United Airlines
7.1.2. Tasks
7.1.2.1. Discuss the FEAT potential exposure of Apple in the credit card case...
7.1.2.1.1. Exhibit: Apple Credit Card Case
7.1.2.2. Team Boards
7.1.2.2.1. Team #1
7.1.2.2.2. Team #2
7.1.2.2.3. Team #3
8. C. Business Case
8.1. Forming a language around the benefits of Big Data...
8.1.1. Impact Assessment and Communication Tools
8.1.1.1. Illustration: Play to Win Choice Cascade
8.1.1.2. Illustration: Dataiku Classification of Returns
8.1.1.3. Illustration: McKinsey Top Bottom Line Benefits
8.1.1.3.1. Extension: McKinsey Prioritisation
8.1.1.4. Illustration: Decomposition with Impact Maps
8.1.2. Recommended: Case Making Map
8.1.3. Tasks
8.1.3.1. Use the case-making map to frame a business case underpinning a "Business Intelligence" (dashboard) investment...
8.1.3.2. Team Boards
8.1.3.2.1. Team #1
8.1.3.2.2. Team #2
8.1.3.2.3. Team #3
9. B. Catalogue
9.1. Optimise Action in a System
9.1.1. "Optimise"
9.1.1.1. "Pattern, Prediction, Prescription"
9.1.1.1.1. Illustration: The Analytics Curve
9.1.2. "System"
9.1.2.1. Human
9.1.2.1.1. Personalisation
9.1.2.2. Machine
9.1.2.2.1. Operational Optimisation
9.1.2.3. Mixed
9.1.2.3.1. Economics and Finance
9.1.2.3.2. Sports
9.1.2.3.3. Commercial
9.1.2.3.4. Environmental
9.1.3. "Action"
9.1.3.1. Decision Support
9.1.3.1.1. Example: Dashboards
9.1.3.2. System Configurations
9.1.3.2.1. Example: Uber
9.1.3.2.2. Example: IFTTT
9.1.3.3. Autonomous Systems
9.1.3.3.1. Example: Level 5 Autonomous Vehicles
9.1.3.3.2. Example: Autonomous Service Bots
9.1.3.3.3. Example: Intelligence Process Automation
10. A. Use Cases
10.1. A canvas technique for design and communication...
10.1.1. Nature(s)
10.1.1.1. Type 1. A strategic narrative...
10.1.1.1.1. Play to Win
10.1.1.2. Type 2. A process architecture...
10.1.1.2.1. Uber Use Case Diagram
10.1.1.3. Type 3. A domains checklist...
10.1.1.3.1. Big Data Management Canvas
10.1.2. Recommended: Use Case Template
10.1.3. Tasks
10.1.3.1. Translate the German Rail example onto a "Use Case" template...
10.1.3.2. Team Boards
10.1.3.2.1. Team #1
10.1.3.2.2. Team #2
10.1.3.2.3. Team #3