1. Big Data
1.1. The 5 Vs of Big Data
1.1.1. Volume
1.1.2. Velocity
1.1.3. Variety
1.1.4. Veracity
1.1.5. Value
1.2. Technologies Used in Big Data
1.2.1. Distributed Computing
1.2.1.1. Hadoop
1.2.1.2. Apache Spark
1.2.2. Data Warehousing
1.2.2.1. Snowflake
1.2.2.2. Google BigQuery,
1.2.3. Data Lakes:
1.2.3.1. AWS S3
1.2.3.2. Azure Data Lake
1.3. Big Data Analytics Techniques
1.3.1. Descriptive Analytics
1.3.1.1. Example
1.3.1.1.1. A company analyzing customer sales data to understand past purchasing behavior.
1.3.1.2. Methods
1.3.1.2.1. Data aggregation, summarization, and visualization techniques to analyze historical data.
1.3.2. Diagnostic Analytics
1.3.2.1. Example
1.3.2.1.1. Investigating why sales dropped in a particular region during a specific period..
1.3.2.2. Methods
1.3.2.2.1. Statistical analysis, correlation analysis, and trend analysis to identify causes behind outcomes.
1.3.3. Predictive Analytics
1.3.3.1. Example
1.3.3.1.1. Predicting customer churn using historical data on customer behaviors.
1.3.3.2. Methods
1.3.3.2.1. Machine learning models, regression analysis, and time series forecasting to predict future trends based on historical data.
1.3.4. Prescriptive Analytics
1.3.4.1. Example
1.3.4.1.1. Recommending marketing strategies to maximize customer engagement based on customer behavior analysis.
1.3.4.2. Methods
1.3.4.2.1. Optimization, simulation, and decision analysis to suggest the best course of action.
1.3.5. Real-Time Analytics:
1.3.5.1. Example
1.3.5.1.1. Monitoring social media in real-time for sentiment analysis or detecting fraudulent transactions as they occur.
1.3.5.2. Methods
1.3.5.2.1. Real-time data processing frameworks like Apache Kafka and Apache Flink.
1.4. Challenges of Big Data
1.4.1. Data Privacy and Security
1.4.1.1. Ensuring that sensitive information (like personal data) is protected and complying with regulations such as GDPR.
1.4.2. Data Integration
1.4.2.1. Integrating data from multiple sources with different formats and structures can be complex.
1.4.3. Data Quality
1.4.3.1. Ensuring that the data is accurate, consistent, and reliable for meaningful analysis.
1.4.4. Scalability
1.4.4.1. As data grows, maintaining systems that can scale efficiently and handle massive amounts of data is crucial.
1.4.5. Cost
1.4.5.1. Storing and processing large datasets can be costly in terms of infrastructure, storage, and computing power.
1.4.6. Skills Shortage
1.4.6.1. The complexity of Big Data requires skilled professionals such as data engineers, data scientists, and analysts, who are often in high demand.
1.5. Applications of Big Data
1.5.1. Healthcare
1.5.1.1. Predictive analytics for disease outbreaks, personalized medicine, improving patient care, and optimizing hospital operations.
1.5.2. E-Commerce
1.5.2.1. Personalized recommendations, customer segmentation, inventory management, and dynamic pricing based on user behavior.
1.5.3. Finance
1.5.3.1. Fraud detection, credit scoring, risk management, and high-frequency trading.
1.5.4. Smart Cities
1.5.4.1. Traffic management, energy optimization, and urban planning through IoT data analysis.
1.5.5. Manufacturing
1.5.5.1. Predictive maintenance, supply chain optimization, and quality control based on sensor data.
1.5.6. Social Media
1.5.6.1. Sentiment analysis, trend detection, and content personalization.
1.6. Big Data Tools and Platforms
1.6.1. Apache Hadoop
1.6.1.1. A framework for distributed storage and processing of large datasets.
1.6.2. Apache Spark
1.6.2.1. A fast, in-memory data processing engine.
1.6.3. Google BigQuery
1.6.3.1. A fully managed cloud data warehouse for large-scale data analytics.
1.6.4. Tableau
1.6.4.1. A data visualization tool used to explore and present Big Data insights.
1.6.5. Apache Kafka
1.6.5.1. A distributed streaming platform for building real-time data pipelines.
1.6.6. Power BI
1.6.6.1. A Microsoft tool for business analytics and visualization.
2. Databases
2.1. SQL
2.1.1. Query Data
2.1.1.1. Select
2.1.1.2. From
2.1.1.3. Where
2.1.1.4. Order By
2.1.1.5. Group By
2.1.1.6. Having
2.1.1.7. Distinct
2.1.1.8. Limıt Or Top
2.1.1.9. Query Order & execution
2.1.2. Data Definition (DDL)
2.1.2.1. Create
2.1.2.2. Alter
2.1.2.3. Drop
2.1.3. Data Manipulation (DML)
2.1.3.1. Insert
2.1.3.2. Update
2.1.3.3. Delete
2.1.4. Filtering Data
2.1.4.1. Comparaison operators
2.1.4.2. Logical operators
2.1.4.2.1. AND
2.1.4.2.2. OR
2.1.4.2.3. NOT
2.1.4.3. BETWEEN
2.1.4.4. IN
2.1.4.5. LIKE
2.1.5. Combining Data
2.1.5.1. Joins
2.1.5.1.1. basics joins
2.1.5.1.2. How to choose the right join
2.1.5.2. Set
2.1.5.2.1. Union
2.1.5.2.2. Union All
2.1.5.2.3. Except
2.1.5.2.4. Intercept
3. Common Challenges in Machine Learning
3.1. Data Quality
3.1.1. Ensuring the data is clean, accurate, and representative of the problem at hand.
3.2. Bias
3.2.1. Ensuring the model does not reinforce harmful biases present in the training data.
3.3. Interpretability
3.3.1. Making complex models (like deep neural networks) understandable to humans.
3.4. Scalability
3.4.1. Ensuring models can handle large datasets efficiently.
4. Maths
4.1. Statistics
4.1.1. Inferences
4.1.1.1. Parameter Estimation
4.1.1.1.1. Point estimation
4.1.1.1.2. Interval estimation:
4.1.1.2. Hypothesis Testing
4.1.1.2.1. Hypotheses to Test
4.1.1.2.2. Types of Hypothesis Tests
4.1.1.3. Regression and Prediction
4.1.1.3.1. Simple linear regression
4.1.1.3.2. Multiple linear regression:
4.1.1.3.3. Logistic regression
4.1.1.4. Bayesian Inference
4.1.1.4.1. Bayes' Theorem
4.2. Probability
4.2.1. Basic Concepts of Probability
4.2.1.1. Experiment
4.2.1.1.1. An action or process that results in one of several possible outcomes (e.g., rolling a die, flipping a coin).
4.2.1.2. Sample Space
4.2.1.2.1. The set of all possible outcomes of an experiment.
4.2.1.3. Event:
4.2.1.3.1. A subset of the sample space. It represents one or more outcomes of the experiment. Example: In a dice roll, the event "rolling an even number" is {2,4,6}.
4.2.2. Probability of an Event
4.2.2.1. The probability of an event is a number between 0 and 1 that represents the likelihood of the event occurring.
4.2.3. Types of Probability
4.2.3.1. Classical Probability:
4.2.3.1.1. Assumes all outcomes are equally likely.
4.2.3.2. Empirical Probability
4.2.3.2.1. Based on observed data.
4.2.3.3. Subjective Probability
4.2.3.3.1. Based on personal judgment or experience rather than data.
4.2.4. Complementary Events
4.2.5. Addition Rule
4.2.5.1. The addition rule helps calculate the probability of the union of two events, i.e., the probability that either one event or another event occurs.
4.2.6. Multiplication Rule
4.2.6.1. The multiplication rule helps calculate the probability of the intersection of two events, i.e., the probability that both events occur.
4.2.7. Conditional Probability
4.2.7.1. Conditional probability is the probability of an event occurring given that another event has already occurred. It is denoted as 𝑃(𝐴∣𝐵), which is read as "the probability of𝐴given𝐵 ".
4.2.8. Bayes' Theorem
4.2.8.1. Bayes' Theorem allows you to update probabilities based on new information. It is useful for solving problems with conditional probabilities.
4.2.9. Discrete vs. Continuous Probability
4.2.9.1. Discrete Probability
4.2.9.1.1. Deals with events that have a finite or countable number of outcomes (e.g., rolling a die, drawing a card from a deck).
4.2.9.1.2. Example
4.2.9.2. Continuous Probability
4.2.9.2.1. Deals with events that can take any value within a range (e.g., the height of a person, the time to run a race). The probability of any specific value is 0, and probabilities are measured over intervals.
4.2.9.2.2. Example
4.2.9.3. Probability Distributions
4.2.9.3.1. Continuous Probability Distribution
4.2.9.3.2. Discrete Probability Distribution
5. Programming languages
5.1. Python
5.1.1. Basics
5.1.1.1. Variables
5.1.1.2. List
5.1.1.3. Sets
5.1.1.4. Tuoles
5.1.1.5. Dictionaries
5.1.1.6. Functions
5.1.1.7. Handling Errors
5.1.1.8. Text
5.1.1.9. String
5.1.2. Libraries
5.1.2.1. Numpy
5.1.2.2. Pandas
5.1.2.3. Matplotlib
5.1.2.4. Seaborn
5.2. R
5.2.1. Basics
5.2.1.1. Vectors
5.2.1.2. Lists
5.2.1.3. DataFrames
5.2.1.4. Matrixs
5.2.1.5. Arrays
5.2.2. Libraries
5.2.2.1. dplyr
5.2.2.1.1. dplyr is a popular R package for data manipulation and transformation. It is designed to be fast, intuitive, and readable, using a verb-based syntax to perform operations on data frames and tibbles.
5.2.2.2. ggplot2
5.2.2.2.1. ggplot2 is a powerful data visualization package in R that implements the grammar of graphics, allowing you to create complex plots from data in a consistent and coherent way. It is widely used for its simplicity and flexibility, enabling you to build plots layer by layer.
5.2.2.3. Tidyr
5.2.2.3.1. tidyr is an R package designed for data tidying, i.e., reshaping and transforming data into a format that is easy to work with for analysis and visualization. It is closely related to the tidyverse ecosystem, and it provides a set of functions that help to "tidy" your data, such as converting between wide and long formats, handling missing values, and separating or uniting columns.
5.2.2.4. Shiny
5.2.2.4.1. Shiny is an R package that makes it easy to build interactive web applications directly from R. With Shiny, you can create data-driven, dynamic web apps for visualizing and exploring data, all without needing to know HTML, CSS, or JavaScript. Shiny apps are reactive, meaning they automatically update outputs based on inputs.
6. Data visualization tools
6.1. Tableau
6.2. Power bi
7. Machines learnings
7.1. Types
7.1.1. Supervised learning
7.1.1.1. Linear Regression
7.1.1.1.1. Used for predicting a continuous output based on one or more input features.
7.1.1.2. Logistic Regression
7.1.1.2.1. Used for binary classification problems (e.g., yes/no, true/false).
7.1.1.3. Decision Trees
7.1.1.3.1. A tree-like structure used for classification and regression tasks.
7.1.1.4. Support Vector Machines (SVM)
7.1.1.4.1. A tree-like structure used for classification and regression tasks.
7.1.1.5. k-Nearest Neighbors (k-NN)
7.1.1.5.1. A simple algorithm that classifies a new data point based on the majority class of its nearest neighbors in the feature space.
7.1.1.6. Random Forests
7.1.1.6.1. An ensemble learning method that builds multiple decision trees and combines their outputs to improve prediction accuracy.
7.1.2. Insupervised learning
7.1.2.1. Clustering
7.1.2.1.1. k-Means
7.1.2.1.2. Hierarchical Clustering
7.1.2.2. Principal Component Analysis (PCA)
7.1.2.2.1. A dimensionality reduction technique that transforms the data into a new set of variables (principal components) that are uncorrelated and capture most of the variance.
7.1.2.3. Autoencoders
7.1.2.3.1. Neural networks used for unsupervised learning to learn an efficient representation of data.
7.1.3. Key Machine Learning Concepts
7.1.3.1. Overfitting vs. Underfitting
7.1.3.1.1. Overfitting
7.1.3.1.2. Underfitting
7.1.3.1.3. Regularization
7.1.3.2. Training and Testing Data
7.1.3.2.1. Training Data
7.1.3.2.2. Testing Data
7.1.3.3. Cross-Validation
7.1.3.3.1. Cross-validation is a technique used to evaluate the model's performance by splitting the data into several subsets (folds) and training and testing the model on different combinations of these folds.
7.1.3.4. Loss Function
7.1.3.4.1. Mean Squared Error (MSE)
7.1.3.4.2. Cross-Entropy Loss
8. Deep Learning
8.1. Neural Networks
8.1.1. A set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns.
8.2. Convolutional Neural Networks (CNNs)
8.2.1. Used primarily in image and video recognition tasks.
8.3. Recurrent Neural Networks (RNNs)
8.3.1. Used for sequence data like time series, speech, and text.
8.4. Generative Adversarial Networks (GANs)
8.4.1. Used for generating new data that mimics the real data (e.g., creating images or video).
9. Evaluation Metrics
9.1. Classification
9.1.1. Accuracy
9.1.1.1. The proportion of correct predictions.
9.1.2. Precision
9.1.2.1. The number of true positive predictions divided by the total number of positive predictions.
9.1.3. Recall
9.1.3.1. The number of true positive predictions divided by the total number of actual positives.
9.1.4. F1 Score
9.1.4.1. The harmonic mean of precision and recall.
9.1.5. ROC Curve and AUC
9.1.5.1. Used to evaluate the performance of binary classifiers.
9.2. Regression
9.2.1. Mean Squared Error (MSE)
9.2.2. Mean Absolute Error (MSE)
9.2.3. Root Mean Absolute Error (RMSE)
10. Popular Machine Learning Frameworks and Libraries
10.1. Scikit-learn
10.1.1. A Python library that provides simple and efficient tools for data mining and machine learning, including algorithms for classification, regression, clustering, and dimensionality reduction.
10.2. TensorFlow
10.2.1. An open-source library developed by Google for deep learning applications.
10.3. Keras
10.3.1. A high-level neural networks API, written in Python and capable of running on top of TensorFlow.
10.4. PyTorch
10.4.1. An open-source machine learning library developed by Facebook for deep learning and artificial intelligence.
10.5. XGBoost
10.5.1. An optimized gradient boosting library for efficient and scalable machine learning.