1. Reporting and visualization
2. Model deployment
3. analytics platform
3.1. data movement
3.2. data processing
3.3. ingestion
3.4. transformation
3.5. real-time event routing
3.6. report building
4. component
4.1. Powerbi
4.2. Databases
4.2.1. Azure SQL database
4.2.2. cosmos DB
4.2.3. Databricks
4.2.4. snowflake
4.2.5. Fabric SQL DB
4.3. Data factory
4.3.1. dataflows
4.3.1.1. data ingestion
4.3.1.1.1. similar to power query
4.3.2. pipelines
4.3.2.1. data transformation activities
4.3.2.1.1. Copy data
4.3.2.1.2. dataflow, gen2
4.3.2.1.3. Notebook
4.3.2.1.4. Store procedure
4.3.2.1.5. delete data
4.3.2.2. control flow activities
4.3.2.2.1. loops
4.3.2.2.2. conditional branching
4.3.2.2.3. variable and parameter
4.3.3. copy jobs
4.3.4. apache airflow
4.4. Industry Solutions
4.5. Real-time intelligence
4.6. Data engineering
4.6.1. lakehouse (structure and unstuctured)
4.6.2. Apache spark
4.6.2.1. for big data
4.6.2.2. use python, R, or scala
4.6.2.3. programmatic data loading and transformations
4.6.2.3.1. sparkjob
4.6.2.4. ML
4.6.2.5. Analyze text, images
4.6.3. Notebook
4.6.4. Data pipeline
5. Data ingestion
5.1. connect and ingest
5.1.1. CRM
5.1.2. e-commerce platforms
5.2. copy jobs
6. Data transformation
6.1. Data engineering
6.2. Dataflow/pipeline (less code)
7. AI model devel and train
8. Practical example
8.1. create a lakehouse in data engineering
8.1.1. create a Dataflow (powerquery)
8.1.1.1. add data destination
8.1.1.1.1. add a dataflow to pipeline
8.2. create a lakehouse
8.2.1. create a pipeline
8.2.1.1. copy data
8.2.1.2. notebook
8.2.1.2.1. pyspark
8.3. apache sparks
8.3.1. environment
8.3.1.1. spark runtime-->version of spark, delta lake, python
8.3.1.2. build-in libraries
8.3.1.3. python libraries
8.3.2. Native execution engine
8.3.2.1. run spark operations diectly on lakehouse infrastracture, works well for Parquet or delta file formats
8.3.3. High concurrency mode
8.3.3.1. share spark sessions across multiple concurrent users or processes
8.3.4. MLFlow
8.3.4.1. ML training and model deployment management
8.3.5. spark catalog
8.3.5.1. metastore for views and tables