1. CI/CD/Automation
1.1. CloudFormation
1.1.1. Stack
1.1.1.1. Stack Updating
1.1.1.2. Stack Creation
1.1.1.2.1. Upload Template to S3
1.1.1.2.2. Syntax Check
1.1.1.2.3. Stack Name
1.1.1.2.4. Parameter Verification
1.1.1.2.5. Processing the template and Stack creation
1.1.1.2.6. Resource Ordering
1.1.1.2.7. Stack Completion or Rollback
1.1.1.3. Stack Nesting
1.1.1.3.1. Why
1.1.2. Template
1.1.2.1. Anatomy
1.1.2.1.1. Paramters
1.1.2.1.2. Mappings
1.1.2.1.3. Conditions
1.1.2.1.4. Recources
1.1.2.1.5. Outputs
1.1.2.2. Intrinsic Functions
1.1.2.2.1. Base64
1.1.2.2.2. FindInMap
1.1.2.2.3. GetAtt
1.1.2.2.4. GetAZs
1.1.2.2.5. Join
1.1.2.2.6. Select
1.1.2.2.7. Ref
1.1.2.2.8. Condition Functions
1.1.2.3. DependOn
1.1.2.4. DeletionPolicy
1.1.2.4.1. Specifies what happen with every resource during stack deleting
1.1.2.4.2. Delete
1.1.2.4.3. Retain
1.1.2.4.4. Snapshot
1.1.2.4.5. Default is Delete
1.1.2.5. Wait Conditions & Handlers
1.1.2.5.1. Wait conditions and handlers are two related components
1.1.2.5.2. Wait condition handler is cloudformation resource that doesn't have properties but it generates signed URL, which can be used to communicate SUCCESS or FALIRE
1.1.2.5.3. Wait condition generally has four components:
1.1.2.6. Custom Resources
1.1.2.6.1. backed by
1.1.2.6.2. Type = Custom::*
1.1.2.6.3. ServiceToken: Ref: LambdaFunction or SNS topic
1.1.2.6.4. Request
1.1.2.6.5. Respond
1.1.2.7. Creation Policy
1.1.2.7.1. Can ONLY be used with EC2 instances and Autoscaling groups (currently)
1.1.2.7.2. Tow Main Components
1.1.2.7.3. When number of signals equal or more than Count in Creation policy that resource is marked as complete
1.1.3. Stack Policy
1.1.3.1. govern what can be changed and by who
1.1.3.2. can't be deleted after creation; only can be changed
1.1.3.3. Stack Policy
1.1.3.3.1. Can prevent resources update
1.1.3.3.2. Evaluated first before changing resources
1.1.3.3.3. By default, absence of stack policy, allows any updates
1.1.3.3.4. Once stack policy is applied it can not be removed
1.1.3.3.5. Once policy was applied, by default ALL resources are protected, Update:* is denied
1.1.3.3.6. To remove Default protection you need to update the policy with explicit "Allow" for one/more/all resources
1.1.3.3.7. Update:*
1.1.3.3.8. Update Impact
1.2. OpsWorks
1.2.1. Two Part
1.2.1.1. CHEF Agent
1.2.1.1.1. Configuration of Machines
1.2.1.2. OpsWork Automation Engine
1.2.1.2.1. Create, Update, Delete various AWS Infrastructures Components
1.2.1.2.2. Handling Load Balancing
1.2.1.2.3. AutoScaling
1.2.1.2.4. AutoHealing
1.2.1.2.5. LifeCycle Events
1.2.2. Chef
1.2.2.1. Cookbooks
1.2.2.2. Recepies
1.2.2.3. OpsWorks and Chef are declarative desired state engines
1.2.3. Instances require INTERNET access (default vpc, public subnets or private subnets with nat)
1.2.4. Linux and Windows machines can NOT be mixed
1.2.5. changes os etc apply to new instances ONLY
1.2.6. RDS instance can ONLY be associated with ONLY one Stack
1.2.7. Stack Clone Operation Does NOT clone RDS instances
1.2.8. Specific Resources
1.2.8.1. EIP
1.2.8.2. RDS
1.2.8.3. Volumes
1.2.9. Layers
1.2.9.1. Opswork
1.2.9.1.1. Traditional Layer
1.2.9.2. ECS
1.2.9.2.1. Integrate with ECS cluster
1.2.9.3. RDS
1.2.10. Events
1.2.10.1. Setup
1.2.10.1.1. instance has been created
1.2.10.2. Configure
1.2.10.2.1. enters or leaves online; associated or disassociated EIP; attach or detach ELB to the layer on all instances in the stack
1.2.10.3. Deploy
1.2.10.3.1. when you run deploy command on the instance
1.2.10.4. Undeploy
1.2.10.4.1. occur when you deleted app on the layer or run undeploy
1.2.10.5. Shutdown
1.2.10.5.1. when instance shutdown before instance is terminated
1.2.11. Instances
1.2.11.1. Stack and Layer influence on Instances
1.2.11.2. Stack has default OS for instances
1.2.11.3. Layers contains recipes, general, network ,disc configs
1.2.11.4. Types
1.2.11.4.1. 24x7
1.2.11.4.2. Time based instances
1.2.11.4.3. Load based instances
1.2.11.5. parameters that can be overwritten
1.2.11.5.1. Subnet Id
1.2.11.5.2. SSH Key
1.2.11.5.3. os
1.2.11.5.4. root device type
1.2.11.5.5. volume type
1.2.11.6. Instance Auto Healing
1.2.11.6.1. Each Opswork instance has Opswork Agent
1.2.11.6.2. Heartbeat style health check
1.2.12. Applications
1.2.12.1. Creating
1.2.12.1.1. App Name
1.2.12.1.2. Document Root
1.2.12.1.3. Data source
1.2.12.1.4. App source
1.2.12.1.5. Env Variables
1.2.12.1.6. Domain Name
1.2.12.1.7. SSL Settings
1.2.12.2. Deployment
1.2.12.2.1. it executes the Deployment of App against instances by Command
1.2.12.2.2. Passed to command is application id
1.2.12.2.3. App Parameters are passed to Chef env
1.2.12.2.4. Deploy recipe access to app source and pull
1.2.12.2.5. 5 version or app are maintained
1.2.12.3. Deployment Commands
1.2.12.3.1. Creates application deployment
1.2.12.3.2. stack level command to be executed against stack
1.2.12.3.3. Commands
1.2.13. Bershelf
1.2.13.1. was provided in Opsworks/Chef 11.10
1.2.14. Databags
1.2.14.1. is a global JSON objects accessible from within CHEF framework
1.2.14.2. There are multiple databags, including: STACK, LAYER, APP, INSTANCE
1.2.14.3. Data is accessed through Chef `data_bag_item` & `search` methods
1.2.15. events can be run manually
1.3. Elastic Beanstalk
1.3.1. deploy, monitor and scale an application
1.3.2. focus towards infrastructure, focusing on components and performance - not configuration and specifications
1.3.3. it attempts to remove or simplify infrastructure management, allowing application to deployed into infrastructure environment easily
1.3.4. components
1.3.4.1. Applications
1.3.4.1.1. your entire application is one EB application OR
1.3.4.1.2. each logical component of your application is EB application
1.3.4.1.3. each application is providing own URL
1.3.4.2. Environments
1.3.4.2.1. Each application can have several environments
1.3.4.2.2. Environment is either single instance or scalable
1.3.4.2.3. types
1.3.4.2.4. App URL can swaped between environment
1.3.4.3. Versions
1.3.4.3.1. app can have many versions
1.3.4.3.2. zip file
1.3.4.3.3. version of app can be deployed in environments
1.3.5. Configuration
1.3.5.1. Web Tier
1.3.5.1.1. Scaling
1.3.5.1.2. Instances
1.3.5.1.3. Notifications
1.3.5.1.4. Software Configuration
1.3.5.1.5. Updates and Deployments
1.3.5.1.6. Health
1.3.5.2. Network Tier
1.3.5.3. Date Tier
1.3.5.3.1. Types
1.3.5.3.2. one db can be used in only one EB
1.3.5.3.3. Cloning EB doesn't clone DB
1.3.5.3.4. DB can be moved to other EB
1.3.6. Supported Environments
1.3.6.1. NodeJS
1.3.6.2. PHP
1.3.6.3. Python
1.3.6.4. Ruby
1.3.6.5. Go
1.3.6.6. Java
1.3.6.7. Tomcat
1.3.6.8. Microsoft IIS
1.3.6.9. Docker
1.3.6.9.1. Glassfish
1.3.6.9.2. Go
1.3.6.9.3. Python
1.3.6.9.4. Custom
1.3.7. ebextantions
1.3.7.1. it is a folder .ebextations
1.3.7.2. it allows granular configuration of EB environment and customisation resources that it has (EC2, ELB and other)
1.3.7.3. YAML format
1.3.7.4. ended by .config
1.3.7.5. Files contain key sections
1.3.7.5.1. option_settings
1.3.7.5.2. recources
1.3.7.5.3. other
1.3.7.6. files will be processed in alphabetical order
1.3.7.7. leader_instance
1.3.7.7.1. leader instance picked up during deployment and after deployment it becomes general instance
1.3.7.7.2. to make sure that some commands were executed once
1.3.8. EB Docker
1.3.8.1. Application Source Bundle
1.3.8.1.1. Files
2. DevOps Core Concept
2.1. CI & CD
2.1.1. Continuous Integration
2.1.1.1. is a process of automating regular codecommit followed through Build and Test processes designed to highlight integration problem
2.1.2. Continuous Deployment
2.1.2.1. takes the form of workflow based process which accepts tested software build payload from CI server
2.2. Deployment
2.2.1. Single Target Deployment
2.2.2. All at once Deployment
2.2.3. Minimum In Service Style Deployment
2.2.4. Rolling Deployment
2.2.5. Blue/Green Deployment
2.3. Testing
2.3.1. A/B Testing
2.3.1.1. is sending percentage traffic to Blue Environment and Percent of traffic to Green
2.4. Bootstraping
2.5. Immutable Architecture
2.5.1. means "Don't diagnose or Fix, throw away and re-create
3. Monitoring/Metric/Logging
3.1. CloudWatch
3.1.1. Metric Gathering Service
3.1.1.1. 2 weeks is the longest period of time for storring metrics
3.1.1.1.1. CloudWatch launched extended retention of metrics in November 1, 2016. The feature enabled storage of all metrics for customers from the previous 14 days to 15 months. CloudWatch Metrics now supports the following three retention schedules: 1 minute datapoints are available for 15 days 5 minute datapoints are available for 63 days 1 hour datapoints are available for 455 days
3.1.1.2. Metrics fall under different namespaces
3.1.1.3. You can aggregate data accros
3.1.1.3.1. ec2
3.1.1.4. Custom Metrics
3.1.1.4.1. can use custom namespace
3.1.2. Monitoring/Alert Service
3.1.2.1. Alarms
3.1.2.1.1. alarm period must be equal or greater than metric frequency
3.1.2.1.2. alarm can't invoke actions because it is in state. Change of state can invoke actions
3.1.2.1.3. alarm action must be in the same region as alarm
3.1.2.1.4. AWS resources doesn't send metric data under certain conditions. For example, if ebs volume is not attached to ec2 instance it won't send data to cloudwatch
3.1.2.1.5. You can create alarm before metric
3.1.2.2. Alarm State
3.1.2.2.1. OK
3.1.2.2.2. ALARM
3.1.2.2.3. INSUFFICEANT_DATA
3.1.2.3. cli
3.1.2.3.1. mon-put-metric-alarm
3.1.2.3.2. mon-[enable/disable]-alarm
3.1.2.3.3. mon-describe-alarms
3.1.3. Graphing Service
3.2. CloudTrail
3.2.1. the service records all AWS API request that made in your account and delivers in Log to you
3.2.2. Logs contain
3.2.2.1. identity of who made the request
3.2.2.2. time of call
3.2.2.3. The source IP of the call
3.2.2.4. request parameters
3.2.2.5. responds elements returned by AWS
3.2.3. Purposes
3.2.3.1. Enable security analysis
3.2.3.2. Track changes in your account
3.2.3.3. provide compliance auditing
3.2.4. Two types of trail
3.2.4.1. All region to one s3 bucket
3.2.4.2. One Region to one s3 bucket
3.2.5. Storage
3.2.5.1. S3 Server Side Encryption
3.2.5.2. Store log as long as you want or apply archiving and life cycle policies
3.2.5.3. logs are delivering within 15 min
3.2.5.4. New log file are published every 5 min
3.2.6. Notes
3.2.6.1. Aws Cloudtrail records API request made by a user or on behalf of the user by AWS service
3.2.6.2. There is 'invokedBy' field in logs
3.3. CloudWatch Logs
3.3.1. Agent for EC2
3.3.1.1. Ubuntu
3.3.1.2. Amazon OS
3.3.1.3. Windows
3.3.2. Main Purposes
3.3.2.1. Monitor logs from EC2 instances in real time
3.3.2.2. Monitor CloudTrail logged events
3.3.2.3. Archive log data
3.3.3. Terminology
3.3.3.1. Log Event
3.3.3.1.1. Timestamp and message
3.3.3.2. Log stream
3.3.3.2.1. it is the sequence log events that shared single event source
3.3.3.3. Log Group
3.3.3.3.1. it is a group of log stream that shared retention period and monitoring and access controle settings
3.3.3.4. Metric Filters
3.3.3.4.1. how service would extract metrics from events
3.3.3.5. Retention Settings
3.3.4. Filter
3.3.4.1. will be applied ONLY to data that was got after creating Filter
3.3.4.2. it will return first 50 result
3.3.5. Subscription Filter
3.3.5.1. Amazon Kinesis Stream
3.3.5.2. AWS Lambda
3.3.5.3. Amazon Kinesis Firehose
3.4. CloudWatch Events
3.4.1. it is a similar service as CloudTrail but more faster
3.4.2. targets
3.4.2.1. Lambda
3.4.2.2. Kinesis Stream
3.4.2.3. SNS topics
3.4.2.4. built-in targets
3.4.2.5. can have multiple targets
4. Security/Governance/Validation
4.1. Delegation and Federation
4.1.1. Federation
4.1.1.1. allows users from other Identity Provider (IdP) to access to you aws system
4.1.1.2. Two Types
4.1.1.2.1. Corporate/Enterprise Identity Federation
4.1.1.2.2. Web or Social Identity Federation
4.1.2. Roles
4.1.2.1. is an object that contains two policy documents
4.1.2.1.1. TRUST policy
4.1.2.1.2. ACCESS policy
4.1.3. Sessions
4.1.3.1. Session is an set or temporary credentials
4.1.3.2. secret and access key with an expiration
4.1.3.3. you obtain them using Security Token Service STS
4.1.3.3.1. sts:AssumeRole
4.1.3.3.2. sts:AssumeRoleWithSAML
4.1.3.3.3. sts:AssumeRoleWithWebIdentity
4.1.3.4. It may or may not involve Congnito - depending it is web federation
4.1.4. Corporate/Enterprise Identity Federation
4.1.4.1. Types or Request
4.1.4.1.1. Console - Assume Role
4.1.4.1.2. API - GetFederateToken
4.1.4.1.3. Console - AssumeRoleWithSAML
4.1.4.2. Federation Proxy has sts:assumerole permissions
4.1.4.3. respond
4.1.4.3.1. AccessKey
4.1.4.3.2. SecretKey
4.1.4.3.3. Token Id
4.1.4.3.4. ExpirationDate
4.1.5. Web Identity Federation
4.1.5.1. Cognito
4.1.5.1.1. can orchestrate unauthenticated identities
4.1.5.1.2. can merge unauthenticated identity to authenticated if both are provided
4.1.5.1.3. can merge multiple identities like fasebook, twiter ..
4.1.5.1.4. when Id is merged - any sync data is merged
4.1.5.1.5. Types:
5. High Availability & Elasticity
5.1. EC2 AutoScaling
5.1.1. Lifecycle
5.1.1.1. Lifecycle Start
5.1.1.1.1. When you/Autoscaling Group launch instance
5.1.1.2. Lifecycle End
5.1.1.2.1. When you/autoscaling group terminate instance
5.1.1.3. If Scale In or fail health check -> instance will be terminated
5.1.1.4. Enter StandBy
5.1.1.4.1. remove instance from in service and troubleshout
5.1.1.4.2. instance is still managed by autoscaling group
5.1.1.5. Detach
5.1.1.5.1. instance will be removed from AG and still running
5.1.2. Lifecycle Hooks
5.1.2.1. EC2_INSTANCE_LAUNCHING
5.1.2.1.1. for example install new app during launcing
5.1.2.2. EC2_INSTANCE_TERMINATING
5.1.2.2.1. for example copy logs before terminating
5.1.2.3. work process
5.1.2.3.1. 1) AG responds scale out event by launching new instance
5.1.2.3.2. 2) AG puts the instance into Pending:Wait state
5.1.2.3.3. 3) AG sends a message to the notification target defined for the hook
5.1.2.3.4. 4) wait until you tell it continue or timeout ends
5.1.2.3.5. 5) you can perform any action, eg install software
5.1.2.3.6. 6) By default, the instance wait for an hour and will change the state to Pending:Proceed, then it will moved to inService state
5.1.2.3.7. If you need more time you can restart timeout period or change timeout
5.1.2.4. Notes:
5.1.2.4.1. You can change heartbeat timeout or you can set the parameter when you creates lifecycle hooks using `heartbeat-timeout` parameter
5.1.2.4.2. You can call `complete-lifecycle-action`
5.1.2.4.3. `record-lifecycle-action-heartbeat` to add more time to timeout
5.1.2.4.4. 48 hours is maximum that you can keep the server in wait condition
5.1.2.4.5. When Spot Instance terminates you must still complete the lifecycle actions
5.1.2.5. Cooldown
5.1.2.5.1. start when instance enter to inservice state
5.1.2.6. Result Response of instance
5.1.2.6.1. Abandon
5.1.2.6.2. Continue
5.1.3. Launch Configuration
5.1.3.1. Note
5.1.3.1.1. When you create AG you MUST specify LG
5.1.3.1.2. One LG can be associated with Many AG
5.1.3.1.3. One AG can have only One LG
5.1.3.1.4. you can NOT update LG, only create new
5.1.3.1.5. If you changed LC in AG -> new instance will use new LC, old instances will NOT recreated
5.1.3.2. Spot Instances
5.1.3.2.1. You CAN use lifecycle hooks with spot instances
5.1.3.2.2. LC for on-demand and spot instance are different
5.1.3.2.3. set bid price in you LC
5.1.3.2.4. If you want to change bid price you need to create new LC
5.1.3.2.5. If instance was terminated it will try to launch new one
5.1.3.2.6. If your price is below the spot price AG will wait
5.1.3.3. LC from EC2 instance
5.1.3.3.1. you can create LC from running EC2 instance
5.1.3.3.2. some properties of Ec2 instances are not supported AG
5.1.3.3.3. Diff in creating LC from scratch and from EC2 instance
5.1.3.4. Self Healing
5.2. RDS
5.2.1. Engines
5.2.1.1. MySQL
5.2.1.2. MariaDB
5.2.1.3. PostgreSQL
5.2.1.4. MSSQL
5.2.1.4.1. Storage
5.2.1.5. Oracle
5.2.1.6. AuroraDB
5.2.2. Managed Administration
5.2.2.1. RDS
5.2.2.1.1. Provisioning Infrastructure
5.2.2.1.2. Install Software
5.2.2.1.3. Automatic Backups
5.2.2.1.4. Automatic Patching
5.2.2.1.5. Synchronous Data Replication
5.2.2.1.6. Automatic Failover
5.2.2.2. You
5.2.2.2.1. Settings
5.2.2.2.2. Schema
5.2.2.2.3. Performance Tuning
5.2.3. Scaling
5.2.3.1. Vertical Scale/ Scale In
5.2.3.1.1. Compute
5.2.3.1.2. Storage
5.2.3.2. Read Scale Out
5.2.3.2.1. Up to 5 Read Replicas
5.2.3.2.2. Read Replica can be promoted to Master
5.2.3.2.3. can create Read Replica of Read Replica
5.2.3.3. Sharding
5.2.3.3.1. Split you tables across multiple DBs
5.2.3.3.2. Split Tables that aren't joined by queries
5.3. Aurora
5.3.1. you can encrypt unencrypted db you need to create new
5.3.2. compatible with MySQL 5.6
5.3.3. Storage
5.3.3.1. Fault Tolerant and Auto Healing
5.3.3.2. Disk fails are repeated in background
5.3.3.3. Detects database crashes and restartes
5.3.3.4. No crash recovery or cache rebuilding
5.3.3.5. Automatic failover to one of up to 15 read replicas
5.3.3.6. Storage autoscaling from 10Gb up to 64 Tb
5.3.3.6.1. without any distruption
5.3.4. Backups
5.3.4.1. Automatic, Continuous and incremental backups
5.3.4.2. Point in time recovery within a second
5.3.4.3. retention period Up to 35 Days
5.3.4.4. Stored in S3
5.3.4.5. no impact on database performance
5.3.5. Snapshots
5.3.5.1. Stored to S3
5.3.5.2. Keep until you delete them
5.3.5.3. Incremental
5.3.6. Data failures
5.3.6.1. 6 copies of your data
5.3.6.2. 3 availability zones
5.3.6.3. restore in healthy AZ
5.3.6.4. restore from PIT and snapshotes
5.3.7. Fault Tolerance
5.3.7.1. data divides into 10 Gb segments across many disks
5.3.7.2. Transparently handle data loss
5.3.7.3. Can lose up to 2 copies of you data without effecting write
5.3.7.4. Can lose up to 3 copies of data without effecting reading
5.3.7.5. All storages are auto healing
5.3.8. Replicas
5.3.8.1. Aurora Replica
5.3.8.1.1. Share same underline volume that the primary instance
5.3.8.1.2. Updates made by primary instance immediately visible across all replicas
5.3.8.1.3. up to 15
5.3.8.1.4. Performance impact on primary is low
5.3.8.1.5. Replica can failover the target without data loss
5.3.8.2. MySQL Replica
5.3.8.2.1. replay by transaction
5.3.8.2.2. up to 5
5.3.8.2.3. high performance impact on primary
5.3.8.2.4. Replica can failover with potential minutes data loss
5.3.9. Security
5.3.9.1. all instances must be created in VPC
5.3.9.2. SSL
5.3.9.3. KMS data at rest
5.3.9.4. encrypted storage
5.3.9.5. encrypted backups
5.3.9.6. encrypted snapshote
5.3.9.7. encrypted replicas
5.3.9.8. you can not encrypt unencrypted db
5.4. DynamoDB
5.4.1. Primer
5.4.1.1. NoSQL
5.4.1.2. Fully managed
5.4.1.3. Predictable fully manageable performace
5.4.1.4. seamless scalability
5.4.1.5. no visible servers
5.4.1.6. no practical storage limitations
5.4.1.7. Fully resilient and highly available
5.4.1.8. Performance scale is linear way
5.4.1.9. Fully integrated with IAM
5.4.1.10. Structure
5.4.1.10.1. Tables
5.4.1.10.2. Items (Rows)
5.4.1.10.3. Atributes
5.4.1.10.4. Hash Key/Partition Key
5.4.1.10.5. Sort Key/Range Key
5.4.1.11. Data Types
5.4.1.11.1. String
5.4.1.11.2. Number
5.4.1.11.3. Binary
5.4.1.11.4. Boolen
5.4.1.11.5. Null
5.4.1.11.6. Document JSON (List or Map)
5.4.1.11.7. Set (array)
5.4.1.12. Write Capacity Unit (WCU)
5.4.1.12.1. Number of 1KB blocks per second
5.4.1.12.2. Write at least to 2 location to finish write
5.4.1.13. Read Capacity Unit (RCU)
5.4.1.13.1. Number of 4 KB blocks per second
5.4.1.13.2. Read from only 1 location
5.4.1.13.3. Eventually Consistent By Default
5.4.1.13.4. You can configure Immediate consistent
5.4.1.13.5. If your items are smaller than 4 KB in size, each read capacity unit will yield one strongly consistent read per second or two eventually consistent reads per second.
5.4.1.14. Unlike SQL - Schema is not part of database level
5.4.2. Partition
5.4.2.1. There is underlying storage and processing nodes of DynamoDB
5.4.2.2. 1 partition
5.4.2.2.1. 10 Gb
5.4.2.2.2. 3000 RCU
5.4.2.2.3. 1000 WCU
5.4.2.3. when > 10Gb or > 3000 RCU or > 1000 WCU
5.4.2.3.1. get new partition
5.4.2.3.2. data automatically spreaded over the time
5.4.2.4. distribution between partitions
5.4.2.4.1. based on HASH / Partition Key
5.4.2.5. Limitations
5.4.2.5.1. Partitions will automatically increase
5.4.2.5.2. Partitions don't decrease
5.4.2.5.3. Allocated WCU and RCU are splitted between partitions
5.4.2.5.4. Each partition key is limited
5.4.2.6. calculating
5.4.2.6.1. MAX((WCU/1000 +RCU/3000),Storage/10Gb) Round Up to whole number
5.4.3. GSI/LSI
5.4.3.1. Global Secondary Indexies
5.4.3.1.1. Allows to use alternative partition key
5.4.3.1.2. As with LSI, you can shoose
5.4.3.1.3. GSI doesn't share WCU and RCU and has own
5.4.3.1.4. changes written Async
5.4.3.1.5. Supports ONLY eventual consistency
5.4.3.2. Local Secondary Indexies
5.4.3.2.1. Contain
5.4.3.2.2. Any data that written to the table copied Async to any LSI
5.4.3.2.3. Shares WCU and RCU with the table
5.4.3.2.4. sparse index
5.4.3.2.5. Operations
5.4.3.2.6. Concerns about LSI
5.4.3.2.7. Supports two consistencies
5.4.4. Stream & Replication
5.4.4.1. Stream
5.4.4.1.1. it is ordered record of updates to dynamo db table
5.4.4.1.2. When stream is enabled
5.4.4.1.3. AWS guaranty that each change in the table occur ONLY once
5.4.4.1.4. All changes to the Table occur in the scream in nr realtime
5.4.4.1.5. Stream contains only Partition Keys from NOW to - 24 hours
5.4.4.1.6. can be used in
5.4.4.1.7. Stream View
5.4.4.1.8. aws dynamodb create-table \ --table-name BarkTable \ --attribute-definitions AttributeName=Username,AttributeType=S AttributeName=Timestamp,AttributeType=S \ --key-schema AttributeName=Username,KeyType=HASH AttributeName=Timestamp,KeyType=RANGE \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \ --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES
5.4.4.2. Replication
5.4.4.2.1. Stream + Kinesis
5.4.5. DeepDive
5.4.5.1. Performance
5.4.5.1.1. Write
5.4.5.1.2. Read
5.4.5.1.3. Add random prefix to partition key to distribute it through partitions
5.5. SQS
5.5.1. messages 256 kb
5.5.2. SQS Extended Library allows to handle > 256 Kb using S3
5.5.3. Ensures delivery of message at least once
5.5.4. Not Fifo
5.5.5. Messages can be retain up to 14 days
5.5.6. Long polling reduce cost of frequent polling
5.5.6.1. polling every 20 seconds if queue is empty
5.5.7. 0.5$ for every million requests plus data transfer charges
5.5.8. Use cases
5.5.8.1. prioritisation using two SQS queues
5.5.8.2. parallel processing reading from one queue different components will do different roles
5.5.8.2.1. SNS -> to many SQS
5.6. Kinesis
5.6.1. Stream
5.6.1.1. Collect and process large streams of data in real time
5.6.1.2. Create data processing application
5.6.1.2.1. Read data from the stream
5.6.1.2.2. Process records
5.6.1.3. Scenarios
5.6.1.3.1. Fast log processing
5.6.1.3.2. Realtime metric and reporting
5.6.1.3.3. Realtime data analitics
5.6.1.3.4. Complex stream processing
5.6.1.4. Benefits
5.6.1.4.1. Realtime aggregation of data
5.6.1.4.2. Loading aggregated data into data warehouses / mapreduce cluster
5.6.1.4.3. Durability and Elasticity
5.6.1.4.4. Parallel application readers
5.6.2. Analitics
5.6.2.1. The easiest way to run SQL queries against streaming data
5.6.3. Firehose
5.6.3.1. The easiest way to load streaming data to
5.6.3.1.1. S3
5.6.3.1.2. Redshift
5.6.3.2. That enables near realtime analitics
5.6.3.3. Batch size or interval to control how often data will be upload to S3 or Redshift
5.6.3.4. Firehose API or Linux agent
5.6.3.5. compression
5.6.3.6. scalable
5.6.3.7. secure - encription
5.6.3.8. monitoring through CloudWatch
5.6.3.9. Pay as you go
5.6.3.9.1. pay only for data and nothing for infrastructure
5.7. Kinesis vs SQS
5.7.1. Kinesis
5.7.1.1. Data can be processed at the same time or within 24 hours by different consumer
5.7.1.2. Data can be replayed within the time
5.7.1.3. 24 hours of retention
5.7.2. SQS
5.7.2.1. You can write to multiple queues. Then you can read from multiple source. But you can not reuse the information later
5.7.2.2. up to 24 days of retention
6. Operations
6.1. Instance Type Deepdive
6.1.1. Virtualization
6.1.1.1. Types
6.1.1.1.1. Para-Virtualisation
6.1.1.1.2. Hardware Virtualization (HVM)
6.1.1.2. AMI supports either first or second virtualisation
6.1.1.2.1. Usually Older AMI supports Para - Virtualisation
6.1.1.2.2. Using HVM gives much more wide choice of instance type and sizes
6.1.1.2.3. Some families only supports HVM (eg T2)
6.1.1.2.4. Some Old instance types don't support HVM (eg. T1)
6.1.2. Instance Types
6.1.2.1. T
6.1.2.1.1. General Burstable
6.1.2.2. M
6.1.2.2.1. General Purpose
6.1.2.3. C
6.1.2.3.1. CPU utilisation
6.1.2.4. R
6.1.2.4.1. RAM optimised
6.1.2.5. G
6.1.2.5.1. GPU, graphic
6.1.2.6. I
6.1.2.6.1. High IO instances
6.1.2.7. D
6.1.2.7.1. Dense Storage - Disk performance
6.1.3. Instance Size
6.1.4. Instance Generation
6.1.5. Instance Features
6.1.5.1. EBS optimisation
6.1.5.1.1. Dedicated bandwidth, consistency higher throughput
6.1.5.1.2. 500Mbs -4000Mbs
6.1.5.2. enhanced networking
6.1.5.2.1. AWS supported SR-IOV
6.1.5.2.2. less jitter
6.1.5.2.3. higher throughput
6.1.5.2.4. VM must have driver
6.1.5.3. Instance Store Volumes
6.1.5.4. GPU availability
6.2. EBS Deep Dives
6.2.1. Types
6.2.1.1. Magnetic
6.2.1.1.1. ~100 IOPS
6.2.1.1.2. 2-40 ms latency
6.2.1.1.3. ~10Mb throughput
6.2.1.1.4. no burst
6.2.1.2. General purpose SSD
6.2.1.2.1. 3 IOPS per Gb
6.2.1.2.2. Burst up to 3000
6.2.1.2.3. up to 160 Mb/s troughput
6.2.1.2.4. Larger volumes can scale to 10000 IOPS
6.2.1.2.5. IOPS limits assume 256 KB block size
6.2.1.2.6. IOPS Burst Pool starts with 5400 000
6.2.1.3. Procision IOPS
6.2.1.3.1. max IOPS 20 000
6.2.1.3.2. up to 320 Mb/s throughtput
6.2.1.3.3. 50 IOPS for Gb of storage max
6.2.1.3.4. 99.9% performance consistency
6.2.2. Terms
6.2.2.1. Capacity
6.2.2.1.1. Amount of data in Gb
6.2.2.2. Troughput
6.2.2.2.1. Data throughput Mb/s for read/write operations
6.2.2.3. Block Size
6.2.2.3.1. Size of each write or read in KB
6.2.2.4. IOPS
6.2.2.4.1. Number read and write operation per second
6.2.2.5. Latency
6.2.2.5.1. Delay between read/write request and completion ms
6.2.3. Performance element
6.2.3.1. Instance
6.2.3.2. IO
6.2.3.2.1. You can not maintain MAX IOPS and MAX throughput in the same time
6.2.3.3. network
6.2.3.4. ebs volumes
6.2.4. IOPS vs Throughput
6.2.4.1. eg, you expect 3000 IOPS and 160Mbs
6.2.4.1.1. 32KB * 3000 IOPS = 96Mbs
6.2.4.1.2. 256KB *750 IOPS =192Mbs
6.2.4.1.3. 256KB * 3000 IOPS = 768 Mbs that beyond 160 MBs MAX
6.2.4.2. Throughput = IOPS * Block Size
6.2.4.3. you need to optimise Block Size
6.2.5. RAID
6.2.5.1. We need more
6.2.5.1.1. Large EBS optimised instance can deliver 32000 IOPS @ 16k 500 Mbs
6.2.5.1.2. Max IOPS 48000 IOPS @ 16k 10Gb
6.2.5.2. RAID 0
6.2.5.3. LVM Stripe
6.2.5.4. You need to freeze filesystem during snapshotting
6.2.6. Burst Pool
6.2.6.1. MAX Burst is 3000 IOPS
6.2.6.2. Start Burst pool 5400 000
6.2.6.3. you don't need to worry about provision large volume to get better performance
6.2.7. Tips
6.2.7.1. Pre-warming is not longer required for new EBS volumes
6.2.7.2. Volumes created from snapshots are Lazy restored from S3 and require PrepWarming to get full performance
6.2.7.3. First snapshot - full copy, next are incremental. Often snapshots - cheaper and faster