Cosmos Account

Azure cosmos

Get Started. It's Free
or sign up with your email address
Cosmos Account by Mind Map: Cosmos Account

1. Integration

1.1. AzureFunction

1.1.1. Create

1.1.1.1. Trigger:CosmosDBTrigger

1.1.1.2. CosmosDBAccount

1.1.1.3. ComsmosDatabase

1.1.1.4. CosmosCollection

1.1.1.5. LeaseCollection

2. DataModeling

2.1. Embed fields when

2.1.1. Read or updated together:

2.1.2. 1:1 relationship

2.1.2.1. Customer:CustomerPassword

2.1.3. 1:Few relationship

2.1.3.1. Customer:CustomerAddresses

2.2. Reference (separate doc) when

2.2.1. Read or updated independently

2.2.2. 1:Many relationship

2.2.3. Many:Many relationship

3. Container

3.1. Physical Partitions

3.1.1. P1

3.1.1.1. Each Physical partition has max 50GB storage and 10K RU's

3.1.2. P2

3.1.3. P3

3.1.3.1. LP1

3.1.3.2. LP1

4. DataModeling

4.1. Data modeling based on query with aggregates

4.1.1. top 10 customers by the number of sales orders

4.1.1.1. he solution here is to denormalize the aggregate value in a new property, salesOrderCount, in the customer document.

4.1.1.1.1. Azure Cosmos DB supports transactions when the data sits within the same logical partition

4.2. If two entities have same partitionKey and similar access patterns then it can be stored together in single container "CustomerSalesOrder"

4.2.1. All the orders of Customer.

4.2.1.1. SELECT * FROM Order O WHERE O.customerId = "A" and o.type ="Order"

4.2.2. Customer Details

4.2.2.1. SELECT * FROM Customer C WHERE C.CustomerID= "A" and c.type="Customer"

4.2.2.1.1. { Id:'A',type:'customer',customerId:"A" } , { Id:"O1",type:"Order",customerId:"A }

4.3. ReferentialIntegrity

4.3.1. Use Change feed

5. Indexing

5.1. Read heavy

5.1.1. index seek

5.1.1.1. =, IN

5.1.2. index scan

5.1.2.1. RANGE

5.1.3. full scan

5.1.3.1. no filter

5.2. write heavy

5.2.1. IndexingMode: None to disable

5.2.2. exclude all and include very few

5.3. SDK index debugging performance and index recommendations

5.3.1. QueryRequestOptions options = new() { PopulateIndexMetrics = true }; Console.WriteLine(response.IndexMetrics);

6. Integrated cache

6.1. Enable Dedicated Gateway

6.1.1. SDK

6.1.1.1. Use gateway connString

6.1.1.2. CosmosClientOptions options = new() { ConnectionMode = ConnectionMode.Gateway };

6.1.1.3. QueryRequestOptions queryOptions = new() { ConsistencyLevel = ConsistencyLevel.Eventual }; //Or Session

6.1.1.3.1. Staleness (Expiry)

7. Region

8. Unique DNS Name

9. Capacity

9.1. Throughput provisioned

9.1.1. unlimited storage, predictable traffic, req consumed > 66 % of hours per month

9.2. Serverless

9.2.1. Max 50GB storage, unpredicted traffic,request consumed < 66% of hours per month

10. DataBase

10.1. Provision throughput

10.1.1. Autoscale

10.1.1.1. Maximum RU's

10.1.1.1.1. Charge will be 10% of the max RU when its not used

10.1.2. Manual

10.1.2.1. Fixed RU's

10.1.2.1.1. Min 400 RU's

11. Container

11.1. PartitionKeyPath

11.2. Provision Dedicated Throughput

11.2.1. Autoscale

11.2.2. Manual

11.3. Unique Key

11.4. Analytical store for Synapse Link integration

11.5. Change Feed

11.5.1. Monitored container

11.5.1.1. This container is monitored for any insert or update operations

11.5.2. Lease container

11.5.2.1. serves as a storage mechanism to manage state across multiple change feed consumers (clients)

11.5.3. Host

11.5.3.1. client application instance that listens for and reacts to changes from the change feed

11.5.3.1.1. ChangeFeedProcessor

11.5.3.1.2. ChangeFeedEstimator

11.5.4. Delegate

11.5.4.1. code within the client application that will implement business logic for each batch of changes

11.5.4.1.1. ChangesHandler<Product> changeHandlerDelegate = async ( IReadOnlyCollection<Product> changes, CancellationToken cancellationToken ) => { foreach(Product product in changes) { await Console.Out.WriteLineAsync($"Detected Operation:\t[{product.id}]\t{product.name}"); // Do something with each change } };

11.5.4.1.2. ChangesEstimationHandler changeEstimationDelegate = async ( long estimation, CancellationToken cancellationToken ) => { // Do something with the estimation };

11.6. SDK

11.6.1. CosmosClient client = new("ConnString")

11.6.1.1. db = client.CreateDatbaseIfNotExistsAsync("dbName");

11.6.1.1.1. container = client.CreateContainerIfNotExistsAsync("Name","PartitionKeyPath","RU")

11.6.2. Logging

11.6.2.1. CosmosClientBuilder b = new(new LogHander()) // : RequestHanlder =>override SendAsync // client = b.build();

12. CosmosClientOptions

13. Troubleshooting 429 Request too large exception

13.1. Azure Cosmos DB Insight

13.1.1. Request tab

13.1.1.1. Total Request by Status Code if worklaod < 5%

13.1.1.1.1. Yes

13.1.1.1.2. No

14. Status codes and meaning

14.1. Actions

14.1.1. Create a Document

14.1.1.1. 400 : Bad Req

14.1.1.1.1. The JSON body is invalid.

14.1.1.2. 403 : Forbidden

14.1.1.2.1. The operation couldn't be completed because the storage limit of the partition has been reached.

14.1.1.3. 409 : Conflict

14.1.1.3.1. The id provided for the new document has been taken by an existing document.

14.1.1.4. 413 : Entity too large

14.1.1.4.1. The document size in the request exceeded the allowable document size.

14.1.2. List documents using ReadFeed

14.1.2.1. 400 : Bad Req

14.1.2.1.1. The override set in x-ms-consistency-level is stronger than the one set during account creation. For example, if the consistency level is Session, the override can't be Strong or Bounded.

14.1.3. Get document by point read

14.1.3.1. 304 : Not modified

14.1.3.1.1. The document requested wasn't modified since the specified eTag value in the If-Match header. The service returns an empty response body.

14.1.3.2. 400 : Bad Req

14.1.3.2.1. The override set in x-ms-consistency-level is stronger than the one set during account creation. For example, if the consistency level is Session, the override can't be Strong or Bounded.

14.1.4. Patch

14.1.4.1. 412: PreCondition Failed

14.1.5. Exception Codes

14.1.5.1. 408: Request timed out

14.1.5.1.1. The operation did not complete within the allotted amount of time. This code is returned when a stored procedure, trigger, or UDF (within a query) does not complete execution within the maximum execution time.

14.1.5.2. 429:Too many requests

14.1.5.2.1. The collection has exceeded the provisioned throughput limit. Retry the request after the server specified retry after duration. For more information, see request units.

15. Azure Cosmos DB Explorer to be able to sign in using RBAC

15.1. enable the property ?feature.enableAadDataPlane=true

16. Always Encrypt Very sensitive details

16.1. Setup Azure Key Vault

16.1.1. 1.Create a new key in KV. 2.Create AD app and generate client secret 3.Geive permissions (Get, List, Unwrap Key, Wrap Key, **Verify, and Sign) in KV access policy to the AD app.

16.2. Initialize the SDK

16.2.1. var tokenCredential = new ClientSecretCredential( "<aad-app-tenant-id>", "<aad-app-client-id>", "<aad-app-secret>"); var keyStoreProvider = new AzureKeyVaultKeyStoreProvider(tokenCredential); var client = new CosmosClient("<connection-string>") .WithEncryption(keyStoreProvider);

16.3. Create a data encryption key

16.3.1. var database = client.GetDatabase("my-database"); await database.CreateClientEncryptionKeyAsync( "my-key", DataEncryptionKeyAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256, new EncryptionKeyWrapMetadata( keyStoreProvider.ProviderName, "akvKey", "https://<my-key-vault>.vault.azure.net/keys/<key>/<version>"));

16.4. Create a container with encryption policy

16.4.1. var path1 = new ClientEncryptionIncludedPath { Path = "/property1", ClientEncryptionKeyId = "my-key", EncryptionType = EncryptionType.Deterministic.ToString(), EncryptionAlgorithm = DataEncryptionKeyAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.ToString() }; var path2 = new ClientEncryptionIncludedPath { Path = "/property2", ClientEncryptionKeyId = "my-key", EncryptionType = EncryptionType.Randomized.ToString(), EncryptionAlgorithm = DataEncryptionKeyAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.ToString() }; await database.DefineContainer("my-container", "/partition-key") .WithClientEncryptionPolicy() .WithIncludedPath(path1) .WithIncludedPath(path2) .Attach() .CreateAsync();

16.5. Filter queries on encrypted properties

16.5.1. var queryDefinition = container.CreateQueryDefinition( "SELECT * FROM c where c.property1 = @Property1"); await queryDefinition.AddParameterAsync( "@Property1", 1234, "/property1");

17. AzCommands

17.1. Create Account: az cosmosdb create \ --name '<account-name>' \ --resource-group '<resource-group>'

17.2. Create Account with Free tier and "Strong" Consistency az cosmosdb create \ --name '<account-name>' \ --resource-group '<resource-group>' \ --default-consistency-level 'strong' \ --enable-free-tier 'true'

17.3. Create Account with Regions az cosmosdb create \ --name '<account-name>' \ --resource-group '<resource-group>' \ --locations regionName='eastus'

17.4. Create database with az cosmosdb sql az cosmosdb sql database create \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --name '<database-name>'

17.5. Create container az cosmosdb sql container create \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --database-name '<database-name>' \ --name '<container-name>' \ --throughput '400' \ --partition-key-path '<partition-key-path-string>'

17.6. Create Container with indexing policy az cosmosdb sql container create \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --database-name '<database-name>' \ --name '<container-name>' \ --partition-key-path '<partition-key-path-string>' \ --idx '@.\policy.json' \ --throughput '400'

17.7. Update Container throughput az cosmosdb sql container throughput update \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --database-name '<database-name>' \ --name '<container-name>' \ --throughput '1000'

17.8. Update database throughput az cosmosdb sql database throughput update \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --name '<database-name>' \ --throughput '4000'

17.9. Update throughput type az cosmosdb sql container throughput migrate \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --database-name '<database-name>' \ --name '<container-name>' \ --throughput-type 'autoscale'

17.10. Update max throughput az cosmosdb sql container throughput update \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --database-name '<database-name>' \ --name '<container-name>' \ --max-throughput '5000'

17.11. Show min throughput az cosmosdb sql container throughput show \ --account-name '<account-name>' \ --resource-group '<resource-group>' \ --database-name '<database-name>' \ --name '<container-name>' \ --query 'resource.minimumThroughput' \ --output 'tsv'

17.12. Add account regions with failover az cosmosdb update \ --name '<account-name>' \ --resource-group '<resource-group>' \ --locations regionName='eastus' failoverPriority=0 isZoneRedundant=False \ --locations regionName='westus2' failoverPriority=1 isZoneRedundant=False \ --locations regionName='centralus' failoverPriority=2 isZoneRedundant=False

17.13. Enable automatic failover az cosmosdb update \ --name '<account-name>' \ --resource-group '<resource-group>' \ --enable-automatic-failover 'true'

17.14. Change failover priorities az cosmosdb failover-priority-change \ --name '<account-name>' \ --resource-group '<resource-group>' \ --failover-policies 'eastus=0' 'centralus=1' 'westus2=2'

17.15. Enable multi-region write az cosmosdb update \ --name '<account-name>' \ --resource-group '<resource-group>' \ --enable-multiple-write-locations 'true'

17.16. Remove account regions az cosmosdb update \ --name '<account-name>' \ --resource-group '<resource-group>' \ --locations regionName='eastus' failoverPriority=0 isZoneRedundant=False \ --locations regionName='westus2' failoverPriority=1 isZoneRedundant=False

17.17. Initiate failovers hanging the region with priority = 0 will trigger a manual failover for an Azure Cosmos account. az cosmosdb failover-priority-change \ --name '<account-name>' \ --resource-group '<resource-group>' \ --failover-policies 'westus2=0' 'eastus=1'

18. ARM Template for creating Acc/Db/Container

18.1. { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json# ", "contentVersion": "1.0.0.0", "resources": [ { "type": "Microsoft.DocumentDB/databaseAccounts", "apiVersion": "2021-05-15", "name": "[concat('csmsarm', uniqueString(resourceGroup().id))]", "location": "[resourceGroup().location]", "properties": { "databaseAccountOfferType": "Standard", "locations": [ { "locationName": "westus" } ] } }, { "type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases", "apiVersion": "2021-05-15", "name": "[concat('csmsarm', uniqueString(resourceGroup().id), '/cosmicworks')]", "dependsOn": [ "[resourceId('Microsoft.DocumentDB/databaseAccounts', concat('csmsarm', uniqueString(resourceGroup().id)))]" ], "properties": { "resource": { "id": "cosmicworks" } } }, { "type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers", "apiVersion": "2021-05-15", "name": "[concat('csmsarm', uniqueString(resourceGroup().id), '/cosmicworks/products')]", "dependsOn": [ "[resourceId('Microsoft.DocumentDB/databaseAccounts', concat('csmsarm', uniqueString(resourceGroup().id)))]", "[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', concat('csmsarm', uniqueString(resourceGroup().id)), 'cosmicworks')]" ], "properties": { "options": { "throughput": 400 }, "resource": { "id": "products", "partitionKey": { "paths": [ "/categoryId" ] } } } } ] }

18.2. Bicep Template resource Account 'Microsoft.DocumentDB/databaseAccounts@2021-05-15' = { name: 'csmsbicep${uniqueString(resourceGroup().id)}' location: resourceGroup().location properties: { databaseAccountOfferType: 'Standard' locations: [ { locationName: 'westus' } ] } } resource Database 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases@2021-05-15' = { parent: Account name: 'cosmicworks' properties: { options: { } resource: { id: 'cosmicworks' } } } resource Container 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2021-05-15' = { parent: Database name: 'customers' properties: { resource: { id: 'customers' partitionKey: { paths: [ '/regionId' ] } } } }

19. Deploy to ResourceGroup

19.1. az deployment group create \ --resource-group '<resource-group>' \ --template-file '.\template.json'

19.1.1. az deployment group create \ --resource-group '<resource-group>' \ --template-file '.\template.bicep'

19.2. az deployment group create \ --resource-group '<resource-group>' \ --name '<deployment-name>' \ --template-file '.\template.json'

19.3. az deployment group create \ --resource-group '<resource-group>' \ --template-file '.\template.json' \ --parameters name='<value>'

19.4. az deployment group create \ --resource-group '<resource-group>' \ --template-file '.\template.json' \ --parameters '@.\parameters.json'

20. Steps to access cosmos connestr from key vault

20.1. Store cosmos constr to Key vault as secret

20.2. Create azure app service and create managed identity

20.3. Provide access in KeyVault using access policy for the app service identity

20.4. Create web application that has logic to connect key vault and get secret key and communicate cosmos db.

20.4.1. AzureServiceTokenProvider tok = new AzureServiceTokenProvider

21. Limits RBAC

21.1. You can create up to 100 role definitions and 2,000 role assignments per Azure Cosmos DB account.

21.2. You can only assign role definitions to Azure AD identities belonging to the same Azure AD tenant as your Azure Cosmos DB account.

21.3. Azure AD group resolution isn't currently supported for identities that belong to more than 200 groups.

21.4. The Azure AD token is currently passed as a header with each individual request sent to the Azure Cosmos DB service, increasing the overall payload size.

22. Disable Access Keys using Arm template

22.1. disableLocalAuth = true

23. Built in roles RBAC

23.1. DocumentDB Account Contributor

23.1.1. Can manage Azure Cosmos DB accounts

23.2. Cosmos DB Account Reader

23.2.1. Can read Azure Cosmos DB account data.

23.3. Cosmos Backup Operator

23.3.1. Can submit a restore request from the Azure portal for a periodic backup-enabled database or a container. Can modify the backup interval and retention on the Azure portal. Can't access any data or use Data Explorer.

23.4. CosmosRestoreOperator

23.4.1. Can perform a restore action for an Azure Cosmos DB account with continuous backup mode

23.5. Cosmos DB Operator

23.5.1. Can provision Azure Cosmos accounts, databases, and containers. Can't access any data or use Data Explorer.

24. Restore

24.1. Only by support request

24.1.1. Eligible roles: Azure Cosmos DB owner, contributor, or have the Cosmos DB Operator (CosmosdbBackupOperator)

24.1.2. Other than "Basic" Plan

24.1.3. Other than "Basic" Plan

24.2. Settings that are not restored

24.2.1. VNET access control lists

24.2.2. Stored procedures, triggers, and user-defined functions

24.2.3. Multi-region settings

25. Backup

25.1. Periodic

25.1.1. Frequency: default value 4 hrs max : 24 hrs

25.1.1.1. Redundancy

25.1.1.1.1. Geo (Default)

25.1.1.1.2. Zone

25.1.1.1.3. local

25.1.1.1.4. can be altered even after account creation

25.1.2. Retention: Default value: 8 hrs Max:30 days

25.2. continuous (incur costs)

25.2.1. Retention 30 days

25.2.1.1. Redundancy

25.2.1.1.1. local (default)

25.2.1.1.2. Zone

25.2.1.1.3. redundancy can't be updated further

25.2.1.1.4. Restore

25.2.2. Not supported for below cases

25.2.2.1. customer managed certificate used

25.2.2.2. Multi-region write accounts

25.2.2.3. ou can't restore an account into a region where the source account did not exist

25.2.2.4. The retention period is 30 days and can't be changed

25.2.2.5. Can't modify or delete IAM policies when restore is in progress

25.2.2.6. Point in time restore always restores to a new Azure Cosmos DB account

25.2.2.7. Collection's consistent indexes may still be rebuilding after completing the restore.

25.2.2.8. Azure Synapse Link and continuous backup mode can't coexist in the same database account

26. monitor server side metrics

26.1. Azure Monitor to monitor metrics Capturing every min Retention 30 days

26.1.1. throughput

26.1.2. storage availability

26.1.3. latency

26.1.4. consistency

26.1.5. system level metrics

26.2. Azure Monitor to monitor diagnostic logs

26.2.1. Telemetries like events and traces are stored as logs ex. throughput change

26.2.1.1. store logs in

26.2.1.1.1. AzureDiagnostics table

26.2.1.1.2. AzureActivity

26.3. The Azure Cosmos DB portal Retention 7 days

26.3.1. All Azure monitor logs

26.4. he Cosmos DB NoSQL API SDKs to programmatically monitor the account

27. logs

27.1. LogAnalytics

27.1.1. AzureDiagnostics

27.1.1.1. AzureDiagnostics | where TimeGenerated >= ago(1h) | where ResourceProvider=="MICROSOFT.DOCUMENTDB" and Category=="DataPlaneRequests"

27.1.2. Resource-specific

27.1.2.1. CDBDataPlaneRequests | where TimeGenerated >= ago(1h)

27.2. event hubs

27.3. storage