Databricks: Python SDK

(Last Updated On: )

This post is how to use the Databricks Python SDK.

Install the Package

  1. pip install databricks-sdk

Update Package

  1. pip install databricks-sdk --upgrade

Check Package Version

  1. pip show databricks-sdk | grep -oP '(?<=Version: )\S+'

Setup WorkspaceClient

  1. from databricks.sdk import WorkspaceClient
  2.  
  3. secret = dbutils.secrets.get(scope = "<SCOPE>", key = "<KEY>")
  4.  
  5. w = WorkspaceClient(
  6. host = 'https://<URL>/'
  7. azure_workspace_resource_id = '<RESOURCE_ID_OF_DATABRICKS>',
  8. azure_tenant_id = '<TENANT_ID>',
  9. azure_client_id = '<CLIENT_ID>',
  10. azure_client_secret = secret
  11. )

Setup AccountClient

You can get the account_id from the databricks account portal. By your id in the top right hand corner.

  1. from databricks.sdk import AccountClient
  2.  
  3. secret = dbutils.secrets.get(scope = "<SCOPE>", key = "<KEY>")
  4.  
  5. a = AccountClient(
  6. host = 'https://accounts.azuredatabricks.net'
  7. account_id = '<ACCOUNT_ID>'
  8. azure_tenant_id = '<TENANT_ID>',
  9. azure_client_id = '<CLIENT_ID>',
  10. azure_client_secret = secret
  11. )

List Workspace Groups

NOTE: You must also setup the workspaceclient to do this.

  1. w.groups.list()

List Account Groups

NOTE: You must also setup the accountclient to do this. You must also be account admin.

  1. a.groups.list()

Create Storage Credential

NOTE: Your SPN must be account admin to do this. You must also setup the workspaceclient to do this.

  1. from databricks.sdk.service.catalog import AzureManagedIdentity
  2.  
  3. storage_credential_name = '<CREDENTIAL_NAME>'
  4. comment = '<COMMENT>'
  5. connector_id = '<DATABRICKS_ACCESS_CONNECTOR>'
  6. az_mi = AzureManagedIdentity(access_connector_id = connector_id)
  7.  
  8. w.storage_credenditals.create(
  9. name = storage_credential_name,
  10. azure_managed_identity = az_mi
  11. comment = comment
  12. )