Databricks: Mounts

(Last Updated On: )

This post is how to mount on Databricks.

Notes

  • Security Issue: They are shared across all clusters and users
  • Should always be unmounted after use
    • Due to Service Prinicpal password rotations
    • Reliability esspecially in BCDR
  • Databricks recommends using Unity Catalog instead of mounts as they are legacy.
  • Could be conflicts in other projects due to naming
  • Do not create mounts manually. Always have your project mount and unmount at the end

List Mounts

  1. dbutils.fs.mounts()

Unmount

  1. dbutils.fs.unmount("<MOUNT>")

Mount

  1. client_id = "<CLIENTID>"
  2. secret = dbutils.secrets.get(scope = "<SCOPE_NAME>", key = "<SECRET_NAME>")
  3. tenant_id = "<TENANT_ID>"
  4. storage_account_name = "<STORAGE_ACCOUNT_NAME>"
  5. container_name = "<CONTAINER_NAME>"
  6.  
  7. configs = {
  8.   "fs.azure.account.auth.type": "OAuth",
  9.   "fs.azure.account.oauth.provider.type": "org.apache.fs.azurebfs.oauth2.ClientCredsTokenProvider",
  10.   "fs.azure.account.oauth2.client.id": client_id,
  11.   "fs.azure.account.oauth2.client.secret": secret,
  12.   "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/"  tenant_id + "/oauth2/token"
  13. }
  14.  
  15. path = "abfss://%s@%s.dfs.core.windows.net/" % (container_name, storage_account_name)
  16.  
  17. dbutils.fs.mount(
  18.     source = path,
  19.     mount_point = "/mnt/<MOUNT_NAME>",
  20.     extra_configs = configs
  21. )