Databricks¶
About¶
Databricks is a cloud-based data analytics platform that enables collaborative data science. It offers a unified analytics platform built on Apache Spark, allowing teams to work together on big data analytics projects.
Product Details¶
Vendor URL: Databricks
Product Type: Monitoring
Product Tier: Tier II
Integration Method: AWS S3 Bucket
Integration URL: AWS S3 - Cyderes Documentation
Log Guide: Audit Log Reference
Parser Details¶
Log Format: JSON
Expected Normalization Rate: 100%
Data Label: DATABRICKS
UDM Fields (list of all UDM fields leveraged in the Parser):
| Log File Field | UDM Field |
|---|---|
| accountId | observer.user.product_object_id |
| actionName | metadata.product_event_type |
| auditLevel | security_result.severity_details |
| requestId | metadata.product_log_id |
| requestParams.aclPermissionSet | target.user.attribute.permissions |
| requestParams.authenticationMethod | extensions.auth.auth_details |
| requestParams.aws_access_key_id | security_result.detection_fields |
| requestParams.catalog_name | target.resource.name |
| requestParams.credential_id | security_result.about.resource.id |
| requestParams.credential_type | security_result.about.resource.resource_subtype |
| requestParams.full_name_arg | target.resource.name |
| requestParams.job_id | target.resource.id |
| requestParams.limit_size | target.resource.attribute.labels |
| requestParams.max_results | target.resource.attribute.labels |
| requestParams.metastore_id | additional.fields |
| requestParams.name | target.file.names |
| requestParams.name | target.resource.name |
| requestParams.name_arg | target.resource.name |
| requestParams.operation | security_result.action_details |
| requestParams.path | target.file.full_path |
| requestParams.resourceId | target.resource.id |
| requestParams.run_id | target.resource.id |
| requestParams.schema_name | target.resource.attribute.labels |
| requestParams.scope | target.resource.resource_subtype |
| requestParams.script-SHA256 | target.file.sha256 |
| requestParams.securable_full_name | target.resource.name |
| requestParams.securable_type | target.resource.resource_subtype |
| requestParams.securables | target.resource.attribute.labels |
| requestParams.shardName | target.group.group_display_name |
| requestParams.source_table_name | target.resource.name |
| requestParams.table_full_name | target.resource.name |
| requestParams.table_id | target.resource.id |
| requestParams.table_name | target.resource.name |
| requestParams.table_name_list | target.resource.name |
| requestParams.targetUserId | target.user.userid |
| requestParams.tokenId | additional.fields |
| requestParams.url | target.url |
| requestParams.userId | principal.user.userid |
| requestParams.userId | principal.user.user_display_name |
| requestParams.volume_full_name | target.resource.name |
| requestParams.volume_id | target.resource.id |
| requestParams.volume_id | target.resource.attribute.labels |
| requestParams.volume_storage_location | target.file.full_path |
| requestParams.workspace_id | additional.fields |
| response.errorMessage | security_result.summary |
| response.result | security_result.outcomes |
| response.statusCode | network.http.response_code |
| serviceName | metadata.description |
| serviceName | observer.application |
| sourceIPAddress | principal.ip |
| userAgent | network.http.user_agent |
| userIdentity.email | principal.user.email_addresses |
Product Event Types¶
| Event | UDM Event Classification |
|---|---|
| changeClusterAcl | RESOURCE_PERMISSIONS_CHANGE |
| changeJobAcl | RESOURCE_PERMISSIONS_CHANGE |
| changeWorkspaceAcl | RESOURCE_PERMISSIONS_CHANGE |
| checkPathAccess | USER_RESOURCE_ACCESS |
| generate... | USER_RESOURCE_CREATION |
| get... | USER_RESOURCE_ACCESS |
| globalInitScripts - create | USER_RESOURCE_CREATION |
| listTables | USER_RESOURCE_ACCESS |
| metadataAndPermissionsSnapshot | USER_RESOURCE_ACCESS |
| tokenLogin | USER_LOGIN |
Log Sample¶
{"version":"2.0","timestamp":1727893040070,"workspaceId":"1234567892345","sourceIPAddress":"10.30.52.114","userAgent":"Apache-HttpClient/4.5.14 (Java/1.8.0_392) RawDBHttpClient/apply Databricks-Service/driver","userIdentity":{"email":"janedoe@example.com"},"serviceName":"accounts","actionName":"tokenLogin","requestId":"c0566582-5d0c-45c7-86d9-c3d34d3ccd87","requestParams":{"user":"janedoe@example.com","tokenId":"abc123abc123abc123abc123abc123abcdefghijklm123456789","authenticationMethod":"API_INT_PAT_TOKEN"},"response":{"statusCode":200},"accountId":"123abc12-1234-abcd-8668-7459a9e0c104","auditLevel":"WORKSPACE_LEVEL"}
Sample Parsing¶
additional.fields["Token ID"] = "abc123abc123abc123abc123abc123abcdefghijklm123456789"
additional.fields["Workspace ID"] = "1234567892345"
extensions.auth.auth_details = "API_INT_PAT_TOKEN"
metadata.description = "accounts"
metadata.event_type = "USER_LOGIN"
metadata.log_type = "DATABRICKS"
metadata.product_event_type = "tokenLogin"
metadata.product_log_id = "c0566582-5d0c-45c7-86d9-c3d34d3ccd87"
metadata.product_version = "2.0"
metadata.vendor_name = "Databricks"
network.http.response_code = 200
network.http.user_agent = "Apache-HttpClient/4.5.14 (Java/1.8.0_392) RawDBHttpClient/apply Databricks-Service/driver"
observer.application = "accounts"
observer.user.product_object_id = "123abc12-1234-abcd-8668-7459a9e0c104"
principal.ip = "10.30.52.114"
principal.user.email_addresses = "janedoe@example.com"
principal.user.userid = "janedoe@example.com"
security_result.action_details = "tokenLogin"
security_result.action = "ALLOW"
security_result.severity_details = "WORKSPACE_LEVEL"
target.user.userid = "janedoe@example.com"