Job Description
Description
:
Responsibilities:
Develop & Optimize Data PipelinesBuild, test, and maintainETL/ELTdata pipelines using AzureDatabricks & Apache Spark (PySpark).Optimizeperformance and cost-efficiencyof Spark jobs.Ensure data quality through validation, monitoring, and alerting mechanisms.Understand cluster types, configuration, and use-case for serverlessImplement Unity Catalog for Data GovernanceDesign and enforceaccess control policiesusing Unity Catalog.Managedata lineage, auditing, and metadata governance.Enable secure data sharing across teams and external stakeholders.Integrate with Cloud Data PlatformsWork withAzure Data Lake Storage / Azure Blob Storage/ Azure Event Hubto integrate Databricks with cloud-baseddata lakes, data warehouses, and event streams.ImplementDelta Lakefor scalable, ACID-compliant storage.Automate & Orchestrate Workf...