Job Description
Role: Data Engineer with Databricks specialization.
Data Pipeline Development
Design, build, test, and maintain high‑performance ingestion and transformation pipelines using Azure Databricks.
Implement modular, reusable business logic and transformations using DBT or Coalesce (or similar transformation frameworks).
Develop and optimize ELT/ETL workflows including CDC-based incremental data loads, data wrangling, cleansing, and enrichment.
Ensure pipelines are scalable, maintainable, resilient, and cost‑efficient across cloud environments.
Quality, Testing & Documentation
Conduct unit, integration, and regression testing to validate transformation logic and data accuracy.
Implement and maintain data quality frameworks, ensuring data completeness, consistency, and reliability.
Produce clear, comprehensive technical design and build documentation.
Collaboration & Governance
Work closely with BI, data governance, and platform teams to ensure consistent data...
Data Pipeline Development
Design, build, test, and maintain high‑performance ingestion and transformation pipelines using Azure Databricks.
Implement modular, reusable business logic and transformations using DBT or Coalesce (or similar transformation frameworks).
Develop and optimize ELT/ETL workflows including CDC-based incremental data loads, data wrangling, cleansing, and enrichment.
Ensure pipelines are scalable, maintainable, resilient, and cost‑efficient across cloud environments.
Quality, Testing & Documentation
Conduct unit, integration, and regression testing to validate transformation logic and data accuracy.
Implement and maintain data quality frameworks, ensuring data completeness, consistency, and reliability.
Produce clear, comprehensive technical design and build documentation.
Collaboration & Governance
Work closely with BI, data governance, and platform teams to ensure consistent data...