Job Description
We're seeking a hands on Data Engineer to help plan and execute the migration of analytics and ETL/ELT workloads from their current solution to an Azure + Databricks lakehouse stack. You'll help design ingestion and transformation pipelines, optimize Spark jobs, and ensure governance and compliance suitable for a pharmaceutical environment
Key Responsibilities
Analyze current pipelines and re‑platform to Databricks (PySpark/SQL) following bronze, silver, gold patterns.
Rebuild schedules/orchestrations (e.g., Databricks Workflows, ADF/Synapse/Fabric) and replace current instance specific operators with native Spark/Delta patterns.
Map data models/virtualized objects to Delta Lake tables with partitioning, Z Ordering, and optimized file layouts.
Develop scalable ETL/ELT in PySpark and SQL
Implement unit/integration tests and data quality checks (freshness, completeness, schema)
Tune Spark (shuffle partitions, broadcast joins, AQE), implement caching, and cost‑optimiz...
Key Responsibilities
Analyze current pipelines and re‑platform to Databricks (PySpark/SQL) following bronze, silver, gold patterns.
Rebuild schedules/orchestrations (e.g., Databricks Workflows, ADF/Synapse/Fabric) and replace current instance specific operators with native Spark/Delta patterns.
Map data models/virtualized objects to Delta Lake tables with partitioning, Z Ordering, and optimized file layouts.
Develop scalable ETL/ELT in PySpark and SQL
Implement unit/integration tests and data quality checks (freshness, completeness, schema)
Tune Spark (shuffle partitions, broadcast joins, AQE), implement caching, and cost‑optimiz...