Job Description
Job Description
Join our team to leverage your data engineering skills in a dynamic environment, ensuring seamless data migration and optimization for advanced AI and ML projects. Apply now to be part of our innovative journey!
Key responsibilities
Data pipeline development:
- Design, develop, and deploy Python-based ETL/ELT pipelines to migrate data from the on-premises MS SQL Server into the Databricks instance,
- Ensure efficient ingestion of historical parquet datasets into Databricks.
Data quality & validation:
- Implement validation, reconciliation, and quality assurance checks to ensure accuracy and completeness of migrated data,
- Handle schema mapping, field transformations, and metadata enrichment to standardize datasets,
- Ensure data governance, quality assurance, and compliance are integral to all migration activities.
Performan...