Job Description
Get AI-powered advice on this job and more exclusive features.
- Develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake
- Design and implement scalable and efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
- Developing scalable and re-usable frameworks for ingesting data sets
- Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms
- Working with event-based/streaming technologies to ingest and process data
- Provide technical guidance and support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases
- Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best pr...