Job Description
We are undergoing a significant data transformation to ensure that accurate and consistent data is available precisely when and where it's needed.
Responsibilities
- Write PySpark and SQL scripts to validate data pipelines, transformations, and integrations.
- Design and run tests for data validation, storage, and retrieval using Azure services like Data Lake, Synapse, and Data Factory, adhering to industry standards.
- Continuously enhance automated tests as new features are developed, ensuring Participate in data reconciliation and verify Data Quality frameworks to maintain data accuracy, completeness, and consistency across the platform.
- Formulate and maintain test strategies—including smoke, performance, functional, and regression testing—to ensure data processing and ETL jobs meet requirements.
- Collaborate with development teams to assess changes in data workflows and update test cases to preserve data integrity.