Role: Azure Data Engineer Silver and Gold layer data models
Location: Norwell, MA- onsite
Duration: 6+Months C2H
Mode: Video
Job Details:
Job Summary:
We are seeking a skilled Data Engineer to design, build, and maintain high-quality, scalable data pipelines with a strong focus on Silver and Gold layer transformations, data quality automation, and CI/CD-driven deployments. The ideal candidate will have hands-on experience with Python, PySpark, Git-based CI/CD pipelines, ARM templates, and data quality libraries such as Great Expectations.
Key Responsibilities:
Design and implement Silver and Gold layer data models following medallion architecture best practices.
Perform data cleansing, standardization, enrichment, and aggregation to support analytics and reporting.
Build optimized PySpark-based transformations for large-scale data processing.
Ensure data consistency, performance, and scalability across datasets.
Build and maintain CI/CD pipelines using Git-based workflows (Azure DevOps / GitHub).
Use ARM templates (or IaC equivalents) for automated infrastructure provisioning.
Enable automated deployment of data pipelines, notebooks, and configurations.
Follow DevOps best practices for version control, branching, and release management.
Develop reusable Python-based automation scripts for pipeline orchestration, validation, and monitoring.
Create modular, maintainable, and testable Python code.
Support automation of metadata, logging, alerting, and operational tasks.
Implement data quality libraries such as Great Expectations.
Define and automate data quality rules (completeness, accuracy, freshness, consistency).
Integrate data quality checks into data pipelines and CI/CD workflows.
Monitor, log, and troubleshoot data quality issues proactively.
Work closely with data architects, analysts, QA, and business stakeholders.
Translate business and analytical requirements into robust data engineering solutions.
Participate in Agile ceremonies and support sprint-based delivery.
Provide production support, performance tuning, and troubleshooting
Required Skills and Qualifications:
5–10+ years of experience in data engineering or analytics engineering.
Strong hands-on experience with Silver & Gold layer development, Python (automation and data processing) and PySpark. Also with Great Expectations (data quality framework) and Experience with CI/CD pipelines using Git-based tools. Hands-on experience with ARM templates or infrastructure-as-code concepts. Strong understanding of data modelling and medallion architecture. Experience working with large datasets in distributed environments.
Good to Have:
Microsoft Certified: Azure Data Engineer Associate or Azure Enterprise Data Analyst Associate.
Wast management or oil and gas domain knowledge
Best Regards,
Navneet Jha
D : 848-999-0314 | Email: navneet.jha@siriinfo.com
Siri InfoSolutions Inc, 3 Ethel Rd, Suite # 302, Edison NJ 08817.
