ob Title: Azure Databricks Engineer with snowflake
Location: Boston, MA ( 5DAYS ONSITE)
Duration: Long Term
Description
We are seeking an experienced Azure Databricks Engineer with strong Snowflake expertise to design, develop, and maintain scalable cloud-based data engineering solutions. The candidate will work with Azure Databricks, PySpark, SQL, Snowflake, and Azure data services to build robust ETL/ELT pipelines and modern data platforms.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
Develop ETL/ELT workflows for ingesting data from structured and unstructured sources.
Build and optimize Snowflake data warehouses, databases, schemas, tables, views, and stored procedures.
Develop complex SQL queries for data transformation, validation, reconciliation, and analysis.
Implement data processing solutions using Databricks, Apache Spark, Delta Lake, and PySpark.
Integrate Azure Databricks with Azure Data Lake Storage Gen2 (ADLS) and other Azure data services.
Develop incremental data loads, CDC processes, and batch/streaming pipelines.
Implement and optimize Delta Lake tables, partitioning, caching, schema evolution, and performance tuning.
Design data pipelines to load transformed data into Snowflake.
Optimize Snowflake performance through appropriate clustering, warehouse sizing, query optimization, and storage strategies.
Implement data quality, validation, error handling, logging, and reconciliation mechanisms.
Work with Azure Data Factory (ADF) for pipeline orchestration and scheduling.
Develop reusable frameworks and components for data ingestion and transformation.
Implement CI/CD practices for Databricks notebooks, code, and data pipelines.
Monitor pipeline performance and troubleshoot failures across Databricks, Azure, and Snowflake environments.
Collaborate with data architects, analysts, BI teams, application teams, and business stakeholders.
Ensure data security, governance, access controls, and compliance across Azure and Snowflake.
Must-Have Skills
Strong hands-on experience with Azure Databricks
Strong PySpark / Apache Spark experience
Advanced SQL
Strong hands-on experience with Snowflake
Experience with Azure Data Lake Storage Gen2 (ADLS)
Experience with Azure Data Factory (ADF)
Strong understanding of ETL/ELT data pipelines
Experience with Delta Lake
Data modeling and warehouse concepts
Performance tuning of Databricks/Spark and Snowflake workloads
Experience with Git and CI/CD
Strong troubleshooting and production support experience
Good-to-Have Skills
Azure Synapse Analytics
Kafka or other streaming technologies
Snowpipe / Snowflake Streams and Tasks
Snowflake Dynamic Tables
Azure Event Hubs
Terraform
Azure DevOps
Databricks Workflows
Unity Catalog
Data governance and data quality
Python development
Experience with dimensional modeling and medallion architecture
