Job Title: Azure Databricks Lead Engineer
Location: St Louis, MO (Onsite)
Contract
Key Requirements:
We are looking for a skilled and experienced Databricks Lead to drive data engineering initiatives, lead solution architecture, and support large-scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory (ADF).
The ideal candidate will possess a strong foundation in cloud data platforms, streaming technologies, and a leadership mindset to mentor and guide teams in delivering high-quality solutions.
Key Responsibilities:
- Lead the design, development, and deployment of scalable data pipelines using Azure Databricks.
- Integrate real-time and batch data sources using Apache Kafka and ADF.
- Implement data transformations, enrichment, and quality checks using PySpark/Scala within the Databricks environment.
- Design and develop integrations with Stelo for structured data loading and transformation (if applicable) Collaborate with data scientists, analysts, and other engineering teams to deliver business-critical insights Optimize pipeline performance, cost, and scalability in the Azure cloud environment.
- Define best practices for data ingestion, processing, storage, and governance.
- Lead and mentor a team of data engineers, conducting code reviews and ensuring development standards. Support troubleshooting and incident management for data-related issues in production.
Regards,
