Databricks Senior Data Engineer -Pyspark
Location NY onsite
About the Role
The Senior Data Engineer Consultant will implement and optimize enterprise-grade
Databricks data platforms across AWS and GCP. You'll collaborate with architecture
leads on design/standards and deliver high-quality pipeline implementations with
exceptional technical communication.
Outstanding verbal and written communication is required—you'll partner with leads,
document implementations, and provide knowledge transfer to client teams.
Key Responsibilities
● Implement scalable Databricks pipelines using Spark/PySpark across AWS and
GCP.
● Design and build optimized ETL/ELT integrating diverse sources into Delta Lake.
● Implement Spark job tuning, cluster optimization, and storage partitioning.
● Build monitoring/alerting/observability for production reliability.
● Author complex BigQuery SQL for cross-platform analytics.
● Collaborate with architecture leads on design reviews, standards adherence, and
technical implementation.
● Create comprehensive documentation, runbooks, and implementation guides.
● Participate in code reviews and provide hands-on technical guidance.
● Develop automated testing and data validation frameworks.
Required Qualifications
● Bachelor's/Master's in CS, Data Engineering, or related field.
● 7+ yrs data engineering | 4+ yrs Databricks/Spark hands-on.
● Expertise: Spark, PySpark, Delta Lake, SQL optimization.
● Proven Databricks on AWS (S3/Glue) + GCP (GCS/BigQuery).
● Advanced Python (Pandas/NumPy) + pytest testing.
● Cloud services: AWS S3/Glue, GCP BigQuery/Dataflow.
● Exceptional communication—presentations + technical writing.
● Consulting experience: End-to-end project delivery.
● Git + CI/CD proficiency.
Preferred Qualifications
● Real-time streaming (Spark Streaming, Kafka, Delta Live Tables).
● Databricks workspace/cluster/Unity Catalog management.
● Docker/Kubernetes.
● Media/entertainment or enterprise analytics experience.
● Jira/Confluence/Lucidchart
|
Neha Chaudhary
Survey Link: The survey is anonymous, and the information collected will be analysed in aggregate to support initiatives that improve the total experience across Compunnel.
|
—
