Senior Databricks Engineer/Architect – C2C – NJ/PA(Hybrid)

Hi,

Senior Databricks Engineer/Architect – C2C – NJ/PA(Hybrid)

Responsibilities:

Evaluate and understand the latest features and capabilities of Databricks platform and assess their potential impact on our organization’s data ecosystem.

Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions using Databricks.
Help to design, develop, and implement scalable and efficient data science/data engineering solutions using Databricks and related technologies.

Optimize and fine-tune existing Databricks workflows to improve performance and reliability.
Provide technical guidance and mentorship to corporate Databricks community about best practices and architecture, predominantly in the U.S.- EST time zone, ideally being able to commute to NJ/PA locations when in-person meetings are appropriate.

Collaborate with data scientists, data engineers, and other stakeholders to ensure seamless integration of Databricks solutions into our data ecosystem.
Understand the typical integrations with other Data Integration / Data Governance frameworks such as Starburst Trino, Collibra, Immuta, etc., and assess their compatibility and potential specific adoption criteria for our organization.
Stay up-to-date with the latest trends and advancements in Databricks and related technologies, and proactively recommend improvements and enhancements to our existing systems.

Requirements:

Proven experience as a Databricks Engineer, with a focus on Databricks-AWS implementation.
5+year of experience in Python, Spark, Scala and big data with AWS platform  
Nice to have experience in Streaming preferably  Kafka and Databricks integrations.
Strong understanding of Databricks architecture and its integration with AWS services.
Experience in designing and implementing scalable data processing pipelines using Databricks.
Familiarity with cloud-based data storage and processing technologies, particularly AWS services.
Strong problem-solving skills and ability to troubleshoot and resolve complex technical issues.
Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
Prior experience in the pharmaceutical or healthcare industry is a plus.

Regards,
Ajith Maria Alban