PYSPARK Tech Lead at c2c it jobs Chicago IL

Job Title: PYSPARK Tech Lead

Location: Chicago IL (Day On Onsite)

Job Type: Contract

Job Description:

·         We are seeking a Pyspark Technical Lead with 12 years of experience to join our team. The ideal candidate will have expertise in Pyspark, Cloudera HUE, Cloudera Data Platform, Apache Iceberg, Spark.

·         The candidate will play a crucial role in driving our research and development initiatives, leveraging their technical skills to deliver impactful solutions.

Mandate Skills: Cloudera HUE, PySpark, Spark Pyspark, Cloudera Data Platform, Apache Iceberg

Qualifications

• Possess strong expertise in Cloudera HUE and Cloudera Data Platform

• Demonstrate proficiency in Apache Iceberg and Spark

• Have extensive experience with PySpark for data processing

• Show a solid understanding of data architecture and data engineering principles

• Exhibit excellent problem-solving skills and attention to detail

• Have a background in research and development is a plus

• Display strong communication and collaboration skills

• Be able to work effectively in a hybrid work model

• Have a proactive approach to learning and staying current with industry trends

• Be capable of mentoring and guiding junior team members

• Demonstrate the ability to deliver high-quality solutions within deadlines

• Show a commitment to continuous improvement and innovation

Possess a strong sense of ownership and accountability for project outcomes.

Responsibilities:

• Lead the design and implementation of data solutions using Cloudera HUE and Cloudera Data Platform

• Supervise the integration of Apache Iceberg and Spark into existing data workflows

• Provide technical guidance and mentorship to junior team members on PySpark best practices

• Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions

• Ensure data quality and integrity through meticulous testing and validation processes

• Develop and maintain documentation for data architectures and workflows

• Optimize data processing pipelines for performance and efficiency

• Conduct code reviews to ensure alignment to coding standards and best practices

• Solve and resolve complex technical issues related to data processing

• Stay updated with the latest advancements in data technologies and incorporate them into projects

• Drive innovation by exploring new tools and techniques in the research and development domain

• Communicate effectively with stakeholders to gather requirements and provide project updates

• Contribute to the overall success of the company by delivering high-quality data solutions that drive business value.