Must have:
• Databricks
• SQL
• Python
Nice to Have Skills:
• Leadership
• Accountability
• SQL
• Python
Nice to Have Skills:
• Leadership
• Accountability
JD:
• Develop and maintain data pipelines using Apache Spark and Scala to process large volumes of data efficiently.
• Implement data transformation logic to cleanse, normalize, and aggregate data from various sources, ensuring high data quality and consistency.
• Integrate data from different systems and sources, including structured and unstructured data, databases, and Mainframe systems, ensuring seamless data flow and compatibility. • Implement data validation and quality checks within data pipelines to ensure data accuracy, consistency, and integrity, and handle exception handling and error logging. • Develop and execute unit tests, integration tests, and end-to-end tests to ensure the reliability and correctness of Spark applications.
• Debug and troubleshoot issues during development and production phases.
• Implement data transformation logic to cleanse, normalize, and aggregate data from various sources, ensuring high data quality and consistency.
• Integrate data from different systems and sources, including structured and unstructured data, databases, and Mainframe systems, ensuring seamless data flow and compatibility. • Implement data validation and quality checks within data pipelines to ensure data accuracy, consistency, and integrity, and handle exception handling and error logging. • Develop and execute unit tests, integration tests, and end-to-end tests to ensure the reliability and correctness of Spark applications.
• Debug and troubleshoot issues during development and production phases.
