Lead Data Engineer

Lead Data Engineer– Data Ingestion Framework
Boston, MA
Long Term
Contract

 

Lead Data Engineer with strong expertise in Python and DBT to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing ETL capabilities with an open-source technology stack leveraging Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

 

The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.

 

Key Responsibilities

1. Data Ingestion Framework Development

• Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.

• Design modular and extensible components to support multiple source systems.

• Develop metadata-driven and configuration-based ingestion capabilities.

• Build reusable connectors and adapters for different source technologies.

• Implement full and incremental data ingestion patterns, including CDC where applicable.

• Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.

2. Data Integration & Lakehouse Architecture

• Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.

• Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.

• Design and implement data lakehouse ingestion patterns using AWS S3.

• Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.

• Implement schema handling, data validation, partitioning, and data movement strategies.

3. Python & dbt Engineering

• Develop production-grade Python applications and reusable ingestion libraries.

• Build and maintain dbt models for data transformation and integration.

• Implement dbt testing, documentation, incremental models, and reusable macros.

• Optimize Python-based ingestion processes and dbt transformations for performance and scalability.

• Follow engineering best practices for modularity, maintainability, and code quality.

4. Workflow Orchestration

• Implement and manage workflow orchestration using Control-M and Dagester.

• Develop reusable workflows for scheduling, dependency management, retries, and pipeline monitoring.

• Integrate orchestration workflows with Python ingestion components and Snowflake.

5. Testing, Deployment & Production Support

• Develop unit tests and integration tests for ingestion framework components.

• Support migration and deployment across DEV, SIT, UAT, and PROD environments.

• Collaborate with QA teams to perform functional, integration, regression, and performance testing.

• Troubleshoot ingestion failures, data discrepancies, and performance bottlenecks.

• Support production deployment, stabilization, documentation, and knowledge transfer.

6. Technical Leadership & Collaboration

• Work closely with the onshore solution architect, customer stakeholders, and offshore engineering teams.

• Provide technical guidance, code reviews, and engineering best practices.

• Participate in architecture discussions, technical design reviews, and sprint planning.

• Mentor developers and ensure adherence to established engineering standards.

Required Technical Skills

• Python: Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.

• DBT: Strong hands-on expertise in DBT Core, including models, macros, tests, incremental processing, and deployment practices.

• Data Engineering: Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.

• Snowflake: Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.

• AWS: Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.

• Data Integration: Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.

• SQL: Advanced SQL development and query optimization skills.

• CI/CD: Experience with Git-based version control, automated builds, testing, and deployment pipelines.

Good-to-Have Skills

• Hands-on experience with Dagster for data orchestration and asset-based pipeline development.

• Experience with utilizing Control-M JOB scheduler.

• Experience with Apache Iceberg and open table formats.

• Experience implementing metadata-driven ingestion frameworks.

• Experience migrating ETL workloads from SnapLogic or similar integration platforms.

• Experience with Docker, Terraform, and infrastructure automation.

• Experience implementing data quality, observability, and monitoring solutions.

• Experience working in an Agile environment utilizing SCRUM

Qualifications

• 10–12 years of overall experience in data engineering and data integration.

• Strong architectural and hands-on engineering capabilities.

• Experience working on enterprise data modernization initiatives.

• Excellent communication and stakeholder management skills.

• Ability to work independently in a distributed onshore/offshore delivery model.

Mandatory Skills

1. Expert-level Python development.

2. Strong hands-on dbt experience.

3. Enterprise data ingestion framework development.

4. Snowflake and AWS S3 experience.

5. Strong SQL and data integration experience.

 

 

 

Munesh

770-838-3829,

munesh@cysphere.net

munesh.reddy.us@gmail.com

CYBER SPHERE LLC