Sr Data Engineer
Mclean, VA Need Local
Long Term
Contract
• Design & Build Scalable Data Pipelines using PySpark for large-scale batch and streaming data processing.
• Develop PySpark Solutions — write production-grade PySpark code to read data from S3 (Parquet/Delta files), perform complex transformations, and process large-scale datasets efficiently.
• Implement Deduplication Logic — design and implement robust deduplication strategies for large datasets using PySpark.
• Performance Tuning (PySpark) — optimize PySpark jobs for reading and processing very large datasets, including partitioning, caching, broadcast joins, and shuffle optimization.
• Develop Cloud-Native Data Solutions on AWS — leveraging services like S3, Glue, EMR, Lambda, Step Functions, and Redshift.
• Engineer Snowflake Data Platforms — build warehouses, schemas, and data models optimized for analytics and reporting.
• Build Snowflake Iceberg Tables — design and implement Apache Iceberg tables in Snowflake for open lakehouse architectures and interoperability.
• Develop Dynamic Tables & Materialized Views — build and maintain Snowflake Dynamic Tables and Materialized Views to support near real-time analytics and query acceleration.
• Snowflake Performance Tuning — optimize Snowflake workloads through clustering, micro-partition pruning, query profiling, warehouse sizing, result caching, and materialization strategies.
• Modernize Legacy ETL — migrate on-prem ETL workloads (e.g., DataStage, Informatica) to PySpark and Snowflake-based cloud pipelines.
• Optimize Performance — tune Spark jobs, Snowflake queries, and AWS resource utilization for speed and cost.
• Ensure Data Quality & Governance — implement validation, lineage, and monitoring across every pipeline.
• Collaborate Across Teams — partner with data architects, analysts, TPMs, and business stakeholders to deliver trustworthy data products.
• Document Everything — from technical designs to runbooks, ensuring every pipeline is maintainable and audit-ready.
What We’re Looking For
Must-Haves
• 6+ years of hands-on data engineering experience in enterprise environments.
• Strong expertise in PySpark — building distributed data processing pipelines at scale.
• Hands-on experience writing PySpark code to read data from S3 Parquet/Delta files, perform transformations, and handle large datasets.
• Proven experience in performance tuning of PySpark jobs when reading and processing very large datasets.
• Experience implementing deduplication logic in PySpark for high-volume data pipelines.
• Deep hands-on experience with Snowflake — data modeling, SnowSQL, Snowpipe, Streams, Tasks, and role-based access.
• Hands-on experience creating Snowflake Iceberg Tables for open table format and lakehouse use cases.
• Experience building Snowflake Dynamic Tables and Materialized Views for incremental data processing and query acceleration.
• Strong Snowflake performance tuning skills — clustering keys, micro-partition pruning, query profiling, warehouse right-sizing, caching strategies, and cost optimization.
• Proven AWS experience — S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, and Redshift.
• Advanced SQL skills — complex joins, subqueries, window functions, CTEs, and performance tuning.
• Strong Python programming skills beyond PySpark — for utilities, automation, and orchestration.
• Experience with orchestration tools — Airflow, AWS Step Functions, or equivalent.
• Solid understanding of data warehousing, ELT/ETL patterns, data lakes, and lakehouse architectures.
• Excellent communication skills — able to articulate technical decisions to both engineers and business stakeholders.
Nice to Have
• Experience with IBM DataStage or other legacy ETL tools (for modernization contexts).
• Familiarity with CI/CD for data pipelines (Git, Jenkins, GitHub Actions, Terraform).
• Exposure to data quality frameworks (Great Expectations, dbt tests).
• Knowledge of streaming platforms (Kafka, Kinesis).
• Experience in regulated environments — financial services, mortgage, or GSE programs (Fannie Mae / Freddie Mac).
• Certifications: AWS Certified Data Analytics / Solutions Architect, SnowPro Core/Advanced.
Munesh
770-838-3829,
CYBER SPHERE LLC
