Sr Data Engineer
Mclean, VA Need Local
Long Term
Contract
Â
•         Design & Build Scalable Data Pipelines using PySpark for large-scale batch and streaming data processing.
•         Develop PySpark Solutions — write production-grade PySpark code to read data from S3 (Parquet/Delta files), perform complex transformations, and process large-scale datasets efficiently.
•         Implement Deduplication Logic — design and implement robust deduplication strategies for large datasets using PySpark.
•         Performance Tuning (PySpark) — optimize PySpark jobs for reading and processing very large datasets, including partitioning, caching, broadcast joins, and shuffle optimization.
•         Develop Cloud-Native Data Solutions on AWS — leveraging services like S3, Glue, EMR, Lambda, Step Functions, and Redshift.
•         Engineer Snowflake Data Platforms — build warehouses, schemas, and data models optimized for analytics and reporting.
•         Build Snowflake Iceberg Tables — design and implement Apache Iceberg tables in Snowflake for open lakehouse architectures and interoperability.
•         Develop Dynamic Tables & Materialized Views — build and maintain Snowflake Dynamic Tables and Materialized Views to support near real-time analytics and query acceleration.
•         Snowflake Performance Tuning — optimize Snowflake workloads through clustering, micro-partition pruning, query profiling, warehouse sizing, result caching, and materialization strategies.
•         Modernize Legacy ETL — migrate on-prem ETL workloads (e.g., DataStage, Informatica) to PySpark and Snowflake-based cloud pipelines.
•         Optimize Performance — tune Spark jobs, Snowflake queries, and AWS resource utilization for speed and cost.
•         Ensure Data Quality & Governance — implement validation, lineage, and monitoring across every pipeline.
•         Collaborate Across Teams — partner with data architects, analysts, TPMs, and business stakeholders to deliver trustworthy data products.
•         Document Everything — from technical designs to runbooks, ensuring every pipeline is maintainable and audit-ready.
What We’re Looking For
Must-Haves
•         6+ years of hands-on data engineering experience in enterprise environments.
•         Strong expertise in PySpark — building distributed data processing pipelines at scale.
•         Hands-on experience writing PySpark code to read data from S3 Parquet/Delta files, perform transformations, and handle large datasets.
•         Experience building Snowflake Dynamic Tables and Materialized Views for incremental data processing and query acceleration.
•         Strong Snowflake performance tuning skills — clustering keys, micro-partition pruning, query profiling, warehouse right-sizing, caching strategies, and cost optimization.
•         Proven AWS experience — S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, and Redshift.
•         Advanced SQL skills — complex joins, subqueries, window functions, CTEs, and performance tuning.
•         Strong Python programming skills beyond PySpark — for utilities, automation, and orchestration.
•         Experience with orchestration tools — Airflow, AWS Step Functions, or equivalent.
•         Solid understanding of data warehousing, ELT/ETL patterns, data lakes, and lakehouse architectures.
•         Excellent communication skills — able to articulate technical decisions to both engineers and business stakeholders.
Nice to Have
•         Experience with IBM DataStage or other legacy ETL tools (for modernization contexts).
•         Familiarity with CI/CD for data pipelines (Git, Jenkins, GitHub Actions, Terraform).
•         Exposure to data quality frameworks (Great Expectations, dbt tests).
•         Knowledge of streaming platforms (Kafka, Kinesis).
•         Experience in regulated environments — financial services, mortgage, or GSE programs (Fannie Mae / Freddie Mac).
•         Certifications: AWS Certified Data Analytics / Solutions Architect, SnowPro Core/Advanced.
Â
Â
Â
Munesh
770-838-3829,
