Need Local candidate only
Position – Lead Data Engineer With GCP
Location – Remote
UST Global/ Costco
Job Description –
Key Responsibilities:
- Automate data workflows using Airflow, PySpark, and Datapres on GCP.
- Develop ETL processes for diverse data sources.
- Manage GCP resources including Rataproc clusters, serverless batches, Vertex Al instance GCS buckets.
- Provide platform and pipeline support, troubleshoot Spark, BigQuery, and Airflow issues.
- Collaborate with data scientists and analysts.
- Offer technical support for internal users.
- Optimize data systems for performance and cost efficiency.
- Perform root cause analysis and implement preventative measures.
- Design and troubleshoot CI/CD pipelines using GCP and Airflow.
- Set up and maintain GCP infrastructure.
- Deploy and manage Airflow environments, handle version upgrades.
- Monitor alerts, provide on-call support, manage user tickets.
- Plan and execute data migrations, resolve performance issues.
- Optimize cloud storage, implement data lifecycle policies.
- Administer containerized workloads using Kubernetes, Docker, and Helm Charts.
Required Skills:
- Strong programming skills in Python, SQL.
- Expertise in GCP data tools: Biggyer, Datarc Vertex Al, Pub/Sub, Cloud Functions.
- Experience with Apache Airflow, PySpark, Python, SparksoL and data modeling,
- Troubleshooting data platform and pipeline failures.
- Background in Linux/UNIX systems administration and command-line tools.
- Hands-on experience with cloud platforms (AWS, Azure, Google Cloud).
- Experience in DevOps or Site Reliability Engineering roles.
- Managing container-based infrastructures: Docker, Kubernetes.
- Knowledge of CI/CD tools, DevOps principles, and cloud security practices.
- Monitoring and logging tools: Prometheus, Grafana, Splunk.
Pratik Kumar | Lead Recruiter
Amaze Systems Inc
USA: 8951 Cypress Waters Blvd, Suite 160, Dallas, TX 75019
