Hi ,
I hope you're doing well.
My name is (Pankaj Singh), and I’m a Senior Associate We are currently hiring for the following opportunity that may align with your expertise:
Job Role: Azure Databricks EngineerJob Location: Local to Cary, NC (F2F)Onsite (5days/week)Job Type: ContractPay Range: $60 –$63 /hour Role Descriptions:Azure Databricks Engineer Key Responsibilities:• Collect, store, process andanalyze large datasets to build and implement extract, transfer, load (ETL)processes.• Lead migration of data pipelinesfrom Synapse to Databricks.• Architect and implement batch +real-time data pipelines using PySpark, Databricks, and Event Hubs.• Establish data engineering bestpractices and mentor junior engineers.• Drive CI/CD automation with AzDOand Git.• Ensure data quality, governance,and compliance (Collibra).• Collaborate with business/techteams; leverage insurance domain knowledge where applicable.• Develop reusable frameworks toreduce the development effort involved thereby ensuring cost savings for theprojects.• Develop quality code withthought through performance optimizations in place right at the developmentstage.• Appetite to learn newtechnologies and be ready to work on new cutting-edge cloud technologies.• Work with team spread across theglobe in driving the delivery of projects and recommend development andperformance improvements. Essential Business Experience andTechnical Skills:• Building and Implementing dataingestion and curation process developed using Big data tools such as Spark(Scala/python/Java), Hive, HDFS, Sqoop, Kafka, Kerberos, Impala etc. and CDP7.x• Ingesting huge volumes data fromvarious platforms for Analytics needs and writing high-performance, reliable,and maintainable ETL code Strong• Strong analytic skills relatedto working with unstructured datasets• Strong experience inbuilding/designing Data warehouses, data stores for analytics consumption Onprem and Cloud (real time as well as batch use cases)• Ability to interact withbusiness analysts and functional analysts in getting the requirements andimplementing the ETL solutions. Required:• 7+ years in data engineering, 3+in a lead/senior role.• Strong expertise in AzureDatabricks, Synapse, PySpark.• Proven experience with real-timestreaming (Event Hubs, Structured Streaming, Kafka).• Deep knowledge of Azureecosystem (Data Lake, Data Factory, Key Vault).• Strong SQL, performance tuning,and CI/CD pipeline skills.• Excellent leadership andcommunication skills• Building and Implementing dataingestion and curation process developed using Big data tools such asSpark(Scala/python/Java), Data bricks, Delta Lake, Hive, HDFS, Sqoop, Kafka,Kerberos, Impala etc.• Hands on expertise in:Implementing analytical data stores in Azure platform using ADLS/ADF/DataBricks and Cosmos DB and CDP 7x• Ingesting huge volumes of datafrom various platforms for Analytics needs and writing high-performance,reliable and maintainable ETL code Strong• SQL knowledge and data analysisskills for data anomaly detection and data quality assurance.• Proficiency and extensiveExperience with Spark & Scala, Python and performance tuning is a MUST• Hive database management andPerformance tuning is a MUST. (Partitioning / Bucketing)• Strong SQL knowledge and dataanalysis skills for data anomaly detection and data quality assurance.• Strong analytic skills relatedto working with unstructured datasets• Performance tuning andproblem-solving skills is a must.• Code versioning experience usingBitbucket/AzDo. Working knowledge of AzDo pipelines would be a big plus.• Monitoring performance andadvising any necessary infrastructure changes.• Strong experience in buildingdesigning Data warehouses, data stores for analytics consumption. (real time aswell as batch use cases)• Eagerness to learn newtechnologies on the fly and ship to production• Expert in technical programdelivery across cross-functional / LOB teams• Expert in driving deliverythrough collaboration in highly complex, matrixed environment• Possesses strong leadership andnegotiation skills• Excellent communication skills,both written and verbal• Ability to interact with seniorleadership teams in IT and business Preferred Preferred:• Collibra, insurance industryexperience, Azure/Databricks certifications.• Expertise in Python andexperience writing Azure functions using Python/Node.js• Experience using Event Hub fordata integrations.• Hands on expertise in:Implementing analytical data stores in Azure platform using ADLS/Azure datafactory /Data Bricks and Cosmos DB (mongo/graph API)• Experience ingesting using Azuredata factory, Complex ETL using Data Bricks.• Eagerness to learn newtechnologies on the fly and ship to production. Essential Skills: Digital :Microsoft Azure| Data EngineerExperience Required: 8-10
If you’re open to exploring new opportunities or currently in the job market, I’d be happy to connect.
You can reach me directly at (singh.pankajk221@gmail.com).
If this role isn’t the right fit, I’d appreciate it if you could share it with someone in your network who might be interested.
Thank you for your time and consideration.
Regards
Pankaj Singh
Senior Associate
