AIML Engineering Cloud Data Platforms
Location Raritan NJ
Â
Role Summary
We are looking for an experienced Databricks Azure professional with strong expertise in cloudscale data engineering lakehouse architecture Agentic AI and Databricks Agent Bricks The ideal candidate will have 7 or more years of experience designing building and operationalizing enterprise data and AI platforms on Azure Databricks with handson exposure to generative AI solutions retrievalaugmented generation vector search AI agents model serving MLflow Unity Catalog and productiongrade governance practices
Roles Responsibilities
Design develop and maintain scalable data engineering and AIML solutions on Azure Databricks using Delta Lake Unity Catalog Databricks Workflows Databricks SQL and MLflow
Build and deploy Agentic AI applications using Databricks Agent Bricks Mosaic AI Agent Framework Knowledge Assistants Supervisor Agents toolcalling agents and multiagent orchestration patterns
Develop retrievalaugmented generation solutions using Databricks Vector Search embeddings model serving endpoints foundation models and governed enterprise knowledge sources
Architect lakehouse solutions across ingestion transformation orchestration governance security performance optimization and monitoring layers
Integrate Azurenative services such as Azure Data Lake Storage Azure Data Factory Azure Key Vault Azure DevOps Microsoft Entra ID Azure Functions and Azure OpenAI where applicable
Implement robust CICD pipelines for Databricks notebooks jobs bundles ML models and AI agent deployments using Azure DevOps or equivalent tools
Establish data and AI governance standards using Unity Catalog access controls lineage audit logging model registry and secure serving patterns
Optimize Spark workloads Delta tables SQL warehouses serverless compute cluster policies job performance and cost utilization
Collaborate with data engineers ML engineers architects product teams and business stakeholders to convert business requirements into reliable data and AI capabilities
Define best practices reusable frameworks coding standards deployment patterns and operational runbooks for Databricks and Agentic AI platforms
Support production operations incident resolution root cause analysis monitoring SLA management and continuous improvement for data and AI workloads
Mentor junior engineers and contribute to knowledge sharing technical design reviews proofofconcepts and platform modernization initiatives
MustHave Skills
7 years of overall experience in data engineering analytics engineering AIML engineering or cloud data platform engineering
Strong handson experience with Azure Databricks Apache Spark PySpark Spark SQL Delta Lake Delta Live Tables Databricks Workflows Databricks SQL and Lakehouse architecture
Practical experience with Databricks Agent Bricks Agentic AI patterns AI agents tool calling multiagent systems Knowledge Assistants Supervisor Agents and governed AI application development
Handson experience building generative AI solutions such as RAG applications LLM chains prompt engineering workflows vector search implementations embeddings pipelines and model serving solutions
Strong understanding of MLflow for tracking model lifecycle management tracing evaluation monitoring and deployment of ML and GenAI workloads
Experience with Unity Catalog data governance access control catalogschematable permissions lineage auditability and secure AIdata sharing patterns
Proficiency in Python SQL PySpark data modeling performance tuning API integration and productiongrade software engineering practices
Experience with Azure services including ADLS Gen2 Azure Data Factory Azure DevOps Azure Key Vault Microsoft Entra ID Azure Monitor and Azure OpenAI or comparable LLM services
Strong knowledge of CICD infrastructureascode concepts Gitbased development Databricks Asset Bundles environment promotion and release management
Ability to troubleshoot complex data pipelines Spark performance issues job failures security constraints and AI model or agent behavior in production environments
Good understanding of Responsible AI data privacy model evaluation hallucination mitigation guardrails prompt safety and enterprise AI governance
Excellent communication stakeholder management documentation and technical leadership skills
GoodtoHave Skills
Experience with LangChain LangGraph OpenAI Agents SDK LlamaIndex MCP servers AI Gateway or custom agent frameworks
Experience with streaming data pipelines using Structured Streaming Event Hu”
Â
|
Neha Chaudhary
Survey Link:Â The survey is anonymous, and the information collected will be analysed in aggregate to support initiatives that improve the total experience across Compunnel.
|
Â
—
