Sr Quality Specialist | Palo Alto CA (Hybrid)

Job Title: Sr Quality Specialist

Location: Palo Alto CA (Hybrid work model, mainly remote with some visits onsite)

Duration : Long Term Contract , || Local to CA || Contract

Interview: Virtual

Visa :: USC/GC/H4/GC EAD

Purpose and Objective:

SAP America, Inc. seeks a Senior Quality Specialist at our Palo Alto CA location to work in a cross functional product team with special focus to quality assurance and test related tasks within the product creation process. Our product is an AI based copilot targeting SAP internal and external consultants and IT staff.

Expectations and Tasks:

Define quality and test strategies for products respecting defined process and product standards. Analyze, define, implement, and execute quality and test related activities according to the set quality and test strategies. Create a detailed test design including test system landscape definition and product standards.

Coordinate delivery deadline in an Agile development environment. Coordinate build deployments monitoring test results.

Prepare tests including customizing and master data. Author and execute manual and automated tests. Analyze test results (issues) and take appropriate follow-up actions.

Serve as a subject matter expert for the team in terms of quality assurance and test related processes, methods, and tools. Initiate process and tool improvements.

Qualifications/Skills and Competencies Experience:

Automation of Assistant Interactions

Experience with tools like Playwright, Selenium, or custom scripting to simulate user queries and validate assistant responses.
Ability to design end-to-end test flows for conversational interactions and service orchestration.
Familiarity with automating RAG-based assistant workflows and validating retrieval and generation steps.

Ticket Triage and Management

Proficiency in using Jira, ServiceNow, or internal dashboards for managing assistant-related issues.
Skills in categorizing, prioritizing, and routing tickets based on severity and impact.
Understanding of incident lifecycle and escalation protocols.

Metrics and Monitoring

Experience tracking and reporting ticket response time, resolution time, and assistant performance metrics.
Ability to define and monitor SLA compliance and user satisfaction indicators.
Familiarity with conversational QA metrics such as BLEU, ROUGE, and relevance scoring.

User Prompt Validation for Negative Cases

Experience designing test cases for negative, edge, and adversarial prompts.
Familiarity with retrieval fallback logic and response suppression when no relevant content is found.
Ability to validate hallucination prevention and guardrail enforcement in RAG-based systems.
Exposure to prompt-response evaluation frameworks and semantic relevance scoring.

Technical experience must involve the following:

Compliance of various Product Standards related to Accessibility, Security, Performance, and functional correctness

Work on cross-functional teams to understand various system requirements and with different teams such as Security, Performance and Automation and DevOps teams