Splunk Engineer

Department: IT Services

Title: Splunk Engineer

Location: 251 W. Washington St., 6th floor, Phoenix, AZ 85003-2295

Notes from Manager :
We would probably want to bring this position on for a full-time 1Yr contract (2080 hours) and we may opt extend if it works out well.  (Monday – Friday 8AM-5PM or similar schedule)

I think this position would be mostly Hybrid if not completely remote,  however we do like to meet folks face-to-face every now and then. Especially for an interview, so ideally, they would need reside near the Phoenix area or be able to commute in when needed.

Primarily we are looking to see if a Splunk Admin/developer is even a prevalent skillset that we could find someone with this specialization…

Additional skills that are worthwhile to us:   Software development, API Integrations,  Business Analytics,  other reporting,monitoring or management systems such as Solarwinds or Catalyst Center would all be a “plus”.

Primarily as Ian listed, someone who is an expert with Splunk, Splunk development and management.

Job Description:

About the position

Onboard new systems and data sources into the City’s enterprise Splunk Cloud environment and developing dashboards, alerts, and analytics to improve operational visibility, security posture, and service reliability. This role is highly technical and requires strong expertise in Splunk administration, data onboarding, and SPL (Search Processing Language) development.

Key Responsibilities

Onboard new systems, logs, and data sources into Splunk, ensuring proper parsing, field extractions, CIM compliance, and data normalization.
Configure and maintain forwarders, ingestion pipelines, and data routing.
Build advanced dashboards, visualizations, and analytics for operational, security, and business use cases.
Develop complex SPL queries, macros, lookups, and scheduled searches.
Troubleshoot ingestion issues, search performance, and data quality problems.
Partner with network, server, application, and security teams to define log requirements and actionable monitoring.
 

Minimum Qualifications

Experience administering and engineering Splunk Enterprise or Splunk Cloud in a medium-to-large environment.
Strong proficiency with SPL for analytics and troubleshooting.
Demonstrated experience onboarding new systems or applications into Splunk.
Experience building dashboards with Splunk Dashboard Studio or Classic Editor.
Knowledge of log ingestion formats (syslog, JSON, XML), data parsing, and field extraction.
Understanding of IT infrastructure fundamentals (servers, networks, firewalls, cloud services).
Experience with Linux command line and Splunk Universal/Heavy Forwarder management.
 

Preferred Qualifications

Experience with automation or scripting (Python, PowerShell).
Experience with Enterprise Security (ES) or ITSI modules.
Familiarity with indexer clustering, search head clustering, and distributed Splunk architectures.
Experience implementing CIM compliance and data models.
 

Ideal Candidate

Strong analytical, troubleshooting, and visualization skills.
Ability to work collaboratively with infrastructure, application, and security teams.
Strong communication skills and the ability to translate technical findings into actionable insights.