AI Automation Engineer with Evaluation exp || Remote

Remote

LLM Coding Benchmarks, Evaluation Infrastructure, Python Automation and Test Harness EXP MUST

Must‑Have Technical Skillsa

Strong software engineering background, with real experience building automation,

developer tooling, or test and validation systems.

• Proficient in at least one general-purpose language such as Python, Java, or JavaScript

— the specific language background is flexible.

• Solid working knowledge of Git, including how branches, history, and working trees

behave, and of containerization with Docker.

• Experience with APIs, development environments, CI/CD pipelines, and typical

engineering workflows.

• Understanding of how AI, LLM, or agent evaluation works and where it goes wrong, such

as why a judge can be consistent but still wrong, why a single run can mislead, and how

benchmark contamination happens.

• Able to troubleshoot technical problems, think clearly about whether a measurement is

valid, and analyze results carefully.

• Hands-on experience using AI coding tools and agentic harnesses such as Claude

Code, Devin, or Cursor, and command of the best practices for working with them

eJectively