AI Specialist – Evaluations (Evals) for AI Agents

About the role

We are seeking an AI Engineer to design, build, and maintain an evaluation (evals) system for our internal AI agents that automate accounting, financial, and support processes. You will own the measurement of correctness, reliability, and safety, setting the quality bar for every agent release.

Key responsibilities

• Design and implement automated evals to assess accuracy, reliability, safety, and performance of internal AI agents.
• Define clear metrics and pass/fail criteria for each use case.
• Create and curate test datasets reflecting real accounting and financial scenarios.
• Integrate evals into CI/CD pipelines to catch regressions before production.
• Maintain and evolve the evals system as models, agents, and business needs change.
• Analyze results, identify failure patterns, and provide concrete improvement recommendations.
• Collaborate with engineering and product teams to prioritize evaluation targets.
• Document methodologies, results, and quality standards.

Required profile

• Strong experience in Python development.
• Experience with large language models, AI agents, or prompt‑based systems.
• Analytical, rigorous mindset with strong attention to detail.
• Ability to translate evaluation outcomes into actionable decisions.
• Familiarity with version control (Git) and CI/CD workflows.
• Professional proficiency in English for technical documentation.

Required skills

• Python
• Git
• CI/CD pipelines
• Large language models (LLMs)
• AI agents / prompt engineering
• Evaluation methodologies (metrics, test sets, regression detection)

What we offer

• Monthly salary of US$3,000–US$5,000 based on experience.
• Full ownership of a critical, growing area.
• On‑site work at our Panama City office.
• Direct impact on the quality of AI products used daily.