Fram

Overview of job

Duration: Full-time for 6 months

The Role

• Your primary mission will be to own the design and implementation of our LLM Evaluation Platform, a critical system that will serve as the quality gate for all our AI features. You will be a key builder on a new initiative, working alongside dedicated Data Engineering and DevOps experts to deliver a tangible, high-impact platform.
• This role is for a hands-on engineer who thrives on building robust systems that provide leverage. You will be fully empowered to own the implementation and success of this project.

Responsibilities

• Design and develop the core backend systems for a new LLM Evaluation Platform, leveraging Arize Phoenix as the foundational framework for traces, evaluations, and experiments.
• Architect and implement the observability backbone for AI services, integrating Phoenix with OpenTelemetry to build a centralized system for logging, tracing, and evaluating LLM behavior in production.
• Design and implement a CI/CD framework for versioning, testing, and deploying prompt-based logic and LLM configurations, ensuring reproducible and auditable deployments across all AI features.
• Make pragmatic technical decisions that prioritize business value and delivery speed, in line with an early-stage startup environment.
• Work closely with the Data Science team to understand their workflow and ensure the platform meets their core needs for experiment tracking and validation.
• Help define and document the initial technical patterns for MLOps and model evaluation, creating the foundation for future development.

• Competitive salary;
• Mon-Fri, flexible remote working;
• English, professional working environment.