Evaluation Engineer (AI Assistant)

Evaluation is becoming a crucial part of modern AI development. The ability to measure, refine, and validate AI models is what separates successful AI-powered products from those that fail.

At JetBrains, we are committed to building robust evaluation systems that enable the development of high-quality AI features. Many companies advocate for evaluation-driven development, and industry experts recognize evaluation as a key differentiator in AI. Evaluation is also essential for integrating CI/CD principles into LLM development.

Despite its importance, evaluation is still an evolving field with many uncertainties and areas for optimization. We are looking for a talented and passionate software developer to join our team and help us build cutting-edge evaluation tooling for the IntelliJ Platform and beyond. The ideal candidate thrives in an environment where they can take initiative, work autonomously, and collaborate effectively with others

As part of our team, you will: 

• Design and develop evaluation tools to assess AI-powered features in JetBrains products.
• Empower teams across the company by providing best-in-class evaluation tooling to improve AI models and workflows.
• Explore and implement modern evaluation techniques, adapting them to the JetBrains ecosystem.
• Build APIs that are intuitive, flexible, and scalable.
• Integrate internal tools with third-party systems to track, analyze, and visualize experiments effectively

Requirements: 

• 3+ years of experience in Java/Kotlin and a solid understanding of software development best practices.
• A strong understanding of development best practices and hands-on experience with CI/CD pipelines.
• Excellent communication skills, with the ability to articulate ideas clearly and work effectively with different teams.
• A strong problem-solving mindset, with a passion for creating tools that empower developers and enhance their workflow

Preferable: 

• Experience developing AI-driven functionality or working with AI-powered tools