Data Scientist - AI Response Monitoring

About Datagrid Forget everything you know about AI assistants. At Datagrid, we’re building AI agents that actually do the work. We’re a team of passionate, hard-working builders, thinkers, and problem-solvers who are genuinely excited about what we do. Our mission is to supercharge the workday by turning complex data and tedious workflows into simple, automated actions. It’s an incredibly exciting time to join us—we’re growing fast, expanding our platform’s capabilities, and partnering with enterprise customers who want to 10x their teams’ output. We thrive on collaboration and are looking for people who are ready to make a tangible impact. If you want to be part of a team that’s not just talking about the future of AI but actively creating it, you’ve come to the right place.

Our Values At Datagrid, our values guide how we work, build, and grow together. Act with Purpose: Everything we do is tied to our mission. You’ll see the impact of your work as we move quickly to solve meaningful problems for our customers. Own the Outcome: We believe in true ownership. You’ll take responsibility for your projects and see them through to success—empowered to make decisions that drive real results. Clarity without Ego: We value honesty, transparency, and trust. You can expect and provide direct feedback in an environment where candor sharpens our ideas and strengthens our team. Creativity with Purpose: Innovation is central to our culture. Your creative thinking will be valued and directed toward solving real-world challenges and creating lasting impact.

About the role:

This is an opportunity to build the core infrastructure that powers the next generation of AI agents. Our agents must ingest, enrich, and vectorize massive, multi-modal dataset from over 100 customer sources. The core challenge is twofold: how do we do this in a way that is radically cost-efficient, while still allowing Agents to deliver thorough responses in seconds. As a Data Scientist, you will be a key player in solving this problem. You will be responsible for building, managing, and optimizing the entire lifecycle of our agent platform, from the large-scale data ingestion pipelines to the core systems that allow agents to reason and act. You will help build the engine that allows our agents to be not just smart, but also incredibly fast and secure for thousands of enterprise users.

What you’ll do:

As a Data Scientist focused on AI Agent Monitoring & Evaluation, you will be the guardian of quality for our AI agents that serve customers across 100+ integrated systems. You will build the infrastructure that ensures our agents deliver accurate, reliable, and safe responses before our customers experience issues. This role sits at the critical intersection of data science, customer empathy, and production system reliability—where your evaluations directly impact the trust customers place in our AI-powered platform. You'll take ownership of sophisticated evaluation frameworks incorporating LLM-as-a-judge methodologies, create real-time monitoring dashboards + alerts that surface quality degradation before it impacts users, and leverage multi-modal analysis to ensure our visual language models perform flawlessly. This is an opportunity to define what "agent quality" means at scale while working with cutting-edge observability tools like Arize Phoenix and vector databases like Milvus.

What you'll have:

Required Qualifications 3–7+ years of experience in data science, machine learning, or AI evaluation roles with demonstrated expertise in production ML systems Track record of translating customer feedback and business priorities into automated quality metrics and evaluation criteria Strong knowledge of dashboarding and alerting best practices (Preferably in Looker + GCP) Direct experience building, deploying, and monitoring ML/GenAI workflows in production Direct experience convincing leaders to allocate millions of dollars in investments

Bonus Qualifications Solid understanding of vector databases and embedding-based search/monitoring, especially for multi-modal and LLM-based systems (Milvus, Pinecone, Weaviate, etc.). Strong understanding of VectorDBs, LLMs, and Agentic Evaluation. Familiarity with state-of-the-art observability tools for production AI/ML (Arize Phoenix, MLflow, etc.). Experience with LLM evaluation and benchmarking frameworks (LLM-as-a-judge, agentic evaluation, prompt engineering, etc.).

Salary & Benefits Salary Range: $125,000 - $185,000 Generous equity compensation Flexible vacation/time-off policy All U.S. federal holidays observed, plus an additional company-wide Week of Rest in December Competitive benefits package - 100% premium coverage for employees and generous coverage for dependents Work-from-home stipend to support your ideal setup 401(k) plan The base pay range target for the role seniority described in this job description is between $125,000 - $185,000. Final offer amounts depend on multiple factors such as candidate experience and expertise, geographic location, total compensation, and market data. In addition to cash pay, full-time regular positions are eligible for equity, 401(k), health benefits, and other benefits; some of these benefits may be available for part-time or temporary positions.