Title: Hi HN,<p>Gentrace is our new evaluation and observability tool for generative AI (open beta).<p>Generative pipelines are hard to evaluate because outputs are subjective. Lots of developers end up just doing “gut checks” on a few inputs before shipping changes, or they build up a spreadsheet of test cases that they manually run through the pipeline. Some companies outsource filling out the spreadsheet. However, in any of these cases, you end up with a very slow and expensive process for evaluation.<p>At one point, we did this too. Gentrace is the result of a pivot; it was an internal tool we used to automatically grade new PRs as developers shipped changes to generative pipelines that other people thought might be useful.<p>Gentrace makes pre-production testing of generative pipelines continuous and nearly instantaneous. In Gentrace, you:<p>- Import and&#x2F;or construct suites of test data
- Use a combination of AI and heuristic evaluators to grade for quality, hallucination, safety, etc
- Use our interface to correct automated grades or add your own (yourself or a member of your team)<p>Gentrace integrates at a code level for evaluation, meaning we test your generative AI pipeline the way you would test normal code. This allows you to test more than just prompt changes; for example, you can compare models (eg Claude 2 vs GPT-4 vs GPT 3.5 vs Llama 2) or see the effects of additional chained steps (”Rewrite the previous answer in the following tone:”).<p>Here’s a video overview that goes into a bit more detail: <a href="https:&#x2F;&#x2F;youtu.be&#x2F;XxgDPSrTWIw" rel="nofollow noreferrer">https:&#x2F;&#x2F;youtu.be&#x2F;XxgDPSrTWIw</a><p>In production, Gentrace observes for speed, cost, and data flow. It also shows real user feedback as well. We do this by integrating via our SDK at a code level; Gentrace does not proxy requests.<p>Soon, we’ll allow you to convert production data into test cases, allowing customer support to turn bad production generations into “failing tests” for AI teams to make pass.<p>We process interim steps and multiple outputs as well, helping evaluate agent flows &#x2F; chains where the “last output” isn’t always the only thing that matters.<p>There’s been a lot of observability tools published recently. We differ from those by focusing more strongly on blending observability with strong evaluation and by using an SDK rather than a “man-in-the-middle” approach to capturing data (ie Gentrace can be down and your request to OpenAI will still succeed).<p>Within the evaluation landscape, we differentiate by integrating with code (see above for benefits) for capturing generative outputs and by providing a customizable UI workflow for building evaluators. In Gentrace, you start with off-the-shelf automated evaluators and then customize them to your specific task. You also build and run new evaluators on old generative outputs. Finally, you easily override automated evaluators and&#x2F;or blend automated evaluation with evaluation by humans on your team.<p>We also focus on being suitable for business use. We are SOC 2 Type 1 compliant (Type 2 coming shortly), have robust legal documentation around data processing, security, and privacy, and have already passed several vendor legal and security reviews at large technology companies.<p>Our standard usage-based pricing is available on the website: <a href="https:&#x2F;&#x2F;gentrace.ai&#x2F;pricing" rel="nofollow noreferrer">https:&#x2F;&#x2F;gentrace.ai&#x2F;pricing</a><p>If you are building features with generative AI, we would love to get your feedback. You can self-serve sign up (without a credit card) for a 14 day trial here: <a href="https:&#x2F;&#x2F;gentrace.ai&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;gentrace.ai&#x2F;</a><p>We’re available right here for feedback and questions. We’re also available at support@gentrace.ai.<p>Best,
Doug, Vivek, and Daniel
Upvote: