The Frontier Design Benchmark is live.

Read on XLinkedIn

Methodology

Leaderboard

Methodology

Contra Labs · 3 min read

Design Taskset

We included 33 tasks as an initial set in Frontier Design. Our goal is to ensure the realism, complexity, and diversity of these tasks. We based 27 of them on real design jobs posted on Contra, rewritten for fictional clients. After we analyzed the 4.6k website projects on Contra in the past year, we summarized that a typical brief covers:

  • client information
  • target audience
  • website pages and user flows
  • branding assets and other business context
  • practical requirements, such as being mobile-first, loading quickly, being accessible, etc.
An example task

We created tasks including these elements, while ensuring the diversity among tasks. The 27 job tasks also cover these output types, including landing pages, web apps, and mobile apps. Examples include a festival site that sells tickets and an audio tour for a museum.

Notably, we also included 6 design tasks elicited from startup partners. Each task is a real design project the company hired a Contra designer to work on. These briefs come with the company’s own materials, such as brand assets, files from their current website, product images and slide decks.

Model Rollouts & Human Golden Designs

To collect model rollouts on the design tasks, we partnered with BenchFlow (benchflow.ai). BenchFlow runs an AI agent on each task in a sandbox and records its output, cost and full trajectory. We used OpenRouter as the model provider. Wherever we could, we ran each model in its own lab’s agent harness.

In addition to model rollouts, we worked with professional designers on Contra to create human golden outputs for the tasks. Designers were first asked to create a moodboard of 3 design directions. We reviewed their moodboards and gave feedback. They then built the site based on the moodboard with tools of their choice. We collected these human golden outputs to show what high-quality human design looks like, and to set a goal for AI models to reach.

Evaluation Metrics

Our benchmark reports a combination of three evaluation metrics: expert pairwise preference, the Design Critique Rubric, and an AI Slop Detector.

Expert Preference & Elo Rating

So far we have recruited 35 designers, casting more than 3.6 thousand votes. Each designer is given pairs of designs on the same prompt, answers a general preference question, and optionally explains their choice, covering color, typography, layout, usability and prompt adherence. We turn the votes into a rating per model with a Bradley-Terry model with Elo scores.

We measured repeatability and ranking coherence. We split the designers into two random halves and ranked the models with each half separately. The two rankings agree at Spearman 0.92, averaged over 200 splits.

Design Critique Rubric

Just an Elo score doesn’t tell you anything about how to improve the model. We acknowledge that there are design principles all good designs follow.

In Frontier Design, we curated a list of 35 high quality design rubrics based on widely acknowledged design principles. These design rubrics capture common design mistakes such as using too many fonts or spacing inconsistencies. Grading the design outputs on these rubrics can establish a baseline of design without mistakes.

We built a rubric library for design critique that satisfies the following:

  1. Predict designers’ preference of a model.
  2. Identify the specific issues with designs that a designer would raise.

We used our Human Creativity Benchmark dataset to curate and validate our rubric list. We also recruited professional designers to judge the design, and then check their alignment with the rubrics.

In our analysis, the results of our design rubrics against AI model outputs have a high correlation with the pairwise ranking of models (r = 0.94, Spearman 0.95).

The design rubric
Rubric scores

Slop Detector / AI Slop Patterns

As part of the Frontier Design benchmark, our research team used statistical analysis on AI model outputs to create an internal Slop Detector engine that identifies AI model signs and tells. We supplemented this engine with a hand-curated AI slop pattern list that our design team has curated over time. As a result, when using our Slop Detector to score a website, we can receive an AI-likelihood score, as well as all design details that support the score.

AI slop habits
Sign inGet in touch