Contra Labs Research · April 2026·No. 01·Appendix

Appendix

Methodology

Network
1.5M+ independent professional creatives
Platform earnings
$250M+ across design, video, dev, content
Total judgments
~15,000
Pairwise judgments
5,940
Scalar ratings
5,940
Qualitative responses
3,675
Models evaluated
12 frontier
Domains
Landing pages · Desktop apps · Ad images · Brand assets · Ad video
Phases
Ideation · Mockup · Refinement
Per-domain coverage
5 evaluators × 6 tournaments × 2 tasks
Pairwise model
Bradley-Terry → ELO
Agreement metric
Kendall's W
Qualitative coding
Two-stage GPT-4o thematic pipeline (blinded)

Model Types and Models Evaluated

Model TypeModels Evaluated
Text to image / Image to imagegpt-image-1.5, gemini-3-pro-image-preview, seedream-4.5, flux-2-pro
Image to videoveo-3.1, kling-v3.0-pro, seedance-v1.5-pro, grok-imagine-video
Text to code / Code to codeclaude-opus-4.6, gemini-3.1-pro-preview, gpt-5.3-codex, qwen3.5-397b-a17b
Twelve frontier models tested across three capability families.

Participants

Twenty-eight independent professional creatives from Contra's network served as evaluators across the five domains. Each evaluator was selected based on skillset and the generative model category most relevant to their workflow.

NameProfileDomainOne-linerSkillsEarnings
Ale Arrazolacontra.com/alejandro_arrazola_Ad ImagesPerformance Creative · Video, Motion, Meta Ads + AIVideo Editing, Motion Design, Digital Marketing, Growth Marketing, Animation$25k+
Alejandra Tellocontra.com/aletelloAd ImagesVideo Growth Partner | YouTube, Reels & Ads That WorkVideo Editing, Content Creation, Video Production, Social Media Marketing$25k+
Artyom Aghinyancontra.com/artyom_aghinyanAd ImagesCrafting visuals that make an impact.Graphic Design, Brand Design, Logo Design, Ad Design, Social Content, Email Marketing Design$25k+
Joao Paulo Bastoscontra.com/jota_pe_design_lonxhmrlAd ImagesCreative Director & Brand StrategistMotion Design, Brand Design, Animation$1k+
Stas Tatarinovcontra.com/stas_tatarinov_5hmx495lAd ImagesAI Image & Video Producer | AI Ads | AI UGCAI Video Production, Video Editing, AI Art$5k+
Alexandru Köllocontra.com/akollo_idca1gjaAd VideoVideo Producer & Audio DesignerVideo Editing, Video Production, AI Video Production, Graphic Design$25k+
Jesse Caliscontra.com/jesse_x61efo2jAd VideoLet's tell stories togetherVideo Editing, Videography, Animation$10k+
Lynn Zengcontra.com/lynnzengAd VideoGenerative AI Director | Video & ArtAI Art, AI Video, Music Production, Brand Design$5k+
Mont Carvercontra.com/montcarverAd VideoEthereal Filmmaking, Photography & Video EditingVideo Editing, AI Art, Photography & Filmmaking
Sonia Ciocancontra.com/soniaciocanAd VideoVideo Producer & Designer | UGC + Social ContentVideo Editing, Motion Design, Graphic Design$25k+
Tanner Notchcontra.com/tanner_notch_ufyik06sAd VideoMotion Design, 3D & AI-Video SpecialistMotion Design, Video Editing, 3D Design$10k+
Brian Alvaradocontra.com/ZeroSevDesignsLanding PageFramer Expert & CRO Specialist for B2B SaaS Framer WebsitesFramer Design & Development, Web Design$100k+
Evgeny Mishincontra.com/evgeny_mishinLanding PageWebflow Developer & Web DesignerWeb Design, Low-Code/No-Code Dev, Webflow Development$25k+
Hakeem Framercontra.com/studioduoLanding PageTop Framer Expert Developer & Website Designer for StartupsFramer Design & Development, Web Design & Development$100k+
Lynnecontra.com/lin_2ccjupm9Landing PageBold, modern design made to stand out — and drive resultsBrand Design, Photography, Motion Design$1k+
Zack Bujaziacontra.com/zackbujaziaLanding PageWebflow expert focused on design & strategyWebflow Development, Brand Design, Web Design & Development$50k+
Daniel G Brightcontra.com/danielgbrightDesktop AppsMake your company look inevitableWeb Design, Framer Design, Framer Development$250k+
Kamil Bettersidecontra.com/better_sideDesktop AppsMobile Apps & MVPs | Building at startup speedFramer Design, Framer Development, Software Engineering$50k+
Divan Rajcontra.com/divanrajofflDesktop AppsProduct Designer · AI · Motion · No-CodeUI & Product Design, Web Design$25k+
Jithin Kumarcontra.com/jithinkumarDesktop AppsGrowth-Driven Website Specialist | Framer ExpertWeb Design, Product Design$100k+
Jacquelyn Conn (Demshick)contra.com/jacquelyn_demshickDesktop AppsMultidisciplinary designer and problem solverUX & Graphic Design, User Research$25k+
Cara Elliscontra.com/onyourtiptoesBrand AssetsDesigning for connectionGraphic & Web Design, Brand Design$50k+
Darya Batyrcontra.com/daryaBrand AssetsCreative partner in your brand's successBrand Design, Graphic Design, Logo Design, Social Content Design$100k+
Diogo Ferreiracontra.com/diogocferreiraBrand AssetsHelping brands to be unique and impactfulArt Direction, Brand Design, Visual Design$5k+
Elsa Toussaintcontra.com/elsatoussaintBrand AssetsElsa ToussaintBrand Design, Graphic Design, Logo Design$25k+
Julie Andriacontra.com/bonjour_49rf9rooBrand AssetsGreat designs for great companiesBrand Design, 3D Design, Framer Design$10k+
Sabrina Polancocontra.com/sabrinapolancoBrand AssetsBranding for DTC wellness brandsBrand Design, Web Design, UI/UX Design$100k+
sana. stdiocontra.com/sanaBrand AssetsCrafting fun & bold brand identitiesWeb Design, Graphic Design$10k+
Evaluator panel by domain, with Contra profile, professional positioning, skill set, and lifetime platform earnings.

Domain insights

Desktop Apps

Ideation

ModelWin RateKrippendorff αFriedman p-value
Claude 4.60.6000.1830.0010
GPT 5.30.0860.5680.0010
Gemini 3.10.2570.1480.0010
Qwen 3.50.057−0.0660.0010

Mockup

ModelWin RateKrippendorff αFriedman p-value
Claude 4.60.4000.1010.3231
GPT 5.30.2290.1160.3231
Gemini 3.10.171−0.0310.3231
Qwen 3.50.2000.2310.3231

Refinement

ModelWin RateKrippendorff αFriedman p-value
Claude 4.60.171−0.0420.1134
GPT 5.30.4000.1190.1134
Gemini 3.10.2000.2430.1134
Qwen 3.50.2290.2340.1134

Landing Pages

Ideation

ModelWin RateKrippendorff αFriedman p-value
Claude 4.60.533−0.0600.0000
Gemini 3.10.367−0.0580.0000
GPT 5.30.1000.0750.0000
Qwen 3.50.000−0.0430.0000

Mockup

ModelWin RateKrippendorff αFriedman p-value
Claude 4.60.3000.0720.3697
Gemini 3.10.4670.0400.3697
GPT 5.30.167−0.0110.3697
Qwen 3.50.0670.1650.3697

Refinement

ModelWin RateKrippendorff αFriedman p-value
Claude 4.60.300−0.0760.4678
Gemini 3.10.2670.0390.4678
GPT 5.30.2330.2090.4678
Qwen 3.50.2000.0260.4678

Brand Assets

Ideation

ModelWin RateKrippendorff αFriedman p-value
Gemini 3 Image0.3810.5900.0010
GPT Image 1.50.3330.0540.0010
Seedream 4.50.1190.1510.0010
Flux 2 Max0.1670.1350.0010

Mockup

ModelWin RateKrippendorff αFriedman p-value
Gemini 3 Image0.5000.0410.0015
GPT Image 1.50.2080.2800.0015
Seedream 4.50.1670.2360.0015
Flux 2 Max0.1250.7700.0015

Refinement

ModelWin RateKrippendorff αFriedman p-value
Gemini 3 Image0.417−0.0310.0033
GPT Image 1.50.3610.3530.0033
Seedream 4.50.1390.0490.0033
Flux 2 Max0.0830.3430.0033

Ad Images

Ideation

ModelWin RateKrippendorff αFriedman p-value
GPT Image 1.50.3330.4760.0010
Seedream 4.50.2330.1320.0010
Gemini 3 Image0.2330.4330.0010
Flux 2 Pro0.200−0.0540.0010

Mockup

ModelWin RateKrippendorff αFriedman p-value
GPT Image 1.50.4000.1830.0056
Seedream 4.50.2330.0330.0056
Gemini 3 Image0.2670.1410.0056
Flux 2 Pro0.100−0.1340.0056

Refinement

ModelWin RateKrippendorff αFriedman p-value
GPT Image 1.50.2330.1120.0398
Seedream 4.50.3670.3430.0398
Gemini 3 Image0.1330.3300.0398
Flux 2 Pro0.2670.3990.0398

Ad Video

Ideation

ModelWin RateKrippendorff αFriedman p-value
Grok Video0.1110.3750.0010
Veo 3.10.4170.1950.0010
Kling 3.00.2780.2730.0010
Seedance 1.50.1940.3100.0010

Mockup

ModelWin RateKrippendorff αFriedman p-value
Grok Video0.1940.5020.5446
Veo 3.10.3330.1300.5446
Kling 3.00.3330.0680.5446
Seedance 1.50.1390.2260.5446

Refinement

ModelWin RateKrippendorff αFriedman p-value
Grok Video0.361−0.0420.1375
Veo 3.10.0830.1760.1375
Kling 3.00.2220.4170.1375
Seedance 1.50.3330.4010.1375

References

  1. [1]Treffinger, D. J., & Poggio, J. P. (1972). Needed research on the measurement of creativity. The Journal of Creative Behavior, 6(4), 263–267.
  2. [2]Lee, H. K. (2022). Rethinking creativity: Creative industries, AI and everyday creativity. Media, Culture & Society, 44(3), 601–612.
  3. [3]Dawid, A. P., & Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C, 28, 20–28.
  4. [4]Inel, O., Khamkham, K., Cristea, T., Dumitrache, A., Rutjes, A., Ploeg, J., Romaszko, L., Aroyo, L., & Sips, R.-J. (2014). CrowdTruth: Machine-human computation framework for harnessing disagreement in gathering annotated data. 486–504.
  5. [5]Basile, V., Fell, M., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M., & Uma, A. (2021). We need to consider disagreement in evaluation. 15–21.
  6. [6]Thanh-Tung, H., & Tran, T. (2020). Catastrophic forgetting and mode collapse in GANs. In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1–10). IEEE.
  7. [7]Bommasani, R., Creel, K. A., Kumar, A., Jurafsky, D., & Liang, P. S. (2022). Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35, 3663–3678.
  8. [8]O'Toole, K., & Horvat, E. A. (2024). Extending human creativity with AI. Journal of Creativity, 34(2), 100080.
  9. [9]Casey, E., & O'Brien, D. (2020). Sociology, sociology and the cultural and creative industries. Sociology, 54(3), 443–459.
  10. [10]Kulesz, O. (2018). Culture, platforms and machines: the impact of artificial intelligence on the diversity of cultural expressions. Intergovernmental committee for the protection and promotion of the diversity of cultural expressions, 12(2).
Back to paper

Connecting with the missing signal: taste

Contra connects top creative minds with AI teams training models to understand taste. This is expert input, not crowd labor. It's the creative layer powering the next generation of AI.

Designers

Writers

Marketers

Engineers

Social Media Experts

Video Editors & Animators

Music & Audio Engineers

1.5M+

creative experts

400+

Skills and tools represented

$250M+

verified expert earnings

Connecting with the missing signal: taste

Contra connects top creative minds with AI teams training models to understand taste. This is expert input, not crowd labor. It's the creative layer powering the next generation of AI.

Designers

Writers

Marketers

Engineers

Social Media Experts

Video Editors & Animators

Music & Audio Engineers

1.5M+

creative experts

400+

Skills and tools represented

$250M+

verified expert earnings

Connecting with the missing signal: taste

Contra connects top creative minds with AI teams training models to understand taste. This is expert input, not crowd labor. It's the creative layer powering the next generation of AI.

Designers

Writers

Marketers

Engineers

Social Media Experts

Video Editors & Animators

Music & Audio Engineers

1.5M+

creative experts

400+

Skills and tools represented

$250M+

verified expert earnings