Loading opportunity…
All opportunities
Mercor 
Anyone AI
Terac
Braintrust
micro1
Turing
Meridial
Verita AI
Handshake AI
Sovrano AI
AI Training · hourly
SWE-Bench Task Auditor
Evaluate the quality, correctness, and reproducibility of software engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric based written feedback.…
PLATFORMMercor
LOCATIONRemote — United States
PAY$70–$90/hr
SCHEDULE40 hrs/week
LAST CHECKED11 Nov 2026
THE OPPORTUNITY
Put your skills to work.
Evaluate the quality, correctness, and reproducibility of software engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric based written feedback.…
- Commitment
- hourly
- Hours per week
- 40
- Field
- Software Engineering
- Open slots
- 3
- Eligible countries
- USA
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
Basic Qualifications
Preferred Qualifications
- 3+ years professional software engineering
- Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
- Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
- Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
- Familiarity with SWE-Bench (Verified) or similar repository benchmarks
- Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
- Prior code-review or task-grading experience
Role details are copied from the Mercor listing and can change. Check the platform before you apply.
Relevant skills
Software EngineeringSWEBench Task Auditor
A few things to check
- Location and experience requirements.
- Schedule, compensation, and assessment process.
- Current availability on Mercor.
KEEP EXPLORING
Similar opportunities.
Related roles across other platforms, selected by field, title and skills.
AI Developer Trace Task Auditor
Evaluate the quality and correctness of AI assisted software development traces used to train and evaluate a frontier AI lab's models. You'll assess end to end coding sessions produced with AI assisted developer tools — judging correctness, workflow soundness, and reasoning — and provide clear, rubric based written…
Software EngineeringAI Developer Trace Task AuditorRemote collaboration
Remote — United States
$70–$90/hr
Anyone AISenior Software Engineer – Open Source & SWE-Bench Evaluation
Anyone AI is recruiting experienced Software Engineers for a specialized project focused on reviewing and evaluating real-world software engineering tasks derived from open-source repositories.The work involves assessing whether coding tasks based on real GitHub issues and pull requests are technically sound, reproduci
Software Engineering
Argentina · remote
$65/hr
Finance and Accounting Professionals: Business Operations and Software Workflows
We are running a paid interview study on the software tools used for everyday finance and accounting workflows. We need business operations professionals to walk us through their daily processes and evaluate their current tech stack.
Finance and Accounting ProfessionalsBusiness Operations and Software WorkflowsRemote collaboration
United States · remote
$71 one-time
BraintrustLead Security Engineer (US-Remote)
Handl Health lists this direct_hire opportunity through Braintrust. Skills include Security Engineering, Cloud Security, Security Auditing.
Security EngineeringCloud SecuritySecurity Auditing
United States only
$185,000–$215,000/yr
micro1AI Trainer (SWE)
Skills include Software Engineering, AI training. 100 openings shown by micro1.
Software EngineeringAI training
Remote · eligibility varies
$60–$120/hr
SWE Bench Engineer - Golang
Analyze and triage GitHub issues for open-source software projects.
Go
Remote · eligibility varies
$25 per approved task
Technical Task Auditor
Audit technical tasks for accuracy, realism, reproducibility and tests. Specialties include GPU kernels, security, machine learning, AWS, Kubernetes and SWE-Bench.
Eligibility varies
$60/hr · varies by task
Verita AITask & Evaluation Specialist
What this role does Design realistic, difficult shopping-related tasks (e.g. meal planning, restocking, gifting, travel prep) that stress-test AI models, then build the rubrics used to grade how well a model performs them. Who we're looking for Subject-matter expertise in retail, e-commerce, consumer shopping behavior,
Software Engineering
Remote · eligibility varies
$60/hr
Energy Auditor
Handshake is looking for building energy and auditing professionals to support AI research through flexible, hourly contract work. This is not a traditional job. You'll draw on your hands-on experience with building energy modeling, energy auditing, and performance benchmarking to evaluate AI-generated content and…
Energy AuditorRemote collaboration
Remote · eligibility varies
Up to $80/hr
Sovrano AISoftware Engineering Expert, AI Trainer
Fully remote, project-based work checking how well AI coding agents handle real software engineering. You write tasks based on your own experience and grade where the agent gets it wrong. $40 to $125 USD an hour, at least 10 hours a week, as an independent contractor. For working software engineers in the US, UK, Irela
Software engineeringCode reviewDebugging
Remote (US, CA, UK, AU, IE)
$40–125/hr