Research Ops

Data & Evaluation Specialist (Agent Eval / RLHF)

$600 – $1,000 / moContractRemoteMid-level

Make our agents measurably better. Build datasets, grade agent trajectories, and run evaluations that decide what ships.

What You Will Do

Curate and annotate datasets for planning, tool-use, and safety behaviors.
Grade agent trajectories with rubrics; identify systematic failure modes.
Run regression evals before every release; publish scorecards.
Design adversarial test suites probing edge cases and misuse.
Collaborate with Research Engineers to convert findings into training/eval improvements.

Requirements

2+ years in data annotation ops, QA, ML data work, or research assistance.
Rigorous attention to detail; able to follow and improve rubrics.
Basic scripting (Python or spreadsheets-to-SQL level) for analysis.
Clear written communication for failure reports and guidelines.
Background or strong interest in AI safety/alignment practices is a plus.

Skills & Tools

Rubric DesignTrajectory GradingDataset CurationFailure AnalysisPython (basic)Documentation

What We Offer

Remote-FirstWork from anywhere in your announced time zone band. Async by default.

Meaningful EquityEvery full-time role ships with an equity package. We win together.

Learning BudgetAnnual budget for courses, conferences, compute credits, and books.

Mission TimeDedicated hours each month to point agents at a problem you personally care about.

How to Apply

Send your CV, portfolio, and a short note on why this mission is yours to hrd@rinejourney.xyz with the subject line “Application — Data & Evaluation Specialist (Agent Eval / RLHF)”. Applications are reviewed on a rolling basis — no portals, no ghosting.

One click opens Gmail with a pre-filled application to our HR team.

Apply via Gmail ↗or use your default mail app