Remote | JSON & Rubric Evaluation Specialist — $45–$170/hour

We are sharing a specialised part-time consulting opportunity for detail-oriented evaluators with experience interpreting JSON-formatted outputs, applying structured grading rubrics, and assessing human data or AI evaluation tasks consistently.

This urgent project supports a short-term evaluation sprint focused on grading AI-generated outputs against detailed task criteria. Selected professionals will review written instructions, interpret JSON-based responses, and assess each output independently using predefined rubrics. Strong accuracy, consistency, and careful attention to small details are essential.

Key Responsibilities

JSON Output Review

• Read, parse, and interpret AI-generated outputs presented in JSON format
• Understand nested fields, structured values, and relationships between output components
• Identify missing, inconsistent, malformed, or incorrectly represented information
• Assess whether each response satisfies the underlying task instructions

Rubric-Based Evaluation

• Review detailed task descriptions and corresponding grading criteria
• Evaluate each JSON output against predefined rubric requirements
• Apply scoring standards consistently across high-volume task batches
• Distinguish minor issues from material failures based on the provided criteria
• Document grading decisions clearly where explanations are required

Quality & Consistency

• Maintain strong attention to detail across repetitive evaluation workflows
• Identify edge cases, ambiguous outputs, and subtle inconsistencies
• Follow project instructions precisely without introducing unsupported assumptions
• Escalate unclear rubric language or conflicting task requirements appropriately
• Incorporate calibration feedback into subsequent evaluations

Independent Assessment

• Complete all assigned assessments without assistance from AI tools
• Manage tasks independently within short turnaround windows
• Review completed work before submission for accuracy and completeness
• Maintain reliable quality throughout the duration of the sprint

Ideal Profile

Strong candidates may have:

• Experience reading and interpreting JSON files or JSON-formatted data
• Familiarity with structured grading rubrics and evaluation criteria
• Prior experience as a grader, reviewer, rater, annotator, or quality evaluator
• Strong written reasoning and attention to detail
• Ability to apply complex instructions consistently across multiple tasks
• Comfort working within fast-paced, high-volume evaluation workflows
• Availability to contribute approximately 20–40 hours per week
• Ability to complete assigned tasks within approximately 12–24 hours
• Current residence in the United States

Educational Background

• A degree in computer science, data analysis, information systems, research, operations, linguistics, or a related discipline may be helpful
• Formal technical qualifications are secondary to practical JSON fluency and strong evaluation experience
• Training in quality assurance, data annotation, structured review, or AI evaluation may strengthen an application
• Equivalent professional or project-based experience may also be considered

Nice to Have

• Extensive experience designing or applying detailed rubrics
• Previous work on human data or AI model evaluation projects
• Familiarity with JSON syntax, nested objects, arrays, and structured data validation
• Background in quality assurance, content review, operations, or data labelling
• Experience completing calibration exercises and maintaining reviewer consistency
• Prior experience working on similar structured evaluation programmes
• Ability to identify subtle grading errors and ambiguous task conditions
• Experience handling urgent, short-duration project sprints

Why This Opportunity

• Apply JSON and rubric-evaluation skills to advanced AI research
• Contribute to a focused, high-priority evaluation sprint
• Work with clearly defined task instructions and grading criteria
• Build experience in structured model-output assessment
• Access potential extension opportunities based on project needs and performance

Contract Details

• Independent contractor role
• Fully remote within the United States
• Immediate availability is required
• Initial sprint expected to run through approximately 27–28 July, with possible extension
• Expected commitment of approximately 20–40 hours per week
• Tasks should generally be completed within 12–24 hours
• Competitive rates between $45–$170 per hour depending on experience, task complexity, and project scope
• Desktop or laptop computer required
• Chromebook devices are not supported
• Assessments must be completed without assistance from AI tools
• Weekly payments via Stripe or Wise
• Projects may be extended, shortened, or adjusted depending on operational requirements and performance
• Work will not involve access to confidential or proprietary information from any employer, client, or institution

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.