SME

Company: Innodata Inc.

Freelance Subject Matter Expert (SME) - Gold Set Prompt Writing
About the Role
We are looking for experienced Subject Matter Experts (SMEs) to support the development of high-quality Gold Set prompts, assessments, and evaluation content for Large Language Models (LLMs).
The role involves using your academic and professional subject expertise to create, review, and validate questions that assess an AI model’s ability to understand complex academic material, reason accurately, interpret evidence, and provide well-supported responses.
This is a freelance, project-based opportunity suitable for professionals, researchers, academics, PhD holders, postgraduate professionals, and experienced subject-matter specialists.
Key Responsibilities
* Develop high-quality prompts and assessment questions based on academic and professional subject matter.
* Create multiple-choice, analytical reasoning, short-answer, long-answer, and evidence-editing questions.
* Develop questions that test reading comprehension, critical thinking, analytical reasoning, evidence interpretation, and synthesis.
* Prepare accurate reference answers, expected answers, and evaluation criteria/rubrics for LLM assessment.
* Review AI-generated responses for factual accuracy, reasoning quality, relevance, and adherence to source material.
* Identify unsupported conclusions, logical errors, overgeneralizations, and misinterpretations of evidence.
* Ensure questions are challenging, unambiguous, and appropriate for the intended academic level.
* Create questions that require the model to distinguish between evidence, interpretation, and unsupported claims.
* Validate assessment content for subject accuracy and consistency.
* Follow detailed project guidelines and quality standards while working independently.
* Provide feedback and revisions based on quality-review requirements.
Required Expertise
Candidates should have strong academic or professional expertise in one or more of the following domains and sub-domains:
Engineering and Technology
* Civil Engineering
* Information Technology
* Computer Engineering
* Materials/Manufacturing Engineering
Physical Sciences
* Chemistry
* Physics
* Materials Science
Arts, Humanities, and Social Sciences
* Sociology
* Psychology
* Human Ecology
* Linguistics
* Library/Information Science
Medical Science
* Medicine
* Medical Technology
* Radiology
* Clinical Laboratory Medicine
Life Science
* Biology
* Botany
* Biochemistry
* Plant Biotechnology
Candidate Profile
* PhD professionals only.
* Strong academic reading and comprehension skills.
* Excellent written English and ability to communicate complex concepts clearly.
* Strong analytical and critical-thinking abilities.
* Ability to distinguish evidence-based conclusions from assumptions or overgeneralizations.
* Experience with academic research, technical writing, assessment development, content creation, or subject-matter review is preferred.
* Familiarity with AI/LLM evaluation, prompt writing, AI-generated content review, or data annotation is an advantage.
* Strong attention to detail and ability to follow detailed instructions and evaluation guidelines.
* Ability to work independently and meet project deadlines.
Example of Work
You may be asked to review an academic study and create questions that assess:
* Academic reading comprehension
* Evidence-based reasoning
* Interpretation of research findings
* Statistical and methodological limitations
* Critical analysis
* Distinguishing correlation, causation, and inference
* Identifying claims that go beyond the available evidence
* Short synthesis and evidence editing
Engagement
* Job Type: Freelance / Contract
* Work Arrangement: Remote
* Engagement: Project-based
* Schedule: Flexible, based on project requirements
* Compensation: Competitive hourly/project-based rate based on expertise and domain

apply