LLM Evaluation Assistant

Company: Blue Oak Consulting

Blue Oak Consulting is looking for an LLM Evaluation Assistant to help us test the logical integrity of AI outputs in high-stakes financial environments. This role is about ensuring that the models we use for commercial analysis produce results that are both accurate and grounded in real-world economics.
About Blue Oak Consulting
Blue Oak Consulting provides independent advisory on pricing, portfolio economics, capital allocation, and acquisition assumptions. We are not a traditional consulting firm focused on broad transformation programs or high level strategy decks. Instead, we interrogate specific commercial decisions where the financial stakes are high and the existing narrative is not sufficient. Our team looks past corporate language and P&L complexity to identify where value is leaking or capital is trapped. We work across sectors like software, manufacturing, and healthcare to test whether the assumptions behind a growth plan or an acquisition actually hold under pressure. We are a small, principal led firm where the people who do the work are the ones accountable for the results. We do not take transaction contingent fees, which allows us to be completely honest about whether the economics of a decision make sense. Our goal is to provide clarity through financial discipline and a healthy amount of skepticism. We stay through the execution phase because decisions that look sound in theory often fail when they meet real world friction.
The Role
In this part-time, fully remote role, you will help us verify the work of large language models that assist in our financial research and data synthesis. You will work directly with our senior leaders to evaluate whether AI-generated insights are accurate or if they are simply repeating corporate jargon. This is not about managing a project: it is about looking at data points and checking the math. The work requires a skeptical eye and the ability to work independently without a physical office environment. You will be expected to review large volumes of text and data to find contradictions and errors that a casual observer might miss.
What You Will Be Doing
* Reviewing AI-generated summaries of financial reports to check for accuracy against the source materials.
* Conducting side-by-side comparisons of model outputs against verified datasets and spreadsheets.
* Identifying instances where the model makes up information or fails to follow basic logical constraints.
* Testing specific prompts designed to extract pricing data and capital allocation details from messy documents.
* Documenting where models struggle with complex financial terminology or specific mathematical calculations.
* Working with senior consultants to improve the quality and structure of data inputs for various AI tools.
* Assisting in the creation of a logic-check library for common commercial scenarios we encounter.
* Flagging inconsistencies in the way different AI models interpret the same financial data points.
* Maintaining a log of common errors found in AI reasoning to help refine our internal standards.
What You Bring
* Ongoing study or a recent degree in finance, economics, mathematics, or computer science.
* A basic understanding of how large language models function and their typical logical limitations.
* The ability to read a balance sheet or income statement and understand how the components relate to each other.
* Exceptional attention to detail: you are the person who notices when a percentage does not add up.
* A skeptical mindset that does not take automated answers or corporate narratives at face value.
* Comfort communicating your findings clearly and concisely in a fully remote team environment.
* Willingness to learn the specifics of commercial advisory and how we analyze portfolio economics.
* Strong organizational skills to manage part-time hours alongside other study or work commitments.
What We Offer
* Fully remote work with no requirement for travel or office attendance.
* Flexible scheduling that fits alongside your university studies or other personal work.
* Attractive and competitive pay for an entry level assistant role.
* Practical experience seeing how AI is used in real commercial advisory scenarios.
* Direct access and mentorship from principals with decades of experience in the field.
* A work culture that values logic and evidence over corporate status and hierarchy.
* The opportunity to develop a deep understanding of corporate economics and financial discipline.
At Blue Oak, we believe that more data is only useful if the logic behind it is sound. This role is a chance to see how commercial decisions are actually scrutinized and to help us ensure that our technical tools meet our high standards for discipline. We value people who are more interested in getting the answer right than in following a script. In an environment where automation is becoming common, the ability to identify a logical flaw is one of the most important skills a junior professional can develop.

apply