Research Scientist - Post-Training

As a member of this team, you will use cutting-edge machine learning techniques to improve model performance in our domains of interest. The work is split into two sub-teams: Agents and Finetuning. Our Agents sub-team focuses on developing the LLM tools and scaffolding to create highly capable LLM-based agents, while our Finetuning Team builds out finetuning pipelines to improve models on our domains of interest. The Post-Training Team is seeking strong Research Scientists to join the team. The priorities of the team include both research-oriented tasks-such as designing new techniques for scaling inference-time computation or developing methodologies for in-depth analysis of agent behaviour-and engineering-oriented tasks-like implementing new tools for our LLM agents or creating pipelines for supporting and fine-tuning large open-source models. We recognise that some technical staff may prefer to span or alternate between engineering and research responsibilities, and this versatility is something we actively look for in our hires. You'll receive mentorship and coaching from your manager and the technical leads on your team, and regularly interact with world-class researchers and other exceptional staff, including alumni from Anthropic, DeepMind and OpenAI. In addition to junior roles, we offer Senior, Staff, and Principal Research Engineer positions for candidates with the requisite seniority and experience., The Internal Fraud function of the Fraud, Error, Debt and Grants Function at the Cabinet Office processes details of civil servants who have been dismissed for committing internal fraud, or who would have been dismissed had they not resigned. The Cabinet Office receives the details from participating government organisations of civil servants who have been dismissed, or who would have been dismissed had they not resigned, for internal fraud. In instances such as this, civil servants are then banned for 5 years from further employment in the civil service. The Cabinet Office then processes this data and discloses a limited dataset back to DLUHC as a participating government organisations. DLUHC then carry out the pre employment checks so as to detect instances where known fraudsters are attempting to reapply for roles in the civil service. In this way, the policy is ensured and the repetition of internal fraud is prevented. For more information please see - Internal Fraud Register. Security Successful candidates must undergo a criminal record check and get baseline personnel security standard (BPSS) clearance before they can be appointed. Additionally, there is a strong preference for eligibility for counter-terrorist check (CTC) clearance. Some roles may require higher levels of clearance, and we will state this by exception in the job advertisement. See our vetting charter here.

You may be a good fit if you have some of the following skills, experience and attitudes:

• Experience conducting empirical machine learning research (e.g. PhD in a technical field and/or papers at top ML conferences), particularly on LLMs.
• Experience with machine learning engineering, or extensive experience as a software engineer with a strong demonstration of relevant skills/knowledge in the machine learning.
• An ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem.
Particularly strong candidates also have the following experience:
• Building LLM agents in industry or open-source collectives, particularly in areas adjacent to the main interests of one of our workstreams e.g. in-IDE coding assistants, research assistants etc. (for our Agents subteam)
• Leading research on improving and measuring the capabilities of LLM agents (for our Agents sub-team)
• Building pipelines for fine-tuning (or pretraining LLMs). Finetuning with RL techniques is particularly relevant (for our finetuning subteam) .
• Finetuning or pretraining LLMs in a research context, particularly to achieve increased performance in specific domains (for our finetuning subteam)., We select based on skills and experience regarding the following areas:
• Research problem selection
• Research Engineering
• Writing code efficiently
• Python
• Frontier model architecture knowledge
• Frontier model training knowledge
• Model evaluations knowledge
• AI safety research knowledge
• Written communication
• Verbal communication
• Teamwork
• Interpersonal skills
• Tackle challenging problems
• Learn through coaching
Desired Experience We additionally may factor in experience with any of the areas that our work-streams specialise in:
• Cyber security
• Chemistry or Biology
• Safeguards
• Safety Cases
• Societal Impacts
  
  AI Safety Institute
London, UK The AI Safety Institute (AISI), launched at the 2023 Bletchley Park AI Safety Summit, is the world's first state-backed organization dedicated to advancing AI safety for the public interest. Our mission is to assess and mitigate risks from frontier AI systems, including cyber attacks on critical infrastructure, AI-enhanced chemical and biological threats, large-scale societal disruptions, and potential loss of control over increasingly powerful AI. In just one year, we've assembled one of the largest and most respected model evaluation teams, featuring renowned scientists and senior researchers from leading AI labs such as Anthropic, DeepMind, and OpenAI. At AISI, we're building the premier institution for impacting both technical AI safety and AI governance. We conduct cutting-edge research, develop novel evaluation tools, and provide crucial insights to governments, companies, and international partners. By joining us, you'll collaborate with the brightest minds in the field, directly shape global AI policies, and tackle complex challenges at the forefront of technology and ethics. Whether you're a researcher, engineer, or policy expert, at AISI, you're not just advancing your career - you're positioned to have significant impact in the age of artificial intelligence. About the Team The Post-Training Team is dedicated to optimising AI systems to achieve state-of-the-art performance across the various risk domains that AISI focuses on. This is accomplished through a combination of scaffolding, prompting, supervised and RL fine-tuning of the AI models which AISI has access to. One of the main focuses of our evaluation teams is estimating how new models might affect the capabilities of AI systems in specific domains. To improve confidence in our assessments, we make significant effort to enhance the model's performance in the domains of interest. For many of our evaluations, this means taking a model we have been given access to and embedding it as part of a wider AI system-for example, in our cybersecurity evaluations, we provide models with access to tools for interacting with the underlying operating system and repeatedly call models to act in such environment. In our evaluations which do not require agentic capabilities, we may use elicitation techniques like fine-tuning and prompt engineering to ensure assessing the model at its full capacity / our assessment does not miss capabilities that might be present in the model.

We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows