Job Title: Strategic Operations Manager, AI
Reports To: Head of Learning.ai (UGHE, CTLT)
Location: University of Global Health Equity (UGHE), Rwanda
Purpose of the Job
Reporting to the Head of Learning.ai (UGHE, CTLT), the Strategic Operations Manager owns the measurement framework and reporting for the Learning.ai portfolio, including the OneSpaceAfrica (OSA) platform and the eLearning program. The role (i) defines and tracks the metrics that show whether OSA is working, including the quality, reliability, safety, and fairness of its AI-enabled features, (ii) builds the outcomes and evaluation framework used to report results to partners, including COMS-A, and to donors, and (iii) works with the team to turn OSA performance data into findings for academic publication. The role also contributes to the design of needs assessments across the COMS-A network.
Key Responsibilities
- Own the Learning.ai measurement framework, defining the core metrics used to evaluate OSA and the broader portfolio, including engagement, efficacy, efficiency, and AI performance, and setting the targets the team is held to. This includes the outcomes used to assess AI quality, reliability, safety, bias and fairness, and human review effectiveness;
- Track progress against targets with the Head of Learning.ai and the Manager, Product & Engineering, holding the roadmap accountable to evidence without owning the roadmap itself;
- Define what AI and product outcomes need to be measured, and works with engineering so the platform captures the data evaluation requires. Product decisions, instrumentation, and the roadmap stay with the Product and Engineering team;
- Lead evaluation design for each funding phase, including primary and secondary outcomes, a baseline and comparator approach drawing on pilot usage and infrastructure data, implementation fidelity measures, adoption and usage thresholds, AI performance and safety measures, and the success thresholds that would justify further investment. Contribute faculty time and cost metrics, including cost per site and cost per student, to funding proposals;
- Contribute to needs-assessment design, working with the Partner Success Coordinator on interview and survey instruments so that findings can feed the evaluation framework;
- Hire and manage the Phase 0 needs-assessment consultant(s) and the Senior Advisor, Metrics, Analytics, and AI Evaluation, and hold them to their terms of reference. Day-to-day direction of needs-assessment fieldwork sits with the Partner Success Coordinator;
- Design the data strategy for performance management, determining what the OSA platform captures directly and what comes from surveys, interviews, and other faculty and student input, first at UGHE and then across the COMS-A network;
- Co-own the data management plan with the Infrastructure and DevOps Engineer, covering de-identification, data deposit, and the ethics approval required before platform data is used for research;
- Lead reporting to partners, donors, and internal leadership, and set the guardrails for how OSA results are reported externally;
- Prepare OSA performance findings for academic publication, working with the team on analysis and write-up;
- Work with external advisors, CTLT, COMS-A, and PIH's monitoring and evaluation team to ground Learning.ai's measurement approach in established practice.
Key Deliverables
- Approved Learning.ai measurement framework, with defined metrics, targets, and AI quality, reliability, safety, and fairness measures
- Evaluation design for each funding phase, including baseline, comparator, and success thresholds
- Data strategy and data management plan in place, with ethics approval secured before platform data is used for research
- Timely and accurate reporting to partners, donors, and internal leadership
- OSA performance findings prepared for academic publication
Qualifications, Knowledge, and Skills
a. Academic qualifications
Bachelor’s degree required in a quantitative or research-oriented field (e.g., Statistics, Economics, Public Health, Social Science, Data Science, or Business); master’s degree preferred
b. Experience
- Experience designing or managing program evaluation, M&E, or impact measurement; experience in education, global health, or international development preferred
- Experience measuring or evaluating digital or AI-enabled systems, including the ability to work with technical and domain experts to define meaningful measures of quality, reliability, safety, and fairness
- Experience with research design and ethics review, including baselines, comparators, and governance of personal data
- Project management experience, including scoping work and managing external consultants
c. Technical competencies
- Strong quantitative skills, with experience defining metrics, building reporting, and turning data into recommendations for decision-makers
- Comfortable working with product and engineering teams to define what data needs to be captured and how
- Excellent oral and written communication skills in English; English proficiency required
- French and/or Kinyarwanda proficiency preferred
d. Behavioral competencies
- Ability to work effectively in a team in a complex, fast-paced environment
- Exceptional interpersonal skills, including the ability to interact professionally with culturally diverse staff, partners, and clients
- Flexibility and agility
- Passion for social justice and global health
How to Apply
Applicants should provide: (1) curriculum vitae, (2) degree, (3) a cover letter describing your interest in UGHE and relevant experience. Please upload these under the ‘Additional Files’ tab on the application page.
University of Global Health Equity is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability status, protected veteran status, or any other characteristic protected by law.
Organizational Profile
UGHE is a new kind of university focused on training the next generation of global leaders in health care delivery. The university launched in Rwanda in September 2015. Through an academic experience uniquely rooted in the values of equity, students are empowered to both ease suffering at the bedside and drive transformational, systemic changes to the health system.
UGHE is an initiative of Partners In Health (PIH), an internationally recognized non-profit organization whose mission is to provide a preferential option for the poor in health care. In over nearly three decades of operating alongside public sectors in countries around the world, PIH has developed a model to deliver high quality healthcare to some of the world’s most marginalized communities. Nowhere has this impact been more profound than in Rwanda, where Inshuti Mu Buzima—PIH’s sister organization—and our government partners have driven innovation for a decade.
Members of the UGHE community are tenacious and resolute in our drive to attain social justice, make common cause with those in need, listen to and learn from others, and operate with honesty and humility as we uphold academic integrity and intellectual curiosity. The University of Global Health Equity seeks individuals committed to these values to join the team.
Commitment to Safeguarding and Prevention of Sexual Exploitation, Abuse, and Sexual Harassment
At UGHE, we are committed to ensuring that those who benefit from our work—including community members—as well as our students, contractors, staff, and visitors to our campus are treated with dignity and respect and protected from sexual exploitation, abuse and harassment and any form of systemic abuse, whilst reducing risk and vulnerabilities. A range of pre-employment checks will be undertaken in conformity with UGHE’s policy.
UGHE will request information from applicants’ previous employers about any findings of fraud, harassment, sexual harassment and any form of systemic abuse or incidents under investigation when the applicant left employment. By applying, the job applicant confirms their understanding of these recruitment procedures.
Disclaimer
We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.