Tech Lead Data Scientist Ai Evaluation And Monitoring
- Location
- Danville, United States
- Posted
- 207 days ago
How well does your resume fit this job?
Upload your resume to see your score for this Geisinger role, what matches, what's missing, and your best other matches.
Scored on our server and not saved there. The file stays in this browser so you don't have to upload it again after you sign up.
What You Will Own
- The technical evaluation methodology applied to AI programs across the enterprise, pre-production validation, production monitoring, and ongoing optimization
- Hands-on guidance to program teams as they design validation studies, equity audits, monitoring plans, and escalation playbooks for their AI systems
- Instrumentation of production monitoring: translating program-specific failure modes into concrete, measurable metrics
- The evaluation toolkit: LLM-as-Judge frameworks, golden sets, simulation harnesses, experimental study designs, drift detection, subgroup fairness analysis
- Reusable evaluation playbooks and templates that let each new program move faster than the last
- Technical direction, design review, and mentorship for a team of data analysts supporting the evaluation function
What You Will Not Own
- People management, HR administration, or formal performance evaluations for the analyst team (those sit with the analysts' line manager; the Tech Lead provides technical input)
- Program-level product strategy or go/no-go decisions
- Final clinical validation judgment on whether a given AI is safe for a given clinical use
- The software infrastructure behind the evaluation and monitoring tooling (built by the AI Platform team — the Tech Lead defines what's measured and how; Platform builds the backend)
This is a role that lives at three altitudes at once
With program teams (hands-on advisory). Partner with program owners early, before evaluations are designed, to shape study approach, sample size, stratification, gold-standard definition, and decision thresholds. Translate ambiguous failure modes into concrete, defensible evaluation designs. Coach teams through the technical work so that what arrives at governance review is rigorous, not performative.
With the evaluation toolkit (hands-on build). Design and operate the reusable assets that let evaluation scale: LLM-as-Judge rubrics and calibration methods, golden sets, simulation harnesses, A/B and shadow-mode study templates, subgroup fairness analyses, and drift monitors. Keep a pragmatic eye on what actually works in a clinical environment versus what works in a paper.
With the analyst team (technical leadership). Set technical direction, assign work across active evaluations, review analysis code and study designs, and raise the technical bar. Mentor analysts on methodology, statistical rigor, and the domain knowledge that makes evaluation credible. Grow them from execution into independent evaluation design.
Methods You'll Use
- Experimental and quasi-experimental design for production AI systems
- LLM and generative AI evaluation: golden sets, judge-based evaluation, hallucination and grounding checks
- Fairness and equity evaluation across patient and stakeholder subgroups
- Production monitoring design: drift detection, performance decay, adoption, and outcome metrics
- Causal inference methods appropriate to healthcare settings where full RCTs are often impractical
- Simulation and adversarial testing for pre-production stress testing
- Python, SQL, modern ML and evaluation tooling, cloud-native data platforms
Work is typically performed in an office or remote environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.
*Relevant experience may be a combination of related work experience and degree obtained (Master's Degree = 2 years; PHD = 4 years).
From Geisinger's careers page, posted on April 29, 2026, first listed on Rolev.ai on November 22, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Tech Lead Data Scientist Ai Evaluation And Monitoring role at Geisinger, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Similar roles
- Tech Lead - Digital Banking Platforms (Remote, AZ, NC, TX)First Citizens BancShares · Raleigh, NC
- Tech Lead ApplicationsBath & Body Works · United States
- Tech Lead Java DeveloperJobsbridge · San Jose, CA
- Tech LeadKRG Technologies · Weston, MA
- Tech Lead- Success factor Employee Central moduleAtria Group · Boston, MA
- Tech Lead, Salesforce 4CGenpact · 3409-G-LLC: 14th Flr, 521 Fifth Avenue, New York
Common questions
Where is the Tech Lead Data Scientist Ai Evaluation And Monitoring job at Geisinger located?
This Tech Lead Data Scientist Ai Evaluation And Monitoring position at Geisinger is located in Danville, United States. Always check Geisinger's application page for specific state eligibility or relocation policies.
How do I apply for the Tech Lead Data Scientist Ai Evaluation And Monitoring position at Geisinger?
This listing links directly to Geisinger's application page. You can apply directly on Geisinger's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.