Sr Principal Machine Learning Engineer (Advanced Threat Protection)
- Pay
- $190,400 – $308,000 per year
- Location
- Santa Clara, United States of America · On-site
- Job type
- Full-time
- Level
- Executive
- Experience
- 10+ years
- Education
- Bachelor's degree
- Listed
- Yesterday
Skills & highlights for this role
- Python
- Apache Spark
- GCP
- Go
- Transformers
Also useful
- EDR
- SIEM
- Deep packet inspection
How well does your resume fit this job?
Upload your resume to see your score for this Palo Alto Networks role, what matches, what's missing, and your best other matches.
Scored on our server and not saved there. The file stays in this browser so you don't have to upload it again after you sign up.
Job Summary
Your Career
As a Senior Principal Machine Learning Engineer, you will serve as the chief technical authority driving architectural strategy, design excellence, and AI innovation across cloud security systems. In this role, you will define the multi-year engineering roadmap, set the technical direction for foundation model adoption, and lead high-stakes architectural design across complex, distributed environments.
You will act as an organizational force multiplier—partnering across product, security research, infrastructure, and engineering teams to transform cutting-edge AI breakthroughs and high-throughput telemetry into resilient, production-grade Security-as-a-Service (SECaaS) offerings.
Your Impact
Strategic Leadership & Team Direction
- Set Technical Direction: Serve as the primary technical compass for the organization. Evaluate frontline AI research, filter transient hype from production-viable technology, and steer engineering investments toward high-leverage architectural paths.
- Cultivate Engineering Culture: Champion engineering rigor by instituting RFC standards, leading architectural review boards (ARBs), and mentoring Staff and Principal engineers to scale technical excellence across teams.
- Cross-Functional Partnership: Bridge the gap between Product Management, Security Researchers, Data Science, and Infrastructure teams. Translate ambiguous business objectives and complex threat landscapes into concrete technical requirements, milestone architectures, and executable designs.
Large-Scale System Design & Architecture
- End-to-End System Design: Lead the macro- and micro-level architecture of highly distributed, fault-tolerant cloud systems that process real-time security data streams at petabyte scale.
- Agentic Ecosystems: Architect robust, production-grade agentic frameworks featuring dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails.
- Data & Telemetry Backbone: Drive the architectural blueprint for ingestion, preprocessing, and feature engineering pipelines using Apache Spark, SQL, and BigQuery across Google Cloud platforms (Dataproc, GKE, and GCP SDKs).
Frontier Modeling, Diagnosis & Remediation
- Frontier Model Craftsmanship: Maintain deep, hands-on mastery of the evolving model landscape. Guide architectural choices across Large Language Models (LLMs), Small Language Models (SLMs), and multimodal systems to optimize for cost, latency, privacy, and domain-specific efficacy.
- Root-Cause Diagnosis & Quality Recovery: Establish systematic frameworks to diagnose underperforming models in production (e.g., drift, context dilution, alignment collapse, hallucination spikes). Pinpoint failures across training data distribution, loss convergence, or prompt/context limits, and lead targeted remediation via fine-tuning (SFT, DPO, RL, LoRA/PEFT).
- GPU & Accelerator Debugging: Troubleshoot low-level compute bottlenecks on distributed GPU clusters, resolving distributed training/inference issues, CUDA out-of-memory (OOM) states, NCCL communication overhead, and kernel performance stalls.
Your Experience
- Experience & Scale: 10+ years of progressive software engineering and systems design experience, with a documented history of technical leadership delivering high-availability, enterprise-scale platforms.
- Architectural Mastery: Proven ability to author complex system designs from scratch, establish data contracts, navigate intricate architectural trade-offs, and steer large-scale legacy-to-modern system migrations.
- Cross-Functional Influence: Demonstrated track record of aligning cross-functional teams (Product, Data, Applied Research, SecOps, Infrastructure) around unified technical visions and resolving competing design priorities.
- Frontier AI/ML Depth: Deep hands-on knowledge of modern AI architectures (Transformers, attention mechanisms, tokenization strategies) and practical operational experience deploying and serving both LLMs and SLMs.
- Model Triage & Fine-Tuning: Expert-level capability in analyzing why models degrade and executing targeted recovery pipelines (data curation, synthetic data generation, supervised fine-tuning, preference alignment, and parameter-efficient tuning).
- Hardware & Systems Debugging: Practical proficiency profiling and debugging distributed workloads on GPUs, diagnosing memory leaks, concurrency locks, and cluster communication bottlenecks.
- Languages & Data Engineering:
- Advanced proficiency in Python for AI/ML pipelines and scientific computing.
- Production-level fluency in Go or Java for low-latency, concurrent backend microservices.
- Strong foundation in data engineering with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE).
- Education: BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.
Preferred Qualifications (Strong Bonus)
- Cybersecurity Domain Expertise: Deep background in threat modeling, endpoint detection and response (EDR), SIEM architectures, or cloud security posture management.
- Network Topologies: Working knowledge of L3–L7 network protocols, packet telemetry, and deep packet inspection systems.
- Industry Thought Leadership: Contributions to open-source foundation model tooling, published research, or active participation in the broader AI systems community.
#LI-TD1
Pay and benefits
The compensation offered for this position will depend on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the starting base salary (for non-sales roles) or base salary + commission target (for sales/com-missioned roles) is expected to be the annual range listed below. The offered compensation may also include restricted stock units and a bonus. A description of our employee benefits may be found here.
$190,400.00 - $308,000.00/yr
Our Commitment
We’re trailblazers that dream big, take risks, and challenge cybersecurity’s status quo. It’s simple: we can’t accomplish our mission without diverse teams innovating, together.
All your information will be kept confidential according to EEO guidelines.
Is role eligible for Immigration Sponsorship?: Yes
Our Mission
At Palo Alto Networks®, we’re united by a shared mission—to protect our digital way of life. We thrive at the intersection of innovation and impact, solving real-world problems with cutting-edge technology and bold thinking. Here, everyone has a voice, and every idea counts. If you’re ready to do the most meaningful work of your career alongside people who are just as passionate as you are, you’re in the right place.
Who We Are
In order to be the cybersecurity partner of choice, we must trailblaze the path and shape the future of our industry. This is something our employees work at each day and is defined by our values: Disruption, Collaboration, Execution, Integrity, and Inclusion. We weave AI into the fabric of everything we do and use it to augment the impact every individual can have. If you are passionate about solving real-world problems and ideating beside the best and the brightest, we invite you to join us!
We believe collaboration thrives in person. That’s why most of our teams work from the office full time, with flexibility when it’s needed. This model supports real-time problem-solving, stronger relationships, and the kind of precision that drives great outcomes.
Equal opportunity and legal notices
We are committed to providing reasonable accommodations for all qualified individuals with a disability. If you require assistance or accommodation due to a disability or special need, please contact us at.
Palo Alto Networks is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.
From Palo Alto Networks's careers page, first listed on Rolev.ai on November 18, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Sr Principal Machine Learning Engineer (Advanced Threat Protection) role at Palo Alto Networks, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Similar roles
All Software Developer jobs- Software Development Engineer - Document IntelligenceWorkday · Pleasanton, CA
- Senior Software EngineerKcura · Illinois
- Expert Analyst Health Plan ApplicationsJefferson Health · Philadelphia, PA
- R&D Software EngineerAbb · Bartlesville, OK
- Senior DeveloperThermofisher · Remote, Pennsylvania
- Senior AI EngineerPillsbury Law · Nashville, TN
Common questions
What is the salary for the Sr Principal Machine Learning Engineer (Advanced Threat Protection) role at Palo Alto Networks?
The Sr Principal Machine Learning Engineer (Advanced Threat Protection) posting at Palo Alto Networks states compensation of $190,400 – $308,000 per year. It is listed as a full-time position. Final pay may depend on location, experience, and interview evaluation on Palo Alto Networks's official career site.
Where is the Sr Principal Machine Learning Engineer (Advanced Threat Protection) job at Palo Alto Networks located?
This Sr Principal Machine Learning Engineer (Advanced Threat Protection) position at Palo Alto Networks is located in Santa Clara, United States of America (On-site). Always check Palo Alto Networks's application page for specific state eligibility or relocation policies.
What qualifications are required for Sr Principal Machine Learning Engineer (Advanced Threat Protection) at Palo Alto Networks?
For the Sr Principal Machine Learning Engineer (Advanced Threat Protection) role, Palo Alto Networks highlights 10+ years of relevant experience; a bachelor's degree; experience with Python, Apache Spark, GCP, Go, Transformers. Rolev.ai can score your existing resume against these requirements and show any skill gaps before you apply.
How do I apply for the Sr Principal Machine Learning Engineer (Advanced Threat Protection) position at Palo Alto Networks?
This listing links directly to Palo Alto Networks's Workday application page. You can apply directly on Palo Alto Networks's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the Workday application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.