HPC Systems Administrator (Hardware & Infrastructure Operations)
The Sherlock HPC cluster is the flagship of Stanford's research computing environment, supporting thousands of users and a massive variety of scientific workloads. We are looking for an HPC Systems Administrator who thrives at the intersection of high-density hardware and Linux systems engineering.
In this role, you will be the primary steward of the physical infrastructure on Sherlock and other platforms. You will ensure that our 1,500+ compute nodes, high-density GPU racks, and petabyte-scale storage arrays are meticulously maintained, expertly tuned, and highly available.
Why Stanford?
You won't just be swapping parts; you will be managing the physical backbone of a world-class research environment. From debugging errors on NVIDIA H200s to optimizing InfiniBand cabling for our Lustre scratch tiers, your work is the foundation upon which Nobel-caliber research is built.
Primary Responsibilities
- Hardware Lifecycle & Deployment: Lead the physical deployment, burn-in, troubleshooting, and decommissioning of compute nodes, GPU servers, and high-density storage systems.
- Diagnostics & Root Cause Analysis: Perform troubleshooting on hardware issues--such as memory errors, GPU thermal throttling, network failures -- and coordinate with vendors for support and replacements.
- Data Center Operations: Collaborate with the data centers team to plan and manage hardware deployments.
- Provisioning & Automation: Work with lead platform administrators on testing and provisioning to ensure rapid, consistent deployment of cluster images across the fleet.
- Health & Telemetry: Refine hardware-level monitoring to proactively identify failing components before they impact active research jobs.
Required Qualifications:
- Education: Bachelor's degree and eight years of relevant experience, or a combination of education and relevant experience.
- Experience: 3-5+ years of experience in Linux Systems Administration, with a strong preference for candidates from HPC, larges-scale data center, or research environments.
- Hardware Proficiency: Solid understanding of x86 server architecture, GPU systems, ethernet,and high-performance interconnects.
- Scripting: Proficiency in scripting languages for automating hardware health checks, log parsing, and routine maintenance tasks.
- Infrastructure Management: Experience using configuration management tools to manage hardware settings and firmware versions at scale. Experience working with data center teams to populate and maintain DCIM solutions preferred.
- Physical Requirements: Ability to lift up to 50 lbs and work comfortably in a data center environment, including racking equipment and managing complex cable topologies.
- Communication: Strong written and verbal communication skills.
Preferred Skills
- Direct experience maintaining hardware for HPC systems and large scale storage systems.
- Familiarity with the Slurm workload manager and how hardware health impacts job scheduling.
- Exposure to liquid cooling solutions or high-density rack power management.
Physical Requirements:
- Constantly perform desk-based computer tasks.
- Frequently sit, grasp lightly/fine manipulation.
- Occasionally stand/walk, writing by hand.
- Rarely use a telephone, lift/carry/push/pull objects that weigh up to 10 pounds.
Consistent with its obligations under the law, the University will provide reasonable accommodations to applicants and employees with disabilities. Applicants requiring a reasonable accommodation for any part of the application or hiring process should contact Stanford University Human Resources by submitting a contact form.
Working Conditions:
- May work extended hours, evenings, and weekends.
Work Standards:
- Interpersonal Skills: Demonstrates the ability to work well with Stanford colleagues and clients and with external organizations.
- Promote Culture of Safety: Demonstrates commitment to personal responsibility and value for safety; communicates safety concerns; uses and promotes safe behaviors based on training and lessons learned.
- Subject to and expected to stay in sync with all applicable University policies and procedures, including but not limited to the personnel policies and other policies found in Stanford's Administrative Guide, http://adminguide.stanford.edu/.
The expected pay range for this position is $150,289 to $171,674 per annum.
Stanford University provides pay ranges representing its good faith estimate of the salary or hourly wage the university reasonably expects to pay for a position. The pay offered to a selected candidate will be determined based on factors such as (but not limited to) the scope and responsibilities of the position, the qualifications of the selected candidate, departmental budget availability, internal equity, geographic location and external market pay for comparable jobs.
At Stanford University, base pay represents only one aspect of the comprehensive rewards package. The Cardinal at Work website (https://cardinalatwork.stanford.edu/benefits-rewards) provides detailed information on Stanford's extensive range of benefits and rewards offered to employees. Specifics about the rewards package for this position may be discussed during the hiring process.
The job duties listed are typical examples of work performed by position in this job classification and are not designed to contain or be interpreted as a comprehensive inventory of all duties, tasks, and responsibilities. Specific duties and responsibilities may vary depending on department or program needs without changing the general nature and scope of the job or level of responsibility. Employees may also perform other duties assigned.
Stanford is an equal employment opportunity and affirmative action employer. All qualifies applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, protected veteran status, or any other characteristic protected by law.
Additional Information
- Schedule: Full-time
- Job Code: 4833
- Employee Status: Regular
- Grade: K
- Requisition ID: 108777
- Work Arrangement : On Site
