HPC Systems Administrator (Hardware & Infrastructure Operations)

The Sherlock HPC cluster is the flagship of Stanford’s research computing environment, supporting thousands of users and a massive variety of scientific workloads. We are looking for an HPC Systems Administrator who thrives at the intersection of high-density hardware and Linux systems engineering.

In this role, you will be the primary steward of the physical infrastructure on Sherlock and other platforms. You will ensure that our 1,500+ compute nodes, high-density GPU racks, and petabyte-scale storage arrays are meticulously maintained, expertly tuned, and highly available.

Why Stanford?

You won't just be swapping parts; you will be managing the physical backbone of a world-class research environment. From debugging errors on NVIDIA H200s to optimizing InfiniBand cabling for our Lustre scratch tiers, your work is the foundation upon which Nobel-caliber research is built.

Primary Responsibilities

• Hardware Lifecycle & Deployment: Lead the physical deployment, burn-in, troubleshooting, and decommissioning of compute nodes, GPU servers, and high-density storage systems.
• Diagnostics & Root Cause Analysis: Perform troubleshooting on hardware issues—such as memory errors, GPU thermal throttling, network failures — and coordinate with vendors for support and replacements.
• Data Center Operations: Collaborate with the data centers team to plan and manage hardware deployments.
• Provisioning & Automation: Work with lead platform administrators on testing and provisioning to ensure rapid, consistent deployment of cluster images across the fleet.
• Health & Telemetry: Refine hardware-level monitoring to proactively identify failing components before they impact active research jobs.

Required Qualifications:

• Education: Bachelor’s degree and eight years of relevant experience, or a combination of education and relevant experience.
• Experience: 3-5+ years of experience in Linux Systems Administration, with a strong preference for candidates from HPC, larges-scale data center, or research environments.
• Hardware Proficiency: Solid understanding of x86 server architecture, GPU systems, ethernet,and high-performance interconnects.
• Scripting: Proficiency in scripting languages for automating hardware health checks, log parsing, and routine maintenance tasks.
• Infrastructure Management: Experience using configuration management tools to manage hardware settings and firmware versions at scale. Experience working with data center teams to populate and maintain DCIM solutions preferred.
• Physical Requirements: Ability to lift up to 50 lbs and work comfortably in a data center environment, including racking equipment and managing complex cable topologies.
• Communication: Strong written and verbal communication skills.

Preferred Skills

• Direct experience maintaining hardware for HPC systems and large scale storage systems.
• Familiarity with the Slurm workload manager and how hardware health impacts job scheduling.
• Exposure to liquid cooling solutions or high-density rack power management.

Physical Requirements*:

• Constantly perform desk-based computer tasks.
• Frequently sit, grasp lightly/fine manipulation.
• Occasionally stand/walk, writing by hand.
• Rarely use a telephone, lift/carry/push/pull objects that weigh up to 10 pounds.

Consistent with its obligations under the law, the University will provide reasonable accommodations to applicants and employees with disabilities. Applicants requiring a reasonable accommodation for any part of the application or hiring process should contact Stanford University Human Resources by submitting a contact form.

Working Conditions:

• May work extended hours, evenings, and weekends.

Work Standards:

• Interpersonal Skills: Demonstrates the ability to work well with Stanford colleagues and clients and with external organizations.
• Promote Culture of Safety: Demonstrates commitment to personal responsibility and value for safety; communicates safety concerns; uses and promotes safe behaviors based on training and lessons learned.
• Subject to and expected to stay in sync with all applicable University policies and procedures, including but not limited to the personnel policies and other policies found in Stanford's Administrative Guide, http://adminguide.stanford.edu/.