On Monday, September 21, 2026, the Alpha cluster will undergo scheduled maintenance. Key changes:

• Institution-specific partitions (e.g. -p columbia, -p nyu, etc.) are being retired. All jobs will run through the shared alpha and grace partitions going forward — no action needed if you already use these.
• Job submissions will now require an explicit --account flag.
• Researchers without an active project will need to use the burst queue, lower-priority queue instead of being blocked from submitting jobs.

Brief interruptions to job scheduling may occur during the maintenance window. No action is required for most users; if you've been submitting jobs to an institution-specific partition by name, switch to alpha or grace going forward.

Rack 2 is complete. One node is failed out due to a hardware issue, for which we have an RMA from nvidia on the way.

Rack 3 drain / reservation is now starting.

Following our meeting yesterday, we have have raised the minimum GPU request for the beta phase from 1 to 4 GPUs. Researchers will now be required to request 4 GPUs to ensure alignment with the workload designed for the NVL72 system. We are currently discussing the creation of a dedicated partition for testing jobs that require only 1 GPU on the NVL72. This partition will feature a significantly lower wall time, allowing for quicker turnaround on tests. We will provide further updates as they become available.

Researchers may encounter the following error message if they attempt to request only 1 GPU:
srun/sbatch: error: QOSMinGRES
srun/sbatch: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

If you have any questions or concerns, please don't hesitate to reach out to us. Researchers can also submit a ticket for assistance.

The Beta control plane (login endpoint, Slurm controller) and Grace/Grace cluster remained up and stable throughout the weekend.

The Empire AI team is working closely with our BMS vendor for root-cause analysis and prevention.

The NVLink switch issues on Rack 1 have been fully resolved. As part of the remediation, the entire rack — switches and compute nodes — has been updated to the latest supported firmware and operating system versions.

All systems have been verified healthy (fabric connectivity, node health checks, and management connectivity) and Rack 1 is back in normal service, ready to resume serving traffic.