5:47 PM EST
We're investigating errors with runs starting at 10:40PM UTC.
8:55 AM - 8:55 AM EST (0min duration)
Some runs stuck in 'pending' status
Due to errors with our kubernetes cluster, runs on several nodes were stuck in 'pending' status starting at 2:30am UTC and ending at approximately 1:40PM UTC
1:07 PM EDT
"failed taking a snapshot for this runner, please contact support"
We are currently investigating elevated error rates, apparently due to degraded filesystem performance
3:15 PM - 3:15 PM EDT (0min duration)
Degraded disk performance on nodes
We are investigating reduced disk performance and snapshotting errors on one of our production nodes.