Workday’s disaster-recovery site connected to its live production databases Friday, forcing the HR software company to take customer systems offline and leaving employees at affected employers unable to log in, clock in or request time off. As of 6:30 a.m. ET Saturday, Workday had not announced that service was restored or said when it would be.
Workday handles HR functions for thousands of employers. Its website says more than 11,500 companies use the platform for tasks such as clocking in and out, time off, training and benefits, and its client list includes Walmart, Target, Morgan Stanley, Charles Schwab and Netflix, LiveNOW from FOX reported. The company has not said how many customers lost access. A spokesperson described the impact only as “some of our customers.”
Workday traced the trouble to its disaster-recovery setup. “While validating the weekly Maintenance Window for Disaster Recovery tenants, we have identified that tenanted services within the DR Site unexpectedly connected to Production databases,” the company said in a notice on its customer status page, which sits behind a login and was reposted by Washington University’s IT team. That left data “in an inconsistent state,” Workday said, which can cause errors reaching tenants and make jobs, integrations and tasks fail. It placed all WD1 production tenants into maintenance while its teams worked out “the best recovery path.” Washington University’s IT team posted the notice at 8:05 p.m. ET Friday.
Florida Atlantic University’s IT status page warned that scheduled jobs, integrations and other automated processes tied to Workday may also fail. LiveNOW reported that the outage stopped workers from clocking in and out and requesting time off.
The San Francisco Chronicle reported that Workday found the problem while testing its backup systems, according to LiveNOW. Workday said in its notice that it would post updates every 30 minutes. “Some of our customers are experiencing a service disruption and we’re working urgently to investigate and resolve the issue,” a Workday spokesperson said in statements to LiveNOW and KRON4. “We sincerely apologize for the challenges this is causing impacted customers.”
Users started feeling it Friday afternoon. Downdetector logged nearly 3,000 outage reports shortly after 5 p.m. ET, according to LiveNOW. Emerson College told its community that Workday took the system offline at about 5:15 p.m. ET for urgent maintenance that the school had not planned or scheduled and that affected multiple Workday customers. At 9:53 p.m. ET, Saint Joseph’s University said Workday’s testing of a potential solution “continues to progress.” Shortly before midnight, Washington University said the application was not expected back until after its overnight maintenance window.
The outage came during open enrollment at some employers. Emerson said it will keep enrollment open until noon ET Monday, and later if Workday is still down then.
It is the latest in a run of major outages this week. Microsoft’s Outlook and Exchange outage ran for more than a day before Microsoft marked services restored late Friday, and the FAA’s nationwide ground stop hit air travel the same afternoon.
Washington University’s technical teams planned to reconvene at 7 a.m. ET Saturday, and the school said it will rerun linked automated jobs once Workday brings service back.
WebProNews is an iEntry Publication