Over the past weekend, the eBridge Connections ePortal suffered from a serious disruption of service. The outage lasted approximately 30 hours from July 1st to July 2nd and it meant that some of our customers faced delays in transacting documents.
We are very sorry about what happened. Our customers expect and deserve a perfect uptime from their data integration platform. Many of our clients have questions about what caused this outage and we would like to be as transparent as possible with this information.
How did this happen?
This weekend, one of our EDI communications partners was performing maintenance on their servers. This maintenance took longer than expected. Once their servers were returned to operation, eBridge became overloaded with backlogged EDI transmissions from our communications partner. This meant that significant delays occurred in the transmission of documents through ePortal, as our servers processed the backlog. Some clients may not have received any documents at all while ePortal managed the overflow of documents. We have an alert system in place that would have notified our team about the delays and ensured that the backlog was buffered appropriately. This alert system failed, for reasons that we are still investigating.
Were any documents lost?
No. At this time, we do not believe that any documents were lost. The entire backlog of documents has now been processed, and the service disruption ended as of Tuesday morning EST.
What is eBridge Connections going to do to ensure that this does not happen again?
Within the next 10 days, our development team will be implementing an improved buffering mechanism with built-in redundancies. This system will be better at identifying and managing spikes in document traffic than our previous system. We also plan to add an independently-hosted system status page, which will provide our customers with information on the status of the network. Our core development team continues to research the outage and we will take any additional steps necessary to avoid future delays in document transmissions.
I have more questions about what happened.
Some elements of this outage are still under investigation. We know that many of our customers have questions about what happened, and we are here to answer them as best we can. Please feel free to give us a call at 1-800-755-6921 or send us an email at email@example.com. You can also contact me personally at firstname.lastname@example.org.
Again, we sincerely apologize to all of our clients for the delay in transactions over the weekend. We understand that our platform is critical for your business and we are grateful for your patience as we worked through this outage.