Classy Database Replication Error 9.8.17

Last Updated: Sep 13, 2017 08:38AM PDT

When did the error begin and when was it resolved?

The error started on Thursday, September 7th at 9:30 PM PST and was fully resolved Friday, September 8th at 2:14PM. Since Friday afternoon, we have been closely monitoring our error logs and have not see any similar interruptions to the system. We are confident that the bug is now resolved and have deployed improved monitoring for any future possible interruptions.

Did the error happen as a consequence of something that Classy did (release, code change, etc.)?

At the moment we are unclear how the error originated but believe it is related to a MySQL database error. We will continue to investigate the cause of this error and will provide more information as we receive it.

What part of the product was affected from this bug?

We began receiving errors from clients on Friday morning that the following areas of the product were affected by this issue:

  • Moving transactions between campaigns or fundraising pages and having that amount appear in the fundraising progress bar

  • Adding offline donations and having it save in the reporting

  • Updating fundraising page details

  • Changes in designer not saving and resulting in “resource not found” error

  • Inconsistent dashboard activity

  • Donation receipts showing incorrect amount (i.e. $0)

  • Donations not appearing on the campaign/fundraising page or in reports

  • Ability to register for a registration with fundraising event

What information will be recovered/captured vs what needs to be redone?

All transactions that were processed during this period should be captured in an organization’s transactions report and should be re-synced to Salesforce, if applicable. Please note that it is possible there were true failed transaction errors that happened during this time period that will not be processed.

Will notification emails, specifically for donations, re-send as a result of the reconciliation process?

We do have the ability to re-send receipts to donors that may have been affected during this interruption and will work to resend any receipts that may have included incomplete information or that did not initiate a send. 

Was the transfer of data to Salesforce affected by this issue?

Yes, Salesforce data transfer may be affected. From our initial investigations the majority of sync failures occurred from ClassyApp. This is primarily due to the method in which the data is collected prior to syncing. We do have a nightly job that compares Salesforce and Classy data occurring in the previous 2 days before the job runs, which will cover any missing data. For instance, when the job runs at 2 a.m. on Saturday morning, it will compare all data created from Thursday at midnight until Saturday at 2 .a.m.