Help Center
Follow

Downtime and Connection Reports

This article will be used to provide updates on downtime and connection errors. Follow this article or follow @GC_Cares on Twitter for realtime updates when new reports are added to this page. 



Date/Time First Reported

02/23/17, 10:12 a.m. EST

Details & Resolution

Get Connected sites were taken down for unplanned maintenance from 10:08 to 10:15 a.m. Development is currently investigating the reason for the interruption in service. All sites were back up by 10:15 a.m.

Development diagnosed the cause as the triggered email that goes out to agency managers alerting them to pending hours. The query pulled too much data, which overloaded the servers. The offending query has been limited to the correct amount of data and is now running in under 1 second.


Date/Time First Reported

09/13/16, approximately 4:15 PM EDT

Details & Resolution

Get Connected sites were taken down for unplanned maintenance this afternoon. A query helps build the Users report in 2.6 Data Explorer took too long to resolve, and this backed up the queue of updates to users' data in Get Connected. The site was taken down for approximately 2 minutes while the query finished running. Get Connected was then restored to normal functionality.

Development is currently investigating the query and how to much it run more efficiently. The offending query has been disabled in the meantime.


Date/Time First Reported

5/23/16, approximately 9:00 AM EST

Details & Resolution

Get Connected sites were taken down for unplanned maintenance this morning. The problem began early in the morning when staff and clients began to report that various sites were slower than usual. Servers were offline for approximately 15 minutes near 10:10 AM EST while unscheduled maintenance was performed and queued requests were processed. Sites were restored at approximately 10:25.

The problem was determined to be related to two interacting issues. The first was a query in the system which was not running optimally. The second was that Get Connected systems were receiving several hundred requests from a small range of IP addresses. The query has been updated to run more smoothly and the range of IP addresses has been blocked. These two solutions make it extremely unlikely for this problem to reoccur.


Date/Time First Reported

4/29/16, approximately 10:08 AM EST

Details & Resolution

Get Connected sites were taken down for unplanned maintenance this morning. The problem began at 10:07 AM EDT and development was alerted and began investigating by 10:08 AM EDT. Development diagnosed the problem as being a a race condition the caused the logging database to run out of disk space. To fix the issue in the short term, the staff removed the incorrect records and made a change that would prevent this type of incorrect record from being produced and logged in the future. Additionally, the development team set an alert to make sure that they would be notified of disk storage issues in the future. Sites were restored at 10:25 AM EDT.

As a result of this problem, we are unable to capture information related to page views.


Date/Time First Reported

4/12/16, approximately 12:10 PM EDT

Details & Resolution

Get Connected sites were down for unplanned maintenance from 12:14 PM EDT to 12:17 PM EDT this afternoon. The outage was initiated after a query which calculates the number of hourly email campaigns took too long to run and disrupted performance from 12:08 PM EDT to 12:14 PM EDT. After the three-minute outage, systems were restored and functionality returned to normal. Developers are currently working to rewrite and restore the offending non-essential query, which has been taken down until it can be rewritten.


Date/Time First Reported

7/31/15, approximately 7:45am EDT

Details & Resolution

All Get Connected platforms were down for several hours. Most of the downtime occurred during the early morning hours of 7/31.

UPDATE (9:00am): Problem identified and steps initiated to bring servers back up.

UPDATE (10:15am): All sites are back up and functioning at optimal levels. Please send questions/concerns and/or reports of future incidents to support@galaxydigital.com.


Date/Time First Reported

7/7/15, approximately 11:30am EDT

Details & Resolution

A few Get Connected platforms reported slow connection speeds. The issue was identified by the development team and adjustments were made to increase connection speeds back to normal levels. As of 12pm on 7/7/15 speeds were improving and are expected to be back to normal levels within the hour. 

UPDATE (3:50pm): Loading speeds continued to remain slow on many platforms prompting a few more actions made by the development team. These recent actions seem to have corrected the issue as speeds are now back to normal.

UPDATE (4:15pm): A server involved in this issue needed to be reset. Sites are currently unresponsive. Connections should be restored once the server reset is complete.

UPDATE (4:50pm): Sites are now back up. Some users may still experience intermittent connection delays. 

UPDATE (5:00pm): All servers are back up. Sites should all be accessible and connection speeds should be back to normal levels. Please report any further incidents to support@galaxydigital.com.

UPDATE (6:08pm): Sites again loading more slowly than they should. 

UPDATE (6:11pm): Additional adjustments made to fix slow loading speeds.

UPDATE (6:26pm): Server once again required a reset. Sites expected to be down for a minute or two.

UPDATE (6:29pm): All sites back up.

UPDATE (7:10pm): Everything is continuing to run normally.

FINAL UPDATE (9:00pm): All servers are running at optimal levels. Issue resolved.


Date/Time First Reported

2/23/15, approximately 3:30pm EST

Details & Resolution

Several Get Connected platforms reported disruptions to service. Ultimately on 3/2/15 the Galaxy development team made an upgrade to the databases' processing power in order to manage all queries more efficiently and prevent any connections from becoming unresponsive.


 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk