[neuron]

There were interruptions in Ecwid operation for about 12% of stores between 20:50 - 21:19 UTC. The problem has been fixed by our engineers. All services are now back to normal functional level. We are investigating the issue now, the more detailed report will be available later.

Please accept our deepest apologies for the issue.

UPDATE:
Unfortunately, we haven't managed to identify the root cause of the issue yet. Our engineers have developed tools that will allow us to get enough information the next time the issue appears. Besides that we have tuned on the monitoring system, so we'll identify the issue before it affects the service availability.

[Kess]

At the moment a little percent of Ecwid users may be facing an issue with the files and images uploading to Ecwid backend. Thanks everyone for the reports, all the info provided is very important.

If you are trying to upload something to your store and result with "Server is busy" error, please accept our sincere apologies for this trouble.

The issue no way affects the stores frontend functionality, so nothing prevents customers from browsing your Ecwid stores and making purchses in them.

Our engineers are working on the issue troubleshooting right now. As soon as the solution is found we will post the update.

UPDATE: The problem with upload still affects a number of customers who upload files in Ecwid Control Panel.
At the moment we are investigating the issue.

UPDATE: The investigation of the issue has shown that most probably the load balancer provided by Amazon breaks the uploads in some rare cases. We use balancer to prevent overload of any of our servers by ensuring they are all loaded evenly. The technical problem in the balancer may break the uploads. The problem is very rare and appears in a manner of waves. That is why we had a number of wrong assumptions before. At the moment we are contacting the Amazon support team in order to investigate the issue with their load balancer.

What have we implemented to protect the store owners? Yesterday, on June 13 at 11:30 UTC, we switched the network settings to exclude the load balancer - so, the uploads are hitting our servers directly. This was done with changing our DNS, and by this moment these changes should already be in action for all the Internet.

So, currently the upload should work without any issues, and our monitoring confirms that so far.

What actions are required from me? None. The change is fully transparent for all of our customers.

What should I do if we still have problems with file or image uploads? Please, report the technical details about your network:
1) open the terminal application (in Windows: click Start, Run..., type 'cmd' without quotes, run; *on Mac: launch Terminal), type "nslookup my.ecwid.com" without quotes, and hit Enter.
2) copy the resulting text from the terminal window after that command completes and register a new ticket in Ecwid Bugtracker, then paste there the result of the nslookup command.
This will greatly help us with the investigation.

We are currently keeping on the investigation of the issue with load balancer and will return the previous network configuration as soon as we are sure that it is safe.

Please accept our apologies for the inconvenience. We are doing our best to resolve the issue completely as soon as possible.

[Eugene Rimmer]

We are currently investigating the technical problem which prevents our stores and the control panel from opening. Please, accept our apologies for the trouble, our engineers are working to bring everything back online in minutes.

Update: currently, the problem is fixed and the access to all Ecwid stores is restored. The details of the incident will follow. We are very sorry for the trouble caused.

[neuron]

Today on June 18 from 08:19 UTC to 08:55 UTC (for 36 minutes) Ecwid storefronts and control panels were unavailable.

We would like to share some details on this outage. At 08:19 UTC one of the service engineers manually updated a service table in the Ecwid database in order to implement a planned maintenance. Along with other information this table contains information on the path to Ecwid's static resources in Content Delivery Network. Unfortunately, the engineer had not specified all necessary parameters in the update query and the whole service table got rewritten. After that Ecwid began to form incorrect URL's for the static resources and become broken.

The system updates the online information from this service table with a delay, so the issue did not appeared immediately and the engineer, who performed the maintenance, did not noticed the flaw right after the update. When we detected and located the issue we restored the actual service table from the latest backup snapshot. Technically speaking, Ecwid has been running as usual except it was unable to finish the load. That is why the automatic monitoring has failed to detect the issue. The whole Ecwid functionality was restored at 08:55 UTC. No data has been lost. We are deeply sorry about the inconvenience and the fail to detect the issue in time.

What are we going to implement in order to avoid such an issue in the future:

1. Any manual interventions may lead to a human error. So we have already created wrapper functions for the maintenance operations which need to be performed manually. These functions are going to perform updates safely and with necessary restrictions. This will eliminate human intervention into this subsystem.

2. We are also going to improve the monitoring service in order to detect a failure even if it appears during additional store resources loading.

Please accept our deepest apologies for the issue once again.

[Qetzal]

We are glad to announce that next Monday, June 25th, we are releasing Ecwid version 11.

While administrators are performing this upgrade, you may experience intermittent periods of down time. The upgrade will be taking place from 5:00 to 6:00 UTC (see this in your timezone).

[Kess]

Today (3:02 UTC) Ecwid storefronts and control panels got unavailable. Because of severe storms in North Virginia, USA our hosting provider (Amazon) had a power outage, which resulted in Ecwid servers down.

By the moment access to most stores has been restored. Amazon engineers are working on the issue resolution at the moment, so all the Ecwid accounts are expected to be up and running soon.

Please accept our apologies for the inconvenience caused by this situation. We'll keep you updated.

UPDATE: The system functionality has been restored. We're checking the data integrity after the recovery.

UPDATE: Most of Ecwid stores are up and available. However a small set of users may still face problems with their accounts access. We’re working on that issue troubleshooting and checking the data integrity after the system recovery. Please, check your stores.

[Qetzal]

Update on the outage:

1. The access to the vast majority of Ecwid stores was restored yesterday. Both storefronts and backend should work fine for almost all accounts.

However the outage affected the data integrity on some servers, so a small subset of Ecwid users has the following issues now:
- some orders or products are not accessible
- problems with log in to the control panel
- "Server is busy" errors while working in the control panel

At this moment our engineers are restoring the integrity of the data. If you have problems like described above, please report your Store ID to the bug-tracker or if you cannot log in to the control panel, send a message to login-issues@ecwid.com

2. We're very sorry for the lack of updates during the last day: our sleepless team is working hard for the last 36 hours to fix problems that were caused by Amazon (our hosting provider) outage. We promise - we will do our best and more to fix all the problems till the things are OK for _every_ our client.

[Eugene Rimmer]

Around 10% of Ecwid shops are not available at this very moment. Our engineers are dealing with the consequences of the Saturday's outage, and to fully restore the service for all customers they now have to reboot one of the database servers, which results in the inaccessibility for the stores using that one database. The ETA of restoration is about 15 minutes. We will keep you updated. Please, accept our apologies for this.

Update: the database server is up again, the affected shops are available now.

[Eugene Rimmer]

The Ecwid shops on Facebook may have problems with setting the size of the page (no scrollbars would appear) due to a Facebook bug. Here is the official message from Facebook:

Page Tab applications relying on `FB.Canvas.setSize` to resize are currently not resizing for users browsing the site on HTTPS.
We have a fix in progress and will update when it's pushed into production.

We expect Facebook engineers to fix this problem on their side soon.

[Eugene Rimmer]

Version 12.0 release planned on July 25th 2012: http://status.ecwid.com/post/2783213...