Hi all and sorry again about this issue.
I want to share some details of the problem which happened yesterday.
What happened
We upgraded Ecwid to the new version: v4.1
This version has improvements in many areas, for example now Ecwid used Content Delivery Network widely to speed up store loading. These changes affected Ecwid API too. Well, they were correct, but caused a big issue, unfortunately. After the upgrade each API request to a product also started to generate three additional requests to аn image, thumbnail and small thumbnail. It works good if you have a small number of requests, but as soon as their number grows, the total number of all requests grows enormously too. Its number was so high, that our server was literally overloaded and was unable to process all requests.
As a result, some stores loaded slowly and others did not load at all.
What we did
As soon as out monitoring system notified us of this issue we started to investigate this situation. The server load was so high that it was even hard to log in to these servers using SSH!
Our investigation showed that API requests caused that problem, so we created a patch and applied a hot-fix to Ecwid.
What we will do to prevent this issue in future
This embarrassing problem shows that even correct code changes can be bad. In order to prevent this problem in future we will implement additional load tests in new releases to find out how new code works under high load.
We're really sorry for this situation and the problem it caused. Please, accept my sincere apologies for our fault.
I want to say thank you to all Ecwid users for their patience. Sorry again.
Update (15 Jun)
Some additional details about the Monday issue.
In order to fix
the issue with API, we had to release a hot-fix version with a solution. During the release we faced a problem with our CDN provider.
So what happened:
Ecwid now uses content delivery network for CSS, JS files and images. Each element is distributed to many servers over the world. It speeds up the loading, since you load all the resources from the closest one to you server. It's a great feature, but we found out that CDN cached some resources incorrectly. They were cached as blank files with zero size. Our CDN stores each resource in different versions, a unique file for each browser and geographic location. Some of the files were corrupted, so for example a store was not able to load in a specific browser or from a specific location.
What we did:
As soon as we found out this issue, we've started the investigation.
It showed that the problem is caused by CDN issue (thanks to everyone who reported this problem, you helped us greatly). Right after that our engineer implemented a solution to fix this issue, created and pushed it to our live servers.
I want to really apologize again for all these issues. We say that Ecwid has seamless upgrades. This upgrade wasn't seamless. It caused the downtime and issues with Ecwid stores.
It was a hard lesson for us and it showed the areas we need to improve. And we of course will.
Many thanks to all Ecwid users for their patience, bug reports and support. We appreciate it very much.
Thank you and sorry again.
Linear Mode
