Around 11:55, we discovered issues related to google hosting responding with 503. After a short time we found many other services like http://nyt.com, amazon.com http://stackoverflow.com was down as well, and the issues seemed to be linked with services using the Fastly CDN.
The issue was resolved within an hour by fastly.
Issue was detected
Fastly found the root cause of issue, and services started working again
Everything seems to be back in order
The root cause was the Fastly CDN having outage for unknown reasons
The issue was resolved by fastly before we’re able to apply any fixes of our own.
A potential fix would have been to move our static hosting of the tool to another CDN network
None, as Fastly is one of the biggest CDN providers in the world, we have to trust that they keep their services running, as so many large companies rely on this
Around 09:47 our system detected the Vev editor was not loading. We discovered this issue was related to Google Firebase routing rules not working as expected, this caused the users not to be able to open the editor.
At the time of writing this report, the issue has still not been resolved by Google, but we have deployed a workaround to make the editor run.
Issue was detected
Root cause of issue was discoverd
Workarround to the firebase rules was deploy
The root cause of the incident was a Google Firebase router breaking. This was a widespread issue for all applications using Google Firebase Hosting, utilizing the hosting rewrite rules
First, we tried to replace scripts loaded from the __ folder with full URL to https://vev-prod.firebaseapp.com/__/ which was working at the time
This failed since these scripts disappeared as well
Changed all sciprst loaded from __ to be cdnjs script or loaded locally
Review all scripts not included in the editor bundle if it advantageous to include them
The following is a detailed overview of the incident leading to downtime of the Vev platform on the 18th of May 2021. We sincerely apologize for any inconvenience this caused our users. Since the crash was reported, the Vev team has been working to ensure this does not happen again and to improve our routines when it comes to updating our users of the status of the platform.
From 22:30 on May 17th, 2021 all services connected to the vev.design domain were inaccessible. The DNS registrar (Namecheap) blocked the domain due to malicious content published by Vev users to the free hosting service (eit.vev.design). The incident was first discovered on Monday 17th May 2021 at 22:30, the root cause of the issue was not resolved until 11:38 am on 18th May 2021.
This incident does not in any way compromise security of other Vev accounts or content published through the Vev platform by other clients. The consequence of the incident was 13 hours downtime of the vev.design domain including published sites with content on the Vev CDN such as images, fonts, videos, and scripts that did not load.
First reports of Vev crash came in
Root cause was discovered and DNS provider was contacted
Every 15 min after 8:28
Talked with new Namecheap support agent
DNS running as normal
The root cause of the incident was due to users in Vev abusing the free publishing (eit.vev.design) to create phishing sites. These types of sites break with the Acceptable Use Policy (AUP) of our domain registrar (Namecheap), and thus the domain got reported as abuse, and therefore everything connected to the domain was blocked. We have since discovered a set of users in the Vev platform using the platform to create malicious sites.
As soon as we discovered the issue we contacted Namecheap support.
1st line support did not have the rights to open the domain because of the abuse reports so had to wait for Namecheap Legal & Abuse Department to review the case.
Contacted 1st line support every 15 min after the first contact.
Tried helping clients in urgent need to publish project using the direct google storage URL, but required too much manual work.
At 11.00 we reached Namecheap Legal & Abuse Department and the blocker of the domain was removed.
Preventative measures completed so far prevent this type of incident from happening again, as all published sites breaking with the Acceptable Use Policy (AUP) of our domain registrar (Namecheap) are taken down. Additional measures will be done consecutively to further ensure security routines:
Suspend all suspicious users from the platform
Analyze all content on eit.vev.design and delete malicious sites
Transfer vev.design domain to our Cloudflare enterprise account Moving our domain registrar over to Cloudflare will make it easier for us to discover issues, and since we’re enterprise clients of Cloudflare would not block our domain on this short notice.
Setup status page for all services in Vev with automatic reporting/subscription features This will enable us to faster discover the root cause of the issue as well as allow users to see what the status of the vev platform is. Live updates will also be shared here in the future .
Transfer eit.vev.design to a.vev.site This will be done as a measurement to make sure user content does not affect the authority of the vev.design domain. So if malicious sites were to be published on the free domain, the abuse will not take down our main domain name. **
Stricter user control We will introduce a stronger measure to validate our users, to make sure they are not using the platform with bad intentions.
Password login will require verification
All emails will run through an anti-spam check and not be approved
Automatic Safe-browsing checks of the published content We will add an automatic analyzer (Web Risk | Google Cloud ) of all links when publishing projects blocking malicious projects from being published, as well as flagging users, abusing the platform.
Client-hosting of Images and Videos Our Engineering and Product team will look into solutions for our users to host images and videos added to Vev projects. **
In Vev Security and Privacy is something we strive to build into our products by design and by default. So, we rely on international and recognized standards to ensure we get the best of both worlds, from design and ease of use, flexibility and security.
This incident does not in any way compromise security of other Vev accounts or content published through the Vev platform by other clients.