New Server!
Added 2020-07-07 16:25:27 +0000 UTCLast week we had some severe instability issues on the site, resulting in error messages on some pages and general slowness.
I'm not 100% sure what the cause was but I'm on the fence between just general growth of the sites reaching a tipping point, and another DDoS attack on Texture Haven. Both Texture Haven and HDRI Haven were running on the same server, so they were both affected.
We'd also been slowly running out of space on the server's HDD for a while so it was about time to upgrade anyway.
Our old server was a 4-core/4GB ram/1TB HDD "managed" dedicated server from Dreamhost, costing us $170 per month. I say "managed" because every time I ran into issues their support team would just tell me to solve them myself, defeating the purpose of paying a premium for help with management.
Our new servers (one for each site) are 8-core/8GB ram/1TB HDD dedicated servers from OneProvider costing only $15 per month each since they are unmanaged.
This over 10x cost reduction for a more powerful server might seem like a no-brainer, but I'm very inexperienced in the backend of managing web servers so it took me some time to learn by setting one up for 3D Model Haven before I was comfortable doing it for the big boys too.
They've been running for a few days (it took quite a while to get everything off the old unresponsive server) and things seem quite stable, though there's some unusual network activity on the TH server and some occasionally high disk bottle-necking on both.
In future there are some relatively easy potential upgrades we could do if things get bad again:
- Use Backblaze or some similar cloud service to host the huge asset files, removing much of the bandwidth and disk stress on the servers.
- Improve caching on the site to avoid unnecessary database queries.
- Use a separate high performance server or managed service for the database.
TLDR: New better cheaper servers which are more stable and give more budget to content creation.
Comments
Good idea yep. If we start to experience issues then I'll evaluate whether it'll be better to host multiple mirrors or use a cloud/cdn service.
Poly Haven
2020-07-08 11:01:47 +0000 UTCI've worked in the web hosting industry for over 20 years. Be very careful with cheap providers as they will drop you very quickly if they sense that they are losing money on you. While these companies like to throw around 1TB of bandwidth figures all day long being just $5/month, the reality is that providing 1TB of bandwidth costs them about $300/month if they are getting access to a large pipe like an OC-3 or higher. They are counting on the majority of their customers not even scratching that and most don't. They will often "shave off" the highest bandwidth utilizers by just canceling them to keep their costs down. I'm not sure about OneProvider, never heard of them. Usually the bigger the provider, the less chance they will notice, but of course its easy to automate the checking of such things. I was browsing the site and noticed that you're just using one name space for everything (hdrihaven.com), The great thing about DNS is that it's easy to create more records and to be honest very few non-IT people actually notice hostname changes. So maybe have your own CDN and records like chi.cdn.hdrihaven.net, lon.cdn.hdrihaven.net and sanfran.cdn.hdrihaven.net and have them point to cheap mirrors. Then if one goes down, you're only losing access to a portion of the files. Then just setup your front end so that it can quickly be repointed to another mirror if needed. Or have CNAME records in the frontend database so that the CNAME records can be changed so you don't have to update a bunch of database records. There are many ways to go about it.
Mark Krenz
2020-07-08 10:59:12 +0000 UTCYeah we'd be pretty screwed if CF decided not to cache our downloads, at the moment they handle about 75% of the combined 80TB per month. There are things we could do to reduce the bandwidth (e.g. DWAA exr isntead of HDR/PNG), and we've had some offers from universities and such to help mirror downloads, but it'd still be pretty devastating.
Poly Haven
2020-07-07 18:03:38 +0000 UTCOne thing I quickly want to point out because I praised the Cloudflare Bandwidth alliance for giving unlimited traffic is that Cloudflare doesn't technically allow Non-HTML Content (e.g. Large file downloads) in their ToS (https://www.cloudflare.com/terms/) Section 2.8, regardless of whether they come from Backblaze or your own server. I never really noticed that clause and I am kind of wondering whether they will ever take action against CC0 Textures or HDRi/Texture Haven. CC0 Textures is up to 30TB/Month by now ...
Lennart (ambientCG)
2020-07-07 17:46:47 +0000 UTC
