The old domain for this site was https://status.mastodonmusic.social and the new domain for this is https://mastodonmusic.stevetures.com. What's listed below the line break is an archive of what was discussed, just in case. Keeping it around is basically free so might as well. But it's all basically anachronistic as the server is gone so migration options are no longer possible.
Here's a handy list of music-oriented mastodon servers to browse or join:
or check out other servers to join at https://joinmastodon.org/servers or https://instances.social.
---
July 2024 - I've decided to shut down mastodonmusic.social on 20 Oct 2024 (date per Mastodon covenant guidance). This should give everyone time to go figure out what they want to do with their data or give everyone a chance to migrate elsewhere.
Options
Migrate to another server.
Pick a server. The Mastodon application supports moving your info and data to a new location, which is pretty neat. There's a raft of great well-run servers now, with nice communities. Take a look at https://joinmastodon.org/servers or https://instances.social/ if you want some ideas on where to move.
For musicians, https://musicians.today/ or https://musicworld.social/ or https://musician.social/ are all good alternatives.
Create your account and make sure it gets approved if needed. Then add your current account@mastodonmusic.social as an alias at that new server. I think you also have to make an alias of the destination one at your source account too? Mine was a little fiddly until I had aliases for each account on both sides (i.e. alias on destination for source, and then source for destination). Also do note that it's probably not a good practice to either reuse your password from one site to another, or to type your source password into your destination server, which shouldn't be necessary anyways.
Head to https://mastodonmusic.social/settings/migration to provide mastodonmusic.social with your new Mastodon account name in the format account@server.domain .
Download your data.
You can do that in the Import / Export section of preferences on the server. https://mastodonmusic.social/settings/export
Do nothing. All this migration stuff is optional, though do note that once the server is down, migration options will no longer be available.
Reasons?
Well, I'm happy to talk about them. None of them are catastrophic, but they aren't what I would consider to be reversible. In other words, with respect, I'm not really open to debating these, but just leaving them here for historical reasons.
Mastodon is pretty messy to administer. It's a delightful hodgepodge of F/OSS technologies, which, ah, really takes some getting used to. Package hell can happen if you're not careful or you don't know what you're doing with your underlying OS. If I were to do it again, I'd probably do the docker version rather than running directly in the host OS's root partition (err well you get the idea). Every update is sort of a nail-biter as you worry you're gonna blow up something or that your backups, while they exist, aren't very well-tested. I'd probably very consciously set up a dev and prod instance if were to do it again, though that means more money. Also cost, wow $200+ a month on GCP, which is nicely available and comes with G's nice legal team to make sure that frivilous search requests are turned down, but wow also pricey. I can afford it, but this morning I found myself asking "why though"...
Doing it right means finding at least one or more other persons to trust. For this instance, it's always just been me rather than put my users at risk of someone I don't fully know having access. But if something were to happen to me, you all would be stuck, floating on a sinking ship with no security patches, no backup, no restore, no one paying the bills, etc. It would go badly pretty quickly. There's some moderation tools for allowing trust, but as for actual sysadmin work, you really have it being all or nothing on finding someone you trust both technically and morally, as well as have the time and inclination to help. I personally have a high bar for all of this trust, and so I worried a lot about the bus factor on my instance. I had friends who probably meet the technical and moral bar, but are busy people who already voiced their concerns just with the generic idea of owning a mastodon server. All the technical, moderation, administration, moral and legal questions are pretty hard to ignore. I basically just did it anyways and ignored parts like the technical (timely patching), moderation (wow moderating trends is a PITA and I didn't want to come up with fairness standards for topics, so basically didn't even do the work) and legal (do I register with DMCA? what do I do if someone sends me a legal letter?).
Probably the happiest reason for this is part of why I started this instance, was that the Mastodon community was in bad shape in late 2022, and it was particularly bad for musicians. There wasn't instance that was functional that had the string "music" in the server name, which is what made me say "well dammit, guess I'll have to make one". Additionally, all of the popular instances were melting, and I felt like helping out. These days, things are a lot better, with shared moderation lists to ban terrible instances, better guard against spammers, regular patching, more people than just Eugen working on Mastodon (!!), great moderators, well-run large instances, integrations with other services like Threads (for better or worse, I do hope they figure out what to do around moderation to make sure that Threads doesn't spoil the party once Threads users show up on Mastodon post / comment pages) and many many more things that are legitimately great. So I'm glad things are a lot better than they were 1.5 years ago.
(Also, get rekt Elon. While Twitter/X isn't gone, it sure seems like a spammy heinous wasteland with plenty of terrible people making the service bad every day. Turns out being terrible to others is bad for business, who knew?)
So there it is. Take care, and should the need and the motivation ever arise, I'll be here to help pitch in and do what I can for musicians. Musicians have had such a shitty run for the last couple decades, even with so many people loving and depending on music in their lives, I hope the world gets back to paying musicians fair amounts, and that touring endlessly isn't the only way for musicians to survive.
Just for future, if you need to contact me, I'll still happily be on Mastodon: https://mastodon.online/@stevetures
DEPRECATED SEE ABOVE
This site has a list of maintenance events happening with Mastodon Music. I'll also do the occasional blog post related to the bigger plan for Mastodon Music. This is a page hosted entirely outside of Mastodon Music, with the hope that if things go unexpectedly completely sideways with the site, you can check here and hoepfully get a status update from me.
Table of Contents
26 Oct 2024 - final update - server self destructed
I was thinking of doing a retrospective here but honestly the shutdown process was a little fraght and I feel like I documented it well enough. Some folks have said some kind words, and I appreciate it. Be sure to also say kind words and support any future Mastodon server admins.
Take care, and find me at https://mastodon.online/@stevetures or anything else listed at https://stevetures.com.
23 Oct 2024 - self-destruct round three - progress!
Happily the new version of self-destruct seems to be a lot more reliable than the 4.2* or prior versions, as the task appears to be churning along. We're at 291 accounts to clean up, down from 441, which feels like slow but steady progress.
I hope this doesn't come off as gleeful, but I feel like it's important to the community and to my users to be sure cleanup and turndown is done correctly. Hopefully the next update will be the final one, maybe with a touch of retrospective. Also I hacked in a message to find status here, so that users aren't confused as to where their server went.
self_destruct: lead_html: Unfortunately, <strong>%{domain}</strong> is permanently closing down. If you had an account there, you will not be able to continue using it, but you can still request a backup of your data. <br><br>See <a href=http://status.mastodonmusic.social/>the status page</a> for more info. title: This server is closing down
22 Oct 2024 - self-destruct round two
After asking in the right discord for ideas, looks like tootctl self-destruct was very much revamped in 4.3, and no longer seemed to dump a ton of things into redis and then into sidekiq for long term processing. There were also some concerns on memory pressure for redis, so I bumped the machine I had from 2core 8GB of ram to 4core 16GB of RAM temporarily. I hope this works right. But I've gone through the pain to a 4.3 update, including fixing rb version, fixing yarm to use the new corepack method, and fixed nodejs, which apparently was never upgrading? It is now, everything's deployed, and I have a new fancy edit on the english template for this, though come to think of it, I may wish to chase down a few more popular languages for a hint to look at this status page.
20 Oct 2024 - mastodonmusic.social shutdown started!
Pour one out for mastodonmusic.social. I shutdown the services and nginx and now servers are correctly not connecting to it. Doing a last pgdump and a full disk backup, and then I might need to start up the mastodon services just to get a sidekiq instance going to process the kill-server request that will probably run for a while (should use screen to virtualize this, come to think of it).
Update - 1:37pm 20 Oct 2024
It's been interesting today. I've done a full pgdump of the db, and did a full disk snapshot in gcp, and then started in on the tootctl self-destruct. It noted that it was going to drop ~7mil tasks into the redis queue in a dry run, which honestly not even sure what to expect, so this seems fine?
So this sat there for a while as the machine started to spin. The main thing that seemed to happen is that after about 10 minutes, I got a connection refused traceback (and the webpage for sidekiq stopped loading) and then ... mystery happens.
The odd part is that only about 2.2mil sidekiq jobs appeared. I hope that's okay. Checking on the source, it looks like a server can understand it's in self destruct mode, so I'm going to wait for the queue to clear (might take a day or more) and then I'll try re-running self-destruct.
The migration work starts around 10am on these graphs. The final sidekiq starts is closer to 10:30am when the memory usage spikes up along with cpu usage and outbound network connections. That alarm bell is where I stoped nginx after the mastodon services and just wanted to see the alert fire once more just to ... dunno see it work once more? I didn't need it much as gcp was pretty reliable. Only paged once iirc.
30 Sept 2024 - Emailed all unmigrated users
Sent this to users who didn't have an account alias that they had migrated to. Hope I got the sql and logic right. 403 addresses.
It says:
"""FromStephen Tures <steve@mastodonmusic.social>to bcc: [403 email addresses][bcc all users who did not have a matching migration job pointing at another server]If you don't care about your Mastodon account on mastodonmusic.social and don't mind it being deleted when the server is shut off in 21 days, then you can stop reading now. Also, if I got the telemetry wrong and you already migrated your account, you can also stop reading now.---Hi there,If you're received this email, it means you have an account under this email at mastodonmusic.social and you might not be aware of it's imminent self-destruct.On July 4, I provided notice to users, as well as server wide alerts, that mastodonmusic.social will be shut down on 20 Oct 2024 which is in 21 days as of this writing.https://mastodonmusic.social/@steve/112730397940881625You have options:1) Do nothing2) Migrate your account to another server.Mastodon supports users moving from one server to another, and taking their followers with them, so I wanted to take a special opportunity to remind users of this feature. Once the shutdown happens on 20 Oct 2024, the data and followers and such will no longer be available to users.If you want more details, suggestions, rationale, etc please see the longer post I wrote at our static status page: https://status.mastodonmusic.social/Thanks and have a good day and may the most wonderful of music, other musicians, and musical fans befall you for many years to come,-Stevesteve@mastodonmusic.social
"""
30 Sept 2024 - Updated to 4.2.13 and 20 OCT SHUTDOWN STILL HAPPENING
Hi all, I patched from version v4.2.10 to v4.2.13 as this new version this morning suggested some medium security issues (DoS vuln?) to address. Responsible patching should always happen, regardless of any future plans such as the deprecation of this server in 21 days.
As always, https://status.mastodonmusic.social has general details and it's probably something I'll leave running even after the mastodon site has been self-destructed.
22 July 2024 - Updated to 4.2.10
Patched to v4.2.10 ... wow there were some spicy security bugfixes in there.
4 July 2024 - Updated to 4.2.10
Patched to v4.2.10 ... wow there were some spicy security bugfixes in there.
31 May 2024 - Updated to 4.2.9
Small update, we're running the new Mastodon minor release v.4.2.9. I'm not expecting any issues but please let me know if you see anything odd. More info here:
4 Mar 2024 - Updated to 4.2.8
Did the needful and updated to 4.2.8. While it's always good to do updates, most of the features in here are moderation default changes that I have basically already turned on, so I didn't feel a burning urgency to do the update. However, to make it clear that I care via staying up to date, here we are.
16 Feb 2024 - Updated to 4.2.6 no wait 4.2.7
It's been a busy week. I already patched to 4.2.6 on 14 Feb (which is a slow process as I have not yet devised an auto patch for my 2000 character post length) and was busy enough with work that I didn't post about it (sorry). But then this morning it's another security patch and damn, here we are again. So we're on 4.2.7.
https://github.com/mastodon/mastodon/releases/tag/v4.2.7
https://github.com/mastodon/mastodon/releases/tag/v4.2.6
1 Feb 2024 - Updated to 4.2.5
Patched to v4.2.5 to mitigate the unembargo'd CVE linked in the release notes. Enjoy and ty Mastodon devs for taking security seriously.
More info: https://github.com/mastodon/mastodon/releases/tag/v4.2.5
27 Jan 2024 - Updated to 4.2.4
Just finished a small maintenance release. Looking ahead to 4.3 branch as it looks fancy. Shiny UX coming soon!
29 Dec 2023 - Updated to 4.2.3
This is mostly just a maintenance release. 4.3 seems to be on the horizon, but isn't ready for consumption yet.
24 Aug 2023 - updated to 4.1.6 plus other basic mgmt
Jumped to 4.1.6 - pretty uneventful. Wrote a precheckout.sh and postcheckout.sh to keep me from doing anything dumb when copying around patches to set the max char count in posts.
30 Jul 2023 - updated to 4.1.5 plus other basic mgmt
Jumped to 4.1.5 and bounced the machine to pick up a new kernel and libc. I also posed an interesting question to the others on the server.
17 July 2023 - updated to 4.1.4 and refreshed ssl cert
Noticed late a small point release for Mastodon which I missed while I was on a work trip last week, and I had to do a ssl cert update to avoid expiry, so both of those are done. Probably about 30 sec for the 4.1.4 services restart and a minute to bounce the whole machine to restart nginx (which I supposed I could have done with service restart but was lazy and thought an occasional reboot would be good to have).
6 July 2023 - updated to 4.1.3
Just a small update to say that I've updated the site to 4.1.3 which is primarily security related. I've also implemented more regimented disk snapshotting, though it's not idealized (as in it doesn't pause or do a backup of the sql db prior to that, though that does happen separately).
17 May 2023 - site offline
12:24pm US Pacific - Got paged, investigating, don't know what broke. Stay tuned for more info.
12:28pm US Pacific - Looks like our VM was restarted by our cloud provider. Machine is back up and seems functional. Digging into why it dropped.
Conclusion: cloud vm provider or cluster of machines serving our VM suffered some sort of network event at 12:21:34PM and the VM itself bounced between 12:25:17PM and 12:26:05PM. Looks like the service was largely booted and recovered around 12:26:55PM. There aren't any comparable status updates, but I can't imagine why the network on our device failed for ~4 minutes before the VM itself without the provider likely bearing some fault.
This comes at a funny time, as I had on my list to deal with a spam issue from a federated server. More on that in a separate post.
4 May 2023 - temporary switch to requiring a reason to sign up for an account
Due to a recent sudden influx of spam accounts, I've proactively gone ahead and temporarily flipped https://mastodonmusic.social into requiring a reason and a request to sign up.
3 May 2023 - small post-checkout change
In order to support longer posts, I've manually bumped the max char limit to 2000 up from 500. I stole this method here, which I'll add to this status page for historical reasons. 500 is definitely short and given how well Mastodon does dor doesnt do with post threads (fixes hinted to be coming soon), this seems like a worthwhile move.
To patch usually after an update, as user mastodon
cd ~/live
cp -p app/javascript/mastodon/features/compose/components/compose_form.js app/javascript/mastodon/features/compose/components/compose_form.js.orig
cp -p app/validators/status_length_validator.rb app/validators/status_length_validator.rb.orig
cp -p app/serializers/rest/instance_serializer.rb app/serializers/rest/instance_serializer.rb.orig
sed -i 's/500/2000/g' app/javascript/mastodon/features/compose/components/compose_form.js
sed -i 's/500/2000/g' app/validators/status_length_validator.rb
sed -i 's/:registrations/:registrations, :max_toot_chars /g' app/serializers/rest/instance_serializer.rb
sed -i 's/private/def max_toot_chars\n 2000\n end\n\n private/g' app/serializers/rest/instance_serializer.rb
RAILS_ENV=production bundle exec rails assets:precompile && echo "pass" || echo "fail"
Then restart web daemon to take effect
service mastodon-web reload
To back out before another git checkout for update
cd ~/live
cp -p app/javascript/mastodon/features/compose/components/compose_form.js.orig app/javascript/mastodon/features/compose/components/compose_form.js
cp -p app/validators/status_length_validator.rb.orig app/validators/status_length_validator.rb
cp -p app/serializers/rest/instance_serializer.rb.orig app/serializers/rest/instance_serializer.rb
29 April 2023 - Blog Post
State of Mastodon Music
Been a while since I wrote down my thoughts on things. So here's some stats.
We're at 155 active users. If one's comparing it to the below stat, instances.social says we have 386 users, so we haven't technically lost a couple hundred users, and still what I would describe as growing at a normal rate. It remains a good mix of musicians, ~independant record labels, and music fans. We're held this trend for about 2-3 months at least without things slipping one way or the other.
Moderation. Thankfully this has been very light on duties. Since starting this, I've only had to handle 6 reports, where the root problem was invariably a blatant breaking of the server rules (spam, not labeling NSFW properly, etc) usually by an account on another instance, so suspending that user's behavior on mastodonmusic was an easy call. I also caught exactly one programmatically generated account using a very supicious mail server that permitted temp email addresses, and dealing with them was trivial. So far so good.
Technical Updates. Haven't had anything particularly earthshattering happen there. Each updates has been preceded with a dump of the db as well as archival snapshots of the disk juuuust in case. I don't feel particularly compelled to rock the boat yet with a technical redesign of the site as it remains responsive and reliable. I have alerts for paging me if the site is down for 5 minutes and I believe the only time it paged was the huge 3.5.3 -> 4.0 update that took a few hours to complete (see previous post). Things seems stable. A huge rush of users could change that.
So, again, how can you help?
The costs are quite stable month-to-month, so I don't feel compelled to request donations. What I'll continue to say is that, if you can afford it, please consider using any donation money you would have sent me instead on buying an artist's physical or bandcamp release or some swag or a concert ticket at a worthy small venue. Go do it right now, so you don't forget. 😉
I'm lucky to continue to be in a place where I can pretty comfortably afford the $140USD/mo that this site and the translation service costs. Rearchitecting the site might reduce the costs in some way (cheaper storage and db), but also raise a lot of concerns that might lead to site performance issues for actions like the typical media cache cleanup operation, or for basic searches for users. So again, for now, no change. If something happens to birdsite or my own income, or some popular music celeb appears on our site, then I'll let everyone know how we're going to end up dealing with things and commit to sinking a weekend into some migrations or standing up a way to enable donations.
17 Apr 2023 - Update to 4.1.2
Another small point release, mostly security fixes. See the general Mastodon release page for more info.
23 Mar 2023 - Update to 4.1.1
Small point release, mostly bug fixes and security tweaks.
12 Feb 2023 - Update to 4.1
Did an update from 4.0 -> 4.1 this day, though I'm backdating this entry as I forgot to do it at the time. I don't remember anything particularly notable happening then, but hooray new features. Announcement post: "Just wrapped up some extra-cautious backups and then an upgrade to the newly released Mastodon v4.1.
More info on that release here: https://github.com/mastodon/mastodon/releases/tag/v4.1.0 "
2 Jan 2023 - Usage and Maintenance Windows
Not much to report about the site. We continue to have a steady bit of usage. Monthly cost is final, which is $124, which seems pretty reasonable to me and something I don't mind paying for. If you feel the burning urge to support me, go buy some music of a great musician's bandcamp (or other profitable-for-the-musician server, just don't stream it and think that will help 😉).
I'm continuing the usual media cache cleanup chores. When federated posts arrive, they come with a copy of whatever media was attached to the post, we hold it for a minimum of 7 days before purging from the cache. This media can be fetched again when a user goes to go visit that federated version of the page here, or directly on the other mastodon server. This saves us ~60GB of storage each time. This doesn't affect the media you post locally to mastodonmusic.social. I still have an interest in migrating to something more scalable, though for now it's not a burning issue and I'm pretty pleased with the single host performance that we have right now.
One thing: I will likely be trying to find a time when I can cleanly power off the postgresql db and nginx and the mastodon services to take a clean snapshot. AIUI, it's likely that the post-maintenance snapshots taken are probably fine, but I'm extra paranoid, so that means shutting everything down to do this action. I'll also look into automating this. Look for announcements for when I figure out a good time to schedule and perform this. Also, another benefit to moving to cloudsql is not needing to do this ever. 🙂
9 Dec 2022 - Small change, translations enabled
I've enabled translations for this server. Go forth and enjoy reading texts from others who don't speak your language. :)
Quick note: it's not free and is provided by DeepL. I don't know what our usage will look like, but I've capped it at $30 a month, though hopefully it will be less. If it's more, I suspect the translate button will stop working.
22 Nov 2022 - Small change, disabling ActivityPub relay
At 8:52am, I've disabled the ActivityPub public relay I initially used to drive post traffic into the Federated section of mastodonmusic.social. This relay contained all largely unfiltered posts (except for posts from suspended / defederated domains due to rules violations, which I inherited at setup time from other instances trusting their moderation). More details here.
19 Nov 2022 - Blog Post
Well, it's been a wild 2 weeks of owning Mastodon Music. I'm glad I started it, and it's great to get exposed to a whole range of music and music lovers. Being a musician in the internet age has some mild upsides (easy to promote and share) but also some huge downsides that I think have really stressed musicians and the industry quite a bit. The somewhat-forced move to streaming and away from album purchases has lead to career musicians putting out albums sometimes as loss-leaders, and has often necessitated that musicians have to go on tour for all but the most widely streamed musicians.
Summary, in case you're in a hurry
I'm happy to have Mastodon Music stood up, and I'm thinking about the future with some technical and admin challenges that a project like this presents. In the meantime, I want this service to stick around and am grateful to be in a position to make that happen. But if you care a lot, have a read.
Background
With that guy taking over Twitter (don't need to mention his name, we all know who), things have gotten that much worse for musicians. When I thought about conciously leaving Twitter, I started to delete my music project's account, and turned around and went to find a music Mastodon server... and I was surprised that the options weren't readily obvious. I know now that there's a few good choices (musicians.today / raveclub.nation, maybe others that I'm forgetting), but the only one I could find then was an Italian instance that had closed registrations. As for myself, I'm lucky to have a pretty good day job in the tech industry, which personally creates a lot of tension, so I look for ways to help.
So I thought I was lucky enough to have some of the knowledge and comfort with tech and the means, and without thinking too long about it, I had acquired a cloud VM registered the name and set up mastodonmusic.social over the course of a few hours. Almost as soon as I published the existence of my site to https://instances.social directory, people were signing up, so I guess I wasn't alone in hoping for a musician or music-lover-friendly instance.
Where are we headed?
So far, I would say things are going pretty well. I'm not through a steady-state billing cycle yet, and probably wont be for another month, but it looks like Mastodon Music will cost about $100/mo to maintain for now, without having someone like Taylor Swift drop in and destroy the server with traffic (see below on exactly why this might be a problem at this moment). ;-) If I'm being blunt, I can manage that and none of this is a stressor for me. So if you're looking for a Patreon right now, I probably won't have one, at least for probably a few months. If you were thinking of helping me, instead go check out other local posts on the server and buy a record or two from someone you're loving. That said, I also see some challenges for this server that I want to get ahead of.
Scaling and Availability (technical and non-technical)
Technical bits. The default Mastodon deployment is intended to just run on a bespoke machine in a datacenter or house and maaaaybe serve 300-500 users (some handwaving), and this is running in a VM that's about that size. The good news is that if we suddenly start running slowly, I have a lot of room to bump up the speed and capacity of this machine, but it's not a trick I can keep playing, and it's an expensive trick long-term. But the good news is that my provider should never lose any of the data, and I take regular snapshots, so the odds of something wiping the data without recovery are really really low (nothing's impossible, but something that will almost certainly never happen).
The default webserver is nginx, which would require me to learn how to put that behind a VIP to scale that, though that's just an assuming as I don't know how that mitigates user session info between instances, but... I'm sure there's a way. The database (posts text, user info, pretty much everything that's not media) is a single-instance PostgreSQL deployment on the same machine. Not ideal for scaling. Moving this to a cloud DB is the scalable thing to do, but it's not done easily or cheaply. A local-disk dir handles all the media, both locally created and also posts from other servers that Mastodon Music users subscribe to. This grows... a lot more than than the database. Also there's some ways that Mastodon shards out tasks, like publish posts to other locations or handling incoming posts for people to read, etc that aren't quite low-hanging fruit, but it'd be interesting to see if I could run those in a more efficient way or even on other VM instances.
Side note: there's a basic prober test that runs on this site, so I should get SMS messages should the site ever go down for some reason. :)
Logo note: the person who did the neat logo is here. I can't claim to have amazing graphic design / illustration skills. More samples, including some unused ones, here.
Note: my day job is a little more IT and some light software engineering, but less so on the own-a-website sort of problem area. I know a bit about this space and some of the basic failures to avoid, but I won't claim expert level knowledge that someone deep into their SRE career might have. Tips on all this welcome. :)
Lastly, and probably worth saying out loud, I don't personally scale. I have x hours in the day to work on this, which is sometimes zero or close to it. I'm a parent and I have a family and some other responsibilities to the world. But I'll try to prioritize this for now. But really it's just up to me at the moment, so I'm going to go through and write a Disaster Recovery doc to share with a few work friends, who, to be clear, haven't actually volunteered to help. People in my industry unfortunately have full knowledge of what owning a site like this really means (interacting with bad social media actors, needing to moderate and also needing to deal with whatever responsibilities might be required to be performed to maintain a site like this, no clear revenue stream to ensure that this gets paid for beyond me funding it and keeping my job) though I could at least say something like "you're not on the hook, but should I suddenly become very unavailable (illness, etc), could you work with volunteers here to keep things moving?" and having that Disaster Recovery doc would really help smooth the amount of effort needed and thus make it more palatable for some to volunteer to help with.
Non-technical summary of the technical bits. There's a few different ways and projects I'll likely be taking on to get this server to a place where it could handle more user signup influx. It might even handle a few persons signed up that might drive a ton of traffic (read: what if someone famous very visibly signs up here. I think I could be ready to deal with it, but I'm not positive with where things are right at this second).
How Can You Help?
So that's probably more info than most everyone needs. I'm not yet ready to onboard volunteers purely because I'm not yet ready to trust anyone with this project yet. To be blunt, there's a lot of power that comes the moderator / admin and also access to the VM, and I need to figure out some safe ways to trust others with that eventually. But, if any of this resonates with you (tech comfort level, any SRE work done in the past, and an interest to help and maintain the moderation / rules structure that exists today), please DM me at @steve@mastodonmusic.social and we can at least start talking about this space. Again, I'm not comfortable yet granting any wide access to anyone that I don't know and trust extremely well, so for now I'll be running things. I think the last thing that musicians need is more drama or any loss of data or access on their social media. But I recognize that I'm a single point of failure, so I'll at least do what I can to ensure this service is running for as long as possible or needed.
Thanks,
@steve@mastodonmusic.social
- Large Update 3.53 -> 4.0
Maintenance Window - 20:00 16 Nov 2022 through to 04:00 17 Nov 2022 (all times US/Los Angeles / Pacific TZ).
Downtime / Sporadic Availability
Upgrade from 3.5.3 to 4.0. Full list of improvements here, but lemme just say being able to follow hashtags is A+ and great to follow something like #music. 😁
Requires some postgresql pre- and post-migration compilation, so not clear what performance penalty will be while this happens.
Change to email notifier. Gonna change things from the flat "admin@mastodonmusic.social" to "Mastodon Music Notifications <admin@mastodonmusic.social>"
Investigate a db-safe snapshotting of the primary disk prior to changes for clean rollback if something fully unexpected happens.
Will try to minimize this, but might lead to notable downtime, sadly I don't have any metrics on how long this might take before I do the upgrade
Will bounce the machine and server to pick up any related upstream updates.
Notes:
21:41 - couldnt figure out how to make the application-safe no-stop snapshot work, had to stop mastodon-sidekiq mastodon-streaming and mastodon-web and also postgresql to do a standard snaphot.
ha my prober alert just fired, so hey, that works. :)
21:47 - happily that didn't take long, snapshot is complete and services started again. Continuing with update.
22:01 - started installing ruby 3.0.4 as recommended and instance is thrashing enough that health checks and a secondary ssh connection isn't succeeding. Hopefully over soon.
22:16 - ruby 3.0.4 taking foreeeeever
23:06 - APPEARS TO WORK, WHEW. will add more in a moment. :D
23:10 - So part of this update is going from ruby 3.0.3 to 3.0.4 (the language mastodon is largely written in for all the moving parts). As I'm not the software engineer, who am I to question the need for this. So I pretty the new branch and it asked me to upgrade, and I said sure. This compiles ruby from source (the very slowest but most compatible / likely-to-be-fast later option) and it took waaaaay too long and eventually wedged the VM entirely and I got booted. Gasp! The VM came back up and I retried building ruby. I noticed that mastodon-web was dead and not starting, which probably lowered the load on the server and made it more likely to succeed? Any second time was a charm (a slow charm). I then did the pre install db migrate, then installed mastodon and let it compile the js / ts and then did the post install compile and restart and... damn. Here it is, working great. Hats off to Gregon for the smooth update.
23:15 - Gonna go breath a sigh of relief and check on a few things and hope nothing else is busted. If it is, sorry! But hopefully everything's fine.
23:35 - So the instance seemed to wedge after 20 minutes, noticed that ram use was pretty high, and while I should know if I've configured this to have swap, I don't. Wups.
23:56 - resized the machine and brought it back up again, then remembers to pgtune the db and things look better. Will monitor for a while.
Check out new memory usage. So upgrading from 4GB to 8GB is yes (see very last data point)