MASTODONMUSIC.SOCIAL HAS BEEN SHUT DOWN.  THANKS FOR ALL THE FISH.

The  old domain for this site was https://status.mastodonmusic.social and the new domain for this is https://mastodonmusic.stevetures.com.  What's listed below the line break is an archive of what was discussed, just in case.  Keeping it around is basically free so might as well.  But it's all basically anachronistic as the server is gone so migration options are no longer possible.

Here's a handy list of music-oriented mastodon servers to browse or join:

or check out other servers to join at https://joinmastodon.org/servers or https://instances.social.


---

July 2024 - I've decided to shut down mastodonmusic.social on 20 Oct 2024 (date per Mastodon covenant guidance).   This should give everyone time to go figure out what they want to do with their data or give everyone a chance to migrate elsewhere.


Options


Reasons?

Well, I'm happy to talk about them.  None of them are catastrophic, but they aren't what I would consider to be reversible. In other words, with respect, I'm not really open to debating these, but just leaving them here for historical reasons.

So there it is.  Take care, and should the need and the motivation ever arise, I'll be here to help pitch in and do what I can for musicians.  Musicians have had such a shitty run for the last couple decades, even with so many people loving and depending on music in their lives, I hope the world gets back to paying musicians fair amounts, and that touring endlessly isn't the only way for musicians to survive.


Just for future, if you need to contact me, I'll still happily be on Mastodon: https://mastodon.online/@stevetures

DEPRECATED SEE ABOVE
This site has a list of maintenance events happening with Mastodon Music.  I'll also do the occasional blog post related to the bigger plan for Mastodon Music.  This is a page hosted entirely outside of Mastodon Music, with the hope that if things go unexpectedly completely sideways with the site, you can check here and hoepfully get a status update from me.


Table of Contents

26 Oct 2024 - final update - server self destructed

I was thinking of doing a retrospective here but honestly the shutdown process was a little fraght and I feel like I documented it well enough.  Some folks have said some kind words, and I appreciate it.  Be sure to also say kind words and support any future Mastodon server admins.

Take care, and find me at https://mastodon.online/@stevetures or anything else listed at https://stevetures.com

mastodon@mastodonmusic:~/live/bin$ RAILS_ENV=production ./tootctl self-destructI, [2024-10-24T15:27:40.819918 #3685162]  INFO -- : [dotenv] Loaded .env.productionSelf-destruct mode is already enabled for this Mastodon server219 accounts are still pending deletion.mastodon@mastodonmusic:~/live/bin$ RAILS_ENV=production ./tootctl self-destructI, [2024-10-24T19:46:57.182649 #4154845]  INFO -- : [dotenv] Loaded .env.productionSelf-destruct mode is already enabled for this Mastodon server169 accounts are still pending deletion.mastodon@mastodonmusic:~/live/bin$ RAILS_ENV=production ./tootctl self-destructI, [2024-10-25T15:24:40.935726 #1533817]  INFO -- : [dotenv] Loaded .env.productionSelf-destruct mode is already enabled for this Mastodon server69 accounts are still pending deletion.mastodon@mastodonmusic:~/live/bin$ RAILS_ENV=production ./tootctl self-destructI, [2024-10-25T19:05:23.346524 #2064115]  INFO -- : [dotenv] Loaded .env.productionSelf-destruct mode is already enabled for this Mastodon server19 accounts are still pending deletion.mastodon@mastodonmusic:~/live/bin$ RAILS_ENV=production ./tootctl self-destructI, [2024-10-26T14:18:41.049911 #3085830]  INFO -- : [dotenv] Loaded .env.productionSelf-destruct mode is already enabled for this Mastodon serverAt least one delivery attempt for each deletion notice has been made, but some have failed and are scheduled for retry

23 Oct 2024 - self-destruct round three - progress!

Happily the new version of self-destruct seems to be a lot more reliable than the 4.2* or prior versions, as the task appears to be churning along.  We're at 291 accounts to clean up, down from 441, which feels like slow but steady progress.

mastodon@mastodonmusic:~/live/bin$ RAILS_ENV=production ./tootctl self-destructI, [2024-10-23T23:10:07.350845 #2235754]  INFO -- : [dotenv] Loaded .env.productionSelf-destruct mode is already enabled for this Mastodon server291 accounts are still pending deletion.

I hope this doesn't come off as gleeful, but I feel like it's important to the community and to my users to be sure cleanup and turndown is done correctly.  Hopefully the next update will be the final one, maybe with a touch of retrospective.  Also I hacked in a message to find status here, so that users aren't confused as to where their server went.

$root/config/locales/en.yml

self_destruct:
    lead_html: Unfortunately, <strong>%{domain}</strong> is permanently closing down. If you had an account there, you will not be able to continue using it, but you can still request a backup of your data. <br><br>See <a href=http://status.mastodonmusic.social/>the status page</a> for more info.    title: This server is closing down

22 Oct 2024 - self-destruct round two

After asking in the right discord for ideas, looks like tootctl self-destruct was very much revamped in 4.3, and no longer seemed to dump a ton of things into redis and then into sidekiq for long term processing.  There were also some concerns on memory pressure for redis, so I bumped the machine I had from 2core 8GB of ram to 4core 16GB of RAM temporarily.  I hope this works right.  But I've gone through the pain to a 4.3 update, including fixing rb version, fixing yarm to use the new corepack method, and fixed nodejs, which apparently was never upgrading?  It is now, everything's deployed, and I have a new fancy edit on the english template for this, though come to think of it, I may wish to chase down a few more popular languages for a hint to look at this status page. 

mastodon@mastodonmusic:~/live/bin$ RAILS_ENV=production ./tootctl self-destructI, [2024-10-22T21:47:31.835833 #26213]  INFO -- : [dotenv] Loaded .env.productionSelf-destruct mode is already enabled for this Mastodon server441 accounts are still pending deletion.

20 Oct 2024 - mastodonmusic.social shutdown started!

Pour one out for mastodonmusic.social.  I shutdown the services and nginx and now servers are correctly not connecting to it.  Doing a last pgdump and a full disk backup, and then I might need to start up the mastodon services just to get a sidekiq instance going to process the kill-server request that will probably run for a while (should use screen to virtualize this, come to think of it).



Update - 1:37pm 20 Oct 2024

It's been interesting today.  I've done a full pgdump of the db, and did a full disk snapshot in gcp, and then started in on the tootctl self-destruct.  It noted that it was going to drop ~7mil tasks into the redis queue in a dry run, which honestly not even sure what to expect, so this seems fine?

So this sat there for a while as the machine started to spin.  The main thing that seemed to happen is that after about 10 minutes, I got a connection refused traceback (and the webpage for sidekiq stopped loading) and then ... mystery happens.

The odd part is that only about 2.2mil sidekiq jobs appeared.  I hope that's okay.  Checking on the source, it looks like a server can understand it's in self destruct mode, so I'm going to wait for the queue to clear (might take a day or more) and then I'll try re-running self-destruct.


The migration work starts around 10am on these graphs.  The final sidekiq starts is closer to 10:30am when the memory usage spikes up along with cpu usage and outbound network connections.  That alarm bell is where I stoped nginx after the mastodon services and just wanted to see the alert fire once more just to ... dunno see it work once more?  I didn't need it much as gcp was pretty reliable.  Only paged once iirc.

30 Sept 2024 - Emailed all unmigrated users

Sent this to users who didn't have an account alias that they had migrated to.  Hope I got the sql and logic right. 403 addresses.


It says:

"""From
Stephen Tures <steve@mastodonmusic.social>
to bcc: [403 email addresses][bcc all users who did not have a matching migration job pointing at another server]If you don't care about your Mastodon account on mastodonmusic.social and don't mind it being deleted when the server is shut off in 21 days, then you can stop reading now.  Also, if I got the telemetry wrong and you already migrated your account, you can also stop reading now.---Hi there,If you're received this email, it means you have an account under this email at mastodonmusic.social and you might not be aware of it's imminent self-destruct.On July 4, I provided notice to users, as well as server wide alerts, that mastodonmusic.social will be shut down on 20 Oct 2024 which is in 21 days as of this writing.https://mastodonmusic.social/@steve/112730397940881625You have options:1) Do nothing2) Migrate your account to another server.Mastodon supports users moving from one server to another, and taking their followers with them, so I wanted to take a special opportunity to remind users of this feature.  Once the shutdown happens on 20 Oct 2024, the data and followers and such will no longer be available to users.If you want more details, suggestions, rationale, etc  please see the longer post I wrote at our static status page: https://status.mastodonmusic.social/Thanks and have a good day and may the most wonderful of music, other musicians, and musical fans befall you for many years to come,-Stevesteve@mastodonmusic.social
"""

30 Sept 2024 - Updated to 4.2.13 and 20 OCT  SHUTDOWN STILL HAPPENING

Hi all, I patched from version v4.2.10 to v4.2.13 as this new version this morning suggested some medium security issues (DoS vuln?) to address.  Responsible patching should always happen, regardless of any future plans such as the deprecation of this server in 21 days.

As always, https://status.mastodonmusic.social has general details and it's probably something I'll leave running even after the mastodon site has been self-destructed.

https://github.com/mastodon/mastodon/releases/tag/v4.2.13

22 July 2024 - Updated to 4.2.10

Patched to v4.2.10 ... wow there were some spicy security bugfixes in there. 

https://github.com/mastodon/mastodon/releases/tag/v4.2.10

4 July 2024 - Updated to 4.2.10

Patched to v4.2.10 ... wow there were some spicy security bugfixes in there. 

https://github.com/mastodon/mastodon/releases/tag/v4.2.10

31 May 2024 - Updated to 4.2.9

Small update, we're running the new Mastodon minor release v.4.2.9.  I'm not expecting any issues but please let me know if you see anything odd.  More info here:

https://github.com/mastodon/mastodon/releases/tag/v4.2.9

4 Mar 2024 - Updated to 4.2.8

Did the needful and updated to 4.2.8.  While it's always good to do updates, most of the features in here are moderation default changes that I have basically already turned on, so I didn't feel a burning urgency to do the update.  However, to make it clear that I care via staying up to date, here we are.

https://github.com/mastodon/mastodon/releases/tag/v4.2.8

16 Feb 2024 - Updated to 4.2.6 no wait 4.2.7

It's been a busy week.  I already patched to 4.2.6 on 14 Feb (which is a slow process as I have not yet devised an auto patch for my 2000 character post length) and was busy enough with work that I didn't post about it (sorry).  But then this morning it's another security patch and damn, here we are again.  So we're on 4.2.7.

https://github.com/mastodon/mastodon/releases/tag/v4.2.7
https://github.com/mastodon/mastodon/releases/tag/v4.2.6

1 Feb 2024 - Updated to 4.2.5

Patched to v4.2.5 to mitigate the unembargo'd CVE linked in the release notes. Enjoy and ty Mastodon devs for taking security seriously.
More info: https://github.com/mastodon/mastodon/releases/tag/v4.2.5

27 Jan 2024 - Updated to 4.2.4

Just finished a small maintenance release.  Looking ahead to 4.3 branch as it looks fancy.  Shiny UX coming soon!

29 Dec 2023 - Updated to 4.2.3

This is mostly just a maintenance release.  4.3 seems to be on the horizon, but isn't ready for consumption yet.

29 Sept 2023 - updated to 4.2.0 !!

This is a pretty big deal with a lot to share.  There's a nice blog post from Eugen and friends  that is pretty easy to read here.  If you want more technical details about what changed, that's over here.  Enjoy!

24 Aug 2023 - updated to 4.1.6 plus other basic mgmt

Jumped to 4.1.6 - pretty uneventful.  Wrote a precheckout.sh and postcheckout.sh to keep me from doing anything dumb when copying around patches to set the max char count in posts.

30 Jul 2023 - updated to 4.1.5 plus other basic mgmt

Jumped to 4.1.5 and bounced the machine to pick up a new kernel and libc. I also posed an interesting question to the others on the server.

17 July 2023 - updated to 4.1.4 and refreshed ssl cert

Noticed late a small point release for Mastodon which I missed while I was on a work trip last week, and I had to do a ssl cert update to avoid expiry, so both of those are done.  Probably about 30 sec for the 4.1.4 services restart and a minute to bounce the whole machine to restart nginx (which I supposed I could have done with service restart but was lazy and thought an occasional reboot would be good to have).

6 July 2023 - updated to 4.1.3

Just a small update to say that I've updated the site to 4.1.3 which is primarily security related.  I've also implemented more regimented disk snapshotting, though it's not idealized (as in it doesn't pause or do a backup of the sql db prior to that, though that does happen separately).

17 May 2023 - site offline

Conclusion: cloud vm provider or cluster of machines serving our VM suffered some sort of network event at 12:21:34PM and the VM itself bounced between 12:25:17PM and 12:26:05PM.  Looks like the service was largely booted and recovered around 12:26:55PM.  There aren't any comparable status updates, but I can't imagine why the network on our device failed for ~4 minutes before the VM itself without the provider likely bearing some fault.

This comes at a funny time, as I had on my list to deal with a spam issue from a federated server.  More on that in a separate post.

4 May 2023 - temporary switch to requiring a reason to sign up for an account

Due to a recent sudden influx of spam accounts, I've proactively gone ahead and temporarily flipped https://mastodonmusic.social into requiring a reason and a request to sign up.


3 May 2023 - small post-checkout change

In order to support longer posts, I've manually bumped the max char limit to 2000 up from 500.  I stole this method here, which I'll add to this status page for historical reasons.  500 is definitely short and given how well Mastodon does dor doesnt do with post threads (fixes hinted to be coming soon), this seems like a worthwhile move.

To patch usually after an update, as user mastodon

cd ~/live
cp -p app/javascript/mastodon/features/compose/components/compose_form.js app/javascript/mastodon/features/compose/components/compose_form.js.orig
cp -p app/validators/status_length_validator.rb app/validators/status_length_validator.rb.orig
cp -p app/serializers/rest/instance_serializer.rb app/serializers/rest/instance_serializer.rb.orig
sed -i 's/500/2000/g' app/javascript/mastodon/features/compose/components/compose_form.js
sed -i 's/500/2000/g' app/validators/status_length_validator.rb
sed -i 's/:registrations/:registrations, :max_toot_chars /g' app/serializers/rest/instance_serializer.rb
sed -i 's/private/def max_toot_chars\n 2000\n end\n\n private/g' app/serializers/rest/instance_serializer.rb

RAILS_ENV=production bundle exec rails assets:precompile && echo "pass" || echo "fail"

Then restart web daemon to take effect

service mastodon-web reload

To back out before another git checkout for update

cd ~/live
cp -p app/javascript/mastodon/features/compose/components/compose_form.js.orig app/javascript/mastodon/features/compose/components/compose_form.js
cp -p app/validators/status_length_validator.rb.orig app/validators/status_length_validator.rb
cp -p app/serializers/rest/instance_serializer.rb.orig app/serializers/rest/instance_serializer.rb

29 April 2023 - Blog Post

State of Mastodon Music

Been a while since I wrote down my thoughts on things. So here's some stats. 

So, again, how can you help?

The costs are quite stable month-to-month, so I don't feel compelled to request donations.  What I'll continue to say is that, if you can afford it, please consider using any donation money you would have sent me instead on buying an artist's physical or bandcamp release or some swag or a concert ticket at a worthy small venue. Go do it right now, so you don't forget. 😉

I'm lucky to continue to be in a place where I can pretty comfortably afford the $140USD/mo that this site and the translation service costs.  Rearchitecting the site might reduce the costs in some way (cheaper storage and db), but also raise a lot of concerns that might lead to site performance issues for actions like the typical media cache cleanup operation, or for basic searches for users.  So again, for now, no change.  If something happens to birdsite or my own income, or some popular music celeb appears on our site, then I'll let everyone know how we're going to end up dealing with things and commit to sinking a weekend into some migrations or standing up a way to enable donations.

17 Apr 2023 - Update to 4.1.2

Another small point release, mostly security fixes.  See the general Mastodon release page for more info.

23 Mar 2023 - Update to 4.1.1

Small point release, mostly bug fixes and security tweaks.

12 Feb 2023 - Update to 4.1

Did an update from 4.0 -> 4.1 this day, though I'm backdating this entry as I forgot to do it at the time.   I don't remember anything particularly notable happening then, but hooray new features.  Announcement post: "Just wrapped up some extra-cautious backups and then an upgrade to the newly released Mastodon v4.1.

More info on that release here: https://github.com/mastodon/mastodon/releases/tag/v4.1.0 "

2 Jan 2023 - Usage and Maintenance Windows

Not much to report about the site.  We continue to have a steady bit of usage.  Monthly cost is final, which is $124, which seems pretty reasonable to me and something I don't mind paying for.  If you feel the burning urge to support me, go buy some music of a great musician's bandcamp (or other profitable-for-the-musician server, just don't stream it and think that will help 😉).

I'm continuing the usual media cache cleanup chores. When federated posts arrive, they come with a copy of whatever media was attached to the post, we hold it for a minimum of 7 days before purging from the cache.  This media can be fetched again when a user goes to go visit that federated version of the page here, or directly on the other mastodon server.  This saves us ~60GB of storage each time.  This doesn't affect the media you post locally to mastodonmusic.social.  I still have an interest in migrating to something more scalable, though for now it's not a burning issue and I'm pretty pleased with the single host performance that we have right now.

One thing: I will likely be trying to find a time when I can cleanly power off the postgresql db and nginx and the mastodon services to take a clean snapshot.  AIUI, it's likely that the post-maintenance snapshots taken are probably fine, but I'm extra paranoid, so that means shutting everything down to do this action.  I'll also look into automating this.  Look for announcements for when I figure out a good time to schedule and perform this.  Also, another benefit to moving to cloudsql is not needing to do this ever. 🙂


9 Dec 2022 - Small change, translations enabled

I've enabled translations for this server.  Go forth and enjoy reading texts from others who don't speak your language.  :)

Quick note: it's not free and is provided by DeepL.  I don't know what our usage will look like, but I've capped it at $30 a month, though hopefully it will be less.  If it's more, I suspect the translate button will stop working.

22 Nov 2022 - Small change, disabling ActivityPub relay

At 8:52am, I've disabled the ActivityPub public relay I initially used to drive post traffic into the Federated section of mastodonmusic.social.  This relay contained all largely unfiltered posts (except for posts from suspended / defederated domains due to rules violations, which I inherited at setup time from other instances trusting their moderation).  More details here.

19 Nov 2022 - Blog Post

Well, it's been a wild 2 weeks of owning Mastodon Music.  I'm glad I started it, and it's great to get exposed to a whole range of music and music lovers.  Being a musician in the internet age has some mild upsides (easy to promote and share) but also some huge downsides that I think have really stressed musicians and the industry quite a bit.  The somewhat-forced move to streaming and away from album purchases has lead to career musicians putting out albums sometimes as loss-leaders, and has often necessitated that musicians have to go on tour for all but the most widely streamed musicians.

Summary, in case you're in a hurry

I'm happy to have Mastodon Music stood up, and I'm thinking about the future with some technical and admin challenges that a project like this presents.  In the meantime, I want this service to stick around and am grateful to be in a position to make that happen.  But if you care a lot, have a read.


Background

With that guy taking over Twitter (don't need to mention his name, we all know who), things have gotten that much worse for musicians.  When I thought about conciously leaving Twitter, I started to delete my music project's account, and turned around and went to find a music Mastodon server... and I was surprised that the options weren't readily obvious.  I know now that there's a few good choices (musicians.today / raveclub.nation, maybe others that I'm forgetting), but the only one I could find then was an Italian instance that had closed registrations.  As for myself, I'm lucky to have a pretty good day job in the tech industry, which personally creates a lot of tension, so I look for ways to help.

So I thought I was lucky enough to have some of the knowledge and comfort with tech and the means, and without thinking too long about it, I had acquired a cloud VM registered the name and set up mastodonmusic.social over the course of a few hours.  Almost as soon as I published the existence of my site to https://instances.social directory, people were signing up, so I guess I wasn't alone in hoping for a musician or music-lover-friendly instance.

Where are we headed?

So far, I would say things are going pretty well.  I'm not through a steady-state billing cycle yet, and probably wont be for another month, but it looks like Mastodon Music will cost about $100/mo to maintain for now, without having someone like Taylor Swift drop in and destroy the server with traffic (see below on exactly why this might be a problem at this moment).  ;-)  If I'm being blunt, I can manage that and none of this is a stressor for me.  So if you're looking for a Patreon right now, I probably won't have one, at least for probably a few months.  If you were thinking of helping me, instead go check out other local posts on the server and buy a record or two from someone you're loving.  That said, I also see some challenges for this server that I want to get ahead of.


How Can You Help?

So that's probably more info than most everyone needs.  I'm not yet ready to onboard volunteers purely because I'm not yet ready to trust anyone with this project yet.  To be blunt, there's a lot of power that comes the moderator / admin and also access to the VM, and I need to figure out some safe ways to trust others with that eventually.  But, if any of this resonates with you (tech comfort level, any SRE work done in the past, and an interest to help and maintain the moderation / rules structure that exists today), please DM me at @steve@mastodonmusic.social and we can at least start talking about this space.  Again, I'm not comfortable yet granting any wide access to anyone that I don't know and trust extremely well, so for now I'll be running things. I think the last thing that musicians need is more drama or any loss of data or access on their social media.   But I recognize that I'm a single point of failure, so I'll at least do what I can to ensure this service is running for as long as possible or needed.

Thanks,


Check out new memory usage.  So upgrading from 4GB to 8GB is yes (see very last data point)