@mastodonmigration yes it totally makes sense. There might be a single point of failure, or a system not reacting well to different PDSs getting overloaded at different times, or they could just multiply their bandwidth by 20 and attack each of hte PDSs.

It's a classic "two things are true at once". On the one. hand It's a good example of both their architectural decentralization -- other PDSs survived! Tthe relay and appview are separate from the PDSs so they didn't go down!! Decentralization FTW!

Then again on the hand other it's also example of the lack of what "operational decentralization" means that in practce targeting Bluesky is enough to bring tit almost completely down. Bummer.

(@Sarahp I thought you did great job of bringing both these points up very crisply in the article!)

@thenexusofprivacy @Sarahp

Maybe. As mentioned, it is possible that a simultaneous attack on all of Bluesky's PDSs could bring them all down at once. Possible, but that is not the way one might expect independent decentralised components to go down. As you point out, perhaps there was some interdependence between all their PDSs. Maybe they share some computing resource that became overloaded. The only point is something seems curious about the proferred explanation.

@mastodonmigration

Was sind "PDS"? Bitte eine einfache Erklärung .

@mastodonmigration @thenexusofprivacy @Sarahp what would happen if six mastodon instances were all simultaneously ddosed?

@hailey @thenexusofprivacy @Sarahp

Good question. Presuming each of them were not sufficiently protected against the attack, they would all suffer either performance degradation or go down. All other Mastodon/Fediverse instances should be unaffected.

There are measures sites take to respond to DDoS attacks and it would be incumbent upon each of them to take their own defensive measures.

@mastodonmigration @thenexusofprivacy @Sarahp right. okay. and how does that differentiate with what happened here? a large number of pdses were attacked (maliciously or not), those instances were severely degraded/inaccessible, and the ones that were not being attacked remained operational (and users on those pdses could still use the network)

@hailey @thenexusofprivacy @Sarahp

If that is what actually happened.

The reason that it would be good to get a more detailed description of the incident is that this scenario does not seem to line up with the way it went down or the contemporaneous reporting.

Granted, it could have been a massive DDoS attack directed only a Bluesky PDSs, but if so, it is actually a much bigger story. If so, know who did it? The statement that it was an "accident by a 3p" just raises even more questions.

Oh come on. The scenario totally lines up with what they said. All their PDSs run the same software, so if there was a route in where an attacker could trigger a crash or cause an extremely expensive operation, or code that shared a dependency on a vulnerable component, of course they'd roll out a fix to all the PDSs in their fleet.

Back to Hailey's question about what if this happened here ... something similar did, to Lemmy a while ago. I forget the exact details but the attackers found a way to make the system do extremely expensive queries -- and it affected every Lemmy instance. After a while (i thikn it was a few days, certainly much longer than 45 minutes) the developers figured out how to mitigate it and rolled out the fix.

And other security bugs happen too, on Mastodon and Pixelfed and everything else. The dynamics the same. Developers fix them, roll the fixes out quickly to instances they control, other instances upgrade (or not). When you've got shared code, what else can you do?

@mastodonmigration @hailey @Sarahp

@thenexusofprivacy @hailey @Sarahp

Interesting speculations. That is exactly the kind of detail that it would be good to get from the company.

What we have from the Bluesky CTO is the information that the entire 'fleet' of Blusky PDSs were subject to a DDoS attack which brought them all down. Further, that this widespead DDoS attack was caused accidentally by a third party. This is pretty cryptic and begs for a more thorough authoritative explanation.

We've got information from the Bluesky CTO describing what happened. We've got absolutely no information implying that it didn't. And, we've got somebody who's been on a National Academies panel on dependable software saying "makes sense to me". Agreed that once the invident's fully resolved (which it might not be yet -- DDOS attackers often adapt their attacks in response to bug fixes) it'd be interesting to see a more detailed retrospective, but I think your expectations here are pretty unreasonable.

And while I get it that you think Paul's lying, but think about it for a second. What's the incentive to lie here?

@mastodonmigration @hailey @Sarahp

@thenexusofprivacy @hailey @Sarahp

The point here is that it is not clear what has happened. Like so much from #Bluesky we get snippets of information and are then left to fill in a coherent story by reading the tea leaves. It would be very easy to spell out what happened, but it is not forthcoming.

1/

@thenexusofprivacy @hailey @Sarahp

You raise the possibility the attack (which has also been described as accidental) was exploiting a vulnerability. Would it not be incumbent then to roll out the urgent patch to everyone? Don't other PDSs have a need to know?

Again, the point is that they are trying to run an open system protocol and they are not being open.

You also suggest the problem may not be over, and that once it is they may provide more information. Let's hope that is the case.

2/2

If you believe Paul, it's clear what happened

You raise the possibility the attack (which has also been described as accidental) was exploiting a vulnerability.

Well, the theory as of Friday morning was that this particular incident was accidental ... still, if a system goes down because of DDOS it's certainly a vulnerability -- even if it was accidental in this case, others could intentionally exploit it. So it's quite possible that the initial fix is partial and they want to do more bulletproofing before discussing the details. Or whatever.

You also suggest the problem may not be over

Indeed, that suggestion is based on Paul saying yesterday morning that he couldn't talk about the situation in detail "since we're still resolving the problem".

If they do wind up discussing more, great, but they're certainly not under any obligation to.

the point is that they are trying to run an open system protocol and they are not being open.

Who's the new Mastodon CEO? OMG they haven't announced it yet, they're trying to run an open system protocol and they're not being open!!!!!!! No, that's obviously ridiculous. "Open" doesn't mean disclosing every single detail, and it doesn't mean dislosing everythiung as soon as it happens.

@mastodonmigration @hailey @Sarahp

@thenexusofprivacy @hailey @Sarahp

Missed the part where he said that he couldn't talk about the situation in detail "since we're still resolving the problem". That suggests they will be more forthcoming when the problem is resolved. Let's hope that is the case.

Would definitely say that such transparency is important to running a successful open system protocol. It will be interesting to see how a full description of what actually happened squares with the information provided to date.

I certainly agree that transparency in general is important. On this particular issue, it's really striking that virtually nobody I've seen posting on Bluesky sees any problems with the transparency here. That's their audience -- not Bluesky-haters on Mastodon who are spinning up conspiracy theories.

@mastodonmigration

@thenexusofprivacy

Glad to hear you regard transparency important. Indeed, of the opinion that in general Bluesky users should require more honesty and transparency from the company, but have covered a lot of that ground elsewhere, and the subject at hand is this outage due to an accidental third party DDoS attack. It will be good in due course to find out more about this and how it relates to their representations regarding being a distributed system.

Yes I know your general opinion. By contrast I'm quite impressed by Bluesky's transparency -- and so are most of the other users there I talk with, including people with a lot of experience dealing with other social networks.

In terms of honestly, well, any company and open-source project is going to spin things. Still I actually think they're a lot better than other commercial social networks as well as high-profile fedi projects like Mastodon (although there are hopeful signs that Mastodon's getting better) and Pixelfed or institutions like the Social Web Foundation.

In any case, Bluesky users and the company couldn't care less what you (or anybody else who's not likely to spend much time on Bluesky any time soon) think they should do. So, if you want to influence them, take the time to get involved, make connections, and see how it looks to people there. And whether or not you want to do that, if you care about transparency and honesty, put your energy to things that are more within your potential area of influence -- by which I mean your own communications and fedi-related stuff.

As it is, you're contributing to the (very understandable) perception that Mastodon people say ignorant things about Bluesky and are oblivious to the problems with Mastodon. We've talked before about how having Mastodon in your name often leads people to think that you've got an official or quasi-official role here, and I've talked to several people who have pointed to your posts as an example of why they warn others away from Mastodon. If that's your goal, great. If not, try approaching it differently.

@mastodonmigration

PDS = "personal data store", the servers in Bluesky's architecture that store posts and media. It's possible for people to host their own PDSs, but today 99.9% of the people there use Bluesky's.

There isn't an exact equivalanet of a PDS in Mastodon; an instance stores data (like a PDS) but also does other stuff that on Bluesky is part of other servers (AppViews, Feed Generators, etc.). ActivityPods is an interesting project to do a Mastodon-compatible server that also allows for data portability ... it's sill work in progress, but promising.

@Mons1serrata @mastodonmigration