--- Log opened dom dic 04 00:00:06 2022 --- Log opened dom dic 04 10:48:35 2022 10:48 -!- Irssi: #friendica: Total of 13 nicks [0 ops, 0 halfops, 0 voices, 13 normal] 10:48 -!- Irssi: Join to #friendica was synced in 22 secs --- Log opened dom dic 04 11:05:16 2022 11:05 -!- Irssi: #friendica: Total of 13 nicks [0 ops, 0 halfops, 0 voices, 13 normal] 11:05 -!- Irssi: Join to #friendica was synced in 18 secs 14:19 < simcop2387> bkil: given the way that looks to work, i'd say it sadly probably is. though maybe not as bad as quickly given the way that i think the workers work. 14:20 < simcop2387> i think it'll clog up the jobs for the workers and effectively end all other processing until that's cleared up 15:00 < simcop2387> bkil: 100% vulnerable 15:00 < simcop2387> i'm getting hit with it right now 15:03 < simcop2387> i'm adding *.activitypub-troll.cf to my blocklist now 17:15 < fikabot> 💬 > <@fikabot:matrix.org> 💬 (simcop2387) [libera] i'm adding *.activitypub-troll.cf to my blocklist now 17:15 < fikabot> 💬 17:15 < fikabot> 💬 This is wreaking havoc on smallweb Fediverse server instances, bringing rPi boxes to their knees across Fediverse ActivityPub platforms. 17:15 < fikabot> 💬 17:15 < fikabot> 💬 Certainly, a protective approach is going to need implementation on each respective ActivityPub enabled product. 17:15 < fikabot> 💬 17:15 < fikabot> 💬 That exposes another problem. Each project has to tackle the issue independently, which isn't always going to happen quickly for projects that aren't actively, or rather, maintained - such as single person projects left with little attention by the devs or largely ignored. 17:15 < fikabot> 💬 17:15 < fikabot> 💬 Also, reach and every instance deployed by average self-hosting folks will need to update their installs once the vul has been addressed and a patched release is made available.... 17:15 < fikabot> 💬 17:16 < fikabot> 💬 This could take some time to disappear. 17:16 < fikabot> 💬 17:16 < fikabot> 💬 In the interim, a reactive strategy will suffice by (I won't address nft or BSD or other Unices here) by simply creating autobans that are triggered placing IPs into a table so you're only checking a couple of rules instead of inspecting a long list of IP addys and eating up CPU time for the firewall. 17:16 < fikabot> 💬 17:16 < fikabot> 💬 You can use ipset(8) for this. 17:16 < fikabot> 💬 17:16 < fikabot> 💬 Since only an idiot would ever deploy a front facing Internet exposed host, suffice it to say that everyone here is familiar with managing their own firewall, so this solution is Easy Peasy 😎 17:16 < fikabot> 💬 17:16 < fikabot> 💬 Ask you need to do is add in a bit of extra logic in your firewal shell l scripts, or Python, Perl (however you manage your firewall(s). 17:16 < fikabot> 💬 17:17 < fikabot> 💬 Since you're using ipset you can change the action with your triggers at anytime but just changing the type of autoban you decide upon. 17:17 < fikabot> 💬 17:17 < fikabot> 💬 I prefer (generally) to do this by filtering at the firewall layer if it's not a big deal, but if it becomes a problem then I go to null routing so the source packets never even get analyzed by the firewall again after the ban goes into affect. 17:17 < fikabot> 💬 17:17 < fikabot> 💬 I also do this at a country level and for offending ISPs I may do this by typically /19 or even /16’s. 17:17 < fikabot> 💬 17:17 < fikabot> 💬 If it ever gets worse than that then I go upstream a bit and start adding in ASNs, but that's very rare (but unheard of). Usually, I don't have to do that because I've already blocked shitty countries. 17:17 < fikabot> 💬 17:17 < fikabot> 💬 So that's three arrays: 17:17 < fikabot> 💬 17:18 < fikabot> 💬 * Individual IPs from hosts run by dikheds 17:18 < fikabot> 💬 tallship: People bridged over IRC and XMPP would thank you if you were more considerate about how your message gets through there. 17:20 < fikabot> 💬 > <@bkil:grin.hu> tallship: People bridged over IRC and XMPP would thank you if you were more considerate about how your message gets through there. 17:20 < fikabot> 💬 17:20 < fikabot> 💬 I haven't actually looked yet... That was in my to-do list from three other day. I'll do that now. What might make it more digestible for them in longer posts, just a single long paragraph? 17:23 < fikabot> 💬 Not breaking lines after every second word and not inserting empty newlines between newlines could improve output drastically. An IRC message can be up to about 400-450 characters (the bridge will automatically split longer ones), while it can be much more on XMPP. Still, I still find it that if most of your comments can't fit within 400-1000 bytes, you need to split it up. 17:24 < fikabot> 💬 I usually send in one paragraph as one logical message. This carries the benefit of also providing for a way to Synapse/Postgres to search a message based on a combination of multiple keywords. You see, if phrases are so short that they usually only contain only one keyword, it could make boolean searching impossible. 17:27 < fikabot> 💬 Submitting short sentences all the time also makes it act as a kind of hard break. Free flowing text could wrap much more nicely for people on computers - not everyone chats on tiny phones. On the other edge of the spectrum, some might want to click reply to respond to certain thoughts individually and this won't work either if you are pasting in a huge wall of text. 17:33 < fikabot> 💬 Okay I'll try to tailor it for better viewing for IRC users. 17:37 < fikabot> 💬 It's prolly going to require me authoring out of band in an editor and then sending each paragraph in short succession, regaining cognizant of Libera's anti-of flood triggers. 17:38 < fikabot> 💬 bkil: what's the XMPP room? I'll take a look there as well in Gajim and Conversations. 18:00 < fikabot> 💬 You can find it in the room topic 18:01 < fikabot> 💬 `support@forum.friendi.ca` XMPP MUC 18:22 < fikabot> 💬 . 18:25 < fikabot> 💬 > <@bkil:grin.hu> You can find it in the room topic 18:25 < fikabot> 💬 18:25 < fikabot> 💬 k. Thanks! 18:35 < fikabot> 💬 > 💬 tallship: People bridged over IRC and XMPP would thank you if you were more considerate about how your message gets through there. 18:35 < fikabot> 💬 18:35 < fikabot> 💬 Okay I can't scroll back in IRC before my room join. Perhaps I can test that later this evening when things get quiet and most are sleeping, if there is such a time, lolz. 18:35 < fikabot> 💬 18:35 < fikabot> 💬 Scrolling back, it looks good here in jabber though - just how I authored the mini HowTo. 18:36 < fikabot> 💬 You can also check past IRC messages via the matrix bouncer #friendica:libera.chat 18:39 < tallship> k. I see what you mean by the paragraph break. The crlf comes through as a blank line. 18:40 < tallship> It's also automatically rate limited so as to avoid a chanflood, which is kewl 18:42 < fikabot> 💬 > 💬 tallship: People bridged over IRC and XMPP would thank you if you were more considerate about how your message gets through there. 18:42 < fikabot> 💬 18:42 < fikabot> 💬 Okay I can't scroll back in IRC before my chan join. Perhaps I can test that later this evening when things get quiet and most are sleeping, if there is such a time, lolz. 18:42 < fikabot> 💬 18:42 < fikabot> 💬 Scrolling back, it looks good here in jabber though - just how I authored the mini HowTo. 18:42 < tallship> hm... 18:43 < tallship> Edits in XMPP are problematic for the IRC bridge. 18:44 < tallship> I'll check the bouncer now. 18:46 < fikabot> 💬 On XMPP at the protocol level, you are only allowed to edit your most recent message. 18:47 < fikabot> 💬 Yah that's standard behavior for Jabber :) 18:48 < fikabot> 💬 Yep, I set up a rate limit of 12s between messages as such messages have caused DDoS protection kicks in the past! Interestingly, Matterbridge stopped serving up your rate limited lines the moment I replied, so you may also want to think about message loss as well then. 18:48 < fikabot> 💬 6100ms to be more exact. MessageLength for splitting is at 360 bytes. 18:48 < fikabot> 💬 Matrox edit check 18:49 < fikabot> 💬 6100ms to be more exact. MessageLength for splitting is at the default (400 bytes). 18:54 < fikabot> 💬 > <@bkil:grin.hu> Yep, I set up a rate limit of 12s between messages as such messages have caused DDoS protection kicks in the past! Interestingly, Matterbridge stopped serving up your rate limited lines the moment I replied, so you may also want to think about message loss as well then. 18:54 < fikabot> 💬 18:54 < fikabot> 💬 Yes, although it's basically still providing the intended nice formatting for the 200+ users in the matrix room though, vs the 15 or so in the IRC Chan. 18:56 < fikabot> 💬 Aside from the truncation issue in IRC, it's still formatting nicely there, and perfectly, from what i can see in the XMPP room. 18:56 < fikabot> 💬 I'm going to go look at it in the bouncer room though to really take s good look though 18:57 < mason> How all of this looks from an IRC client: https://imgur.com/a/Kw815O2 18:58 < fikabot> 💬 > <@fikabot:matrix.org> 💬 (mason) [libera] How all of this looks from an IRC client: https://imgur.com/a/Kw815O2 18:58 < fikabot> 💬 18:58 < fikabot> 💬 mason: Oh thanks for that :) 19:04 < fikabot> 💬 Okay I'mma scroll back in the bouncer room and examine the formatting of that long mini-HowTo 19:17 < fikabot> 💬 bkil: Hey I'm but seeing any history in that bouncer room going back to 03 July - #friendica:libera.chat 19:18 < fikabot> 💬 Except for my join in the IRC Chan earlier today and a join by someone else on 01 December. 19:23 < fikabot> 💬 Well, the mini HowTo looks really good here in Jabber, formatting is perfect. 19:24 < fikabot> 💬 I'll test that edit in Matrix and see how that acts in IRC so I know, and I already know how it behaves in IRC for a Jabber edit now 19:27 < fikabot> 💬 Matrix edit check - edited now ;) 19:46 < tallship> Okay now I'm seeing history in the Matrix bouncer room - but only for the past hour 19:47 < tallship> I guess it will just accumulate from here out 21:40 < fikabot> 💬 So, after going through the fork bomb code, the mechanism is the following. It synthesizes a new user for each webfinger query. That user has a featured collection that contains a single note. This note mentions two other random usernames. A featured collection holds statuses pinned on the user profile: https://docs.joinmastodon.org/spec/activitypub/#featured 21:41 < fikabot> 💬 The root cause of the issue is that Mastodon will recursively query all pinned statuses of each mentioned user. Why would you do that in the first place, to show that information when the user hovers over the username? But to serve such a hover action, a single level of indirection should suffice. So it could be related to eagerness. 21:46 < fikabot> 💬 Featured collections landed in Friendica 2022.06 via https://github.com/friendica/friendica/pull/11506 Telltale comment: "Fetch featured posts through a worker. This prevents possible endless loop. (the contact update can be called via the process that fetches posts, so this can cause a problem)" It can be disabled with the config option system->fetch_featured_posts. 21:50 < fikabot> 💬 Makes sense from what I saw. It'd still be nice to have a way to limit that recursion. I had 86k domains in my db after getting hit. 21:51 < fikabot> 💬 wow 21:51 < fikabot> 💬 is this nefariousness or accidental? 21:51 < fikabot> 💬 when it shows up 21:51 < fikabot> 💬 simcop2387: on Friendica? 21:51 < mason> Hank, nefarious. 21:52 < fikabot> 💬 Hank: It was published as a proof of concept so that fediverse servers would fix the issue. 21:52 < mason> bkil: It was published, then someone else decided to do it? 21:52 < mason> "Here, look what can happen" and then crushing instances wouldn't be neighborly. 21:53 < mason> Anyway, yeah, simcop2387's fallout was on Friendica. 21:53 < fikabot> 💬 so it is assuming nefariousness but hasn't happened yet 21:54 < fikabot> 💬 good to be proactive because you know the bomb throwers 21:54 < fikabot> 💬 will pull that shit one way or the other 21:55 < simcop2387> bkil, yea. want a copy of the logs? 21:55 < fikabot> 💬 mason: But did Friendica itself trigger the recursion or was it just federating posts from Mastodon (that it then proceeded to look up via the above path)? 21:55 < fikabot> 💬 It would be nice if you could share a relevant snippet. 21:55 < simcop2387> sure let me go grab it. 21:57 < simcop2387> this'll take a minute.... my log file is 3.2GB after that :) 21:57 < fikabot> 💬 I hope you don't want to send more than 1MB of compressed log... 21:57 < simcop2387> i have debugging on still so there's a log 21:57 < fikabot> 💬 😱 21:57 < simcop2387> lot* 21:59 < simcop2387> gonna grab 1M lines from it, 50 before and after each occurance of activitypub-troll.cf and compress it up. 22:00 < simcop2387> ok that's still 360M, 10k lines of that 22:00 < fikabot> 💬 It would help more to grab like 1k lines before and 10k after the _first_ occurrence of that domain, not every occurrence. It should be pretty repetitive. 22:01 < fikabot> 💬 grep -m1 -B1000 -A10000 I think 22:01 < fikabot> 💬 it is pretty repetative, let me redo the grep 22:01 < fikabot> 💬 there's only 3160137 total lines 22:02 < fikabot> 💬 with the 50-50 22:02 < fikabot> 💬 Are you subscribed to a relay? 22:03 < fikabot> 💬 i'm not even sure how to do that 22:03 < fikabot> 💬 mind if i send the logs via matrix? 22:04 < fikabot> 💬 Fine with me. But please zip it before 22:04 < fikabot> 💬 Although, for privacy, this is better https://pb.envs.net/ 22:04 < simcop2387> right now there's basically nobody on it but me and two other people who aren't actively using it yet so no real privacy concerns of note right now 22:06 < simcop2387> i like gzip over zip for things like that since you can use zless to read it without directly decompressing it 22:08 < fikabot> 💬 Did you switch on logging only after you started having issues? From the looks of it, your host table has already been poisoned by this time. 22:09 < fikabot> 💬 Sure, gzip is fine. 22:09 < simcop2387> no it's been on for a few days while i was debugging self inflicted issues 22:10 < fikabot> 💬 Do you have fetch featured posts enabled in the config? Anyway, I don't seem to find any post fetching in this snippet. It's only a huge iteration of `UpdateServerPeers` over all hosts in your database. 22:10 < simcop2387> i'm going to make sure i have the logs backed up in case you can think of anything to search for in them related to the attack 22:10 < simcop2387> probably do, let me see 22:14 < simcop2387> where is that setting in the config? 22:22 < simcop2387> can't find it in the ui since i don't know what i'm looking for but checking the config in the DB there's nothing set for that key 22:23 < simcop2387> so i don't believe i have it set based on the default config file too. so i don't have that enabled 22:23 < fikabot> 💬 I'm starting to see better. So Friendica talks to each of his peers (at least once every 7 days) and polls each of their directory to explore every neighbor of theirs and save it into its gserver table. So after Mastodon did the fork bomb, it has poisoned its server directory and that was in turn copied over to Friendica. 22:24 < fikabot> 💬 And just polling all these dumb servers individually wastes precious space in the queue and a bit of network bandwidth, but at least it's not infinite. Wonder how this could be handled better automatically. 22:24 < simcop2387> that makes some sense, and then because i have the setting on to fetch that for auto-discovered contacts it probably is why i was affected 22:25 < simcop2387> misskey is dealing with it with a recursion limit, maybe some kind of TTL/six degrees of the fediverse type restriction would keep it from blowing up to such huge numbers easilyt 22:26 < simcop2387> but that might still run into issues with already poisoned servers out there giving out their gigantic directories 22:27 < fikabot> 💬 Oh wow, it even queries through the-federation.info every day for servers... 22:28 < simcop2387> geeze i wonder how much their server costs are if other systems are doing the same 22:28 < simcop2387> also that explains how it's discovered a number of other weird mastodon instances with impolite names 22:30 < fikabot> 💬 It queries both the peers of a peer and their server directory (if public). 22:31 < fikabot> 💬 But why do you need server discovery in the first place? It seems futile to attempt to replicate the global directory around the world on a tiny instance. What's the benefit? 22:32 < simcop2387> i saw no reason to not have it on 22:32 < fikabot> 💬 What was the name of the setting? 22:33 < fikabot> 💬 I haven't checked, but I think matrix also only federates lazily - it discovers servers when they start sharing data (i.e., join in a room with one of their users). 22:33 < fikabot> 💬 https://matrix.org/_matrix/media/v1/download/matrix.org/UanWeHxNIxzLXveJhaJQQCDl 22:33 < fikabot> 💬 i assume it's this section 22:35 < fikabot> 💬 Okay, thanks. So it should serve being able to find users based on partial content or metadata, such as real name and tags. So if you added every contact of yours through their URL or user ID you would not need this I think. 22:36 < fikabot> 💬 yea and since i'm hoping this instance once i publicize it will be useful to any perl programmers that want to get on the fediverse i feel like it should be useful to have that 22:36 < fikabot> 💬 I wonder whether it would purge out old entries after a while if one disabled these options. 22:36 < fikabot> 💬 i believe there is a cleanup job somewhere 22:40 < simcop2387> so it does look like then that friendica is less impacted by the whole vulnerability and also has some ways to deal with it already even if it is by turning off a feature. a fix would be nice still though, but doesn't look like it's nearly as impactful as it is to mastodon 22:41 < fikabot> 💬 I know items, conversations and media are cliened up. The comment are a bit vague about other things, I would guess gserver would remain poisoned as of now. But I found an option to help you: I think setting `gserver_update_limit` to a smaller number (such as 1) could be helpful and give a chance for others to catch such abuse so you could block the offending servers before it fills up the table. 22:42 < fikabot> 💬 yea i'll give that a look. thankfully friendica is already very light on resources compared to mastodon and the like. so even with 86k entries everything was functioning fine (though the page to block the domain took several minutes to transfer/render :)) 22:43 < fikabot> 💬 i've had to turn my cat's fediverse site off for the time being while i wait on the fix for mastodon. don't want to deal with cleaning it up there and it's only for my cat to be posted. in the mean time i can link to his federated posts :) https://perl.social/display/8667d310-e474ac026489b351-66c0880d 22:47 < fikabot> 💬 You can also make the process lighter by increasing system->poco_requery_days. The default if missing is 7, and you could increase it to 30 or 90 for example. 22:48 < fikabot> 💬 yea something to tune when/if it gets bigger 22:49 < fikabot> 💬 But this also means that there exist another trivial fork bomb if they just generate a bunch of random peers and/or local server directory entries recursively for Friendica to endlessly add to its table. 22:50 < fikabot> 💬 unfortunately that's likely to be the case. can mitigate that with the local contacts only mode though, since it wouldn't end up discovering anything other than normal humans generally speaking. 22:51 < fikabot> 💬 and shouldn't recurse too far that way actually 22:53 < fikabot> 💬 Yes. The tragedy of attempting to build a global directory based on unauthenticated gossip is that you will run into this sooner or later. Hence why I also advocate for only radius-based discovery. I mean - why would you want to discovery completely random people, assuming the federation had been deployed worldwide having a small world topology? 22:53 < fikabot> 💬 in my case i'm wanting it for people to be able to search for other perl programmers that they know are out there. not sure how well that's going to work in practice so i'll probably restrict it to local only once it grows enough 22:58 < fikabot> 💬 In practice, you would open an instance/directory that only enlists Perl programmers and you curate participation in it. No other plausible way against SEO spam anyway. 22:59 < fikabot> 💬 i'm hoping i can control that spam nicely once it happens by turning on the captcha support and possibly other things too 22:59 < fikabot> 💬 Doing a rewrite of Friendica in Perl CGI on free hosting would be funny (tongue in cheek?) 22:59 < fikabot> 💬 I'll try to sleep a bit on this, thanks for cooperating in the analysis so far. 23:00 < fikabot> 💬 there's an old one, https://luc.frama.io/mightynetwork/ 23:01 < fikabot> 💬 https://framagit.org/narf/mightynetwork # for the source 23:02 < fikabot> 💬 wait wrong source code, dang where was it 23:03 < fikabot> 💬 eh it doesn't matter it's old and very unfeatured, basically just implements the activitypub api and that's about it --- Log closed lun dic 05 00:00:05 2022