Mastodon 🐘

4 posts4 participants0 posts today

𝕂𝚞𝚋𝚒𝚔ℙ𝚒𝚡𝚎𝚕Open source devs are fighting AI crawlers with cleverness and vengeanceAI web-crawling bots are the cockroaches of the internet, many software developers believe. Some devs have started fighting back in ingenuous, often humorous ways.🌐 <a href="https://techcrunch.com/2025/03/27/open-source-devs-are-fighting-ai-crawlers-with-cleverness-and-vengeance/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://techcrunch.com/2025/03/27/open-source-devs-are-fighting-ai-crawlers-with-cleverness-and-vengeance/</a><a href="https://chaos.social/tags/opensource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensource</a> <a href="https://chaos.social/tags/software" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#software</a> <a href="https://chaos.social/tags/crawler" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crawler</a> <a href="https://chaos.social/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ai</a> <a href="https://chaos.social/tags/dev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dev</a> <a href="https://chaos.social/tags/development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#development</a> <a href="https://chaos.social/tags/aicrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#aicrawlers</a> <a href="https://chaos.social/tags/webcrawler" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#webcrawler</a> <a href="https://chaos.social/tags/web" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#web</a> <a href="https://chaos.social/tags/internet" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#internet</a> <a href="https://chaos.social/tags/software" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#software</a> <a href="https://chaos.social/tags/fightback" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fightback</a> <a href="https://chaos.social/tags/it" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#it</a> <a href="https://chaos.social/tags/codeberg" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#codeberg</a> <a href="https://chaos.social/tags/code" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#code</a>

PUPUWEB BlogDevelopers report aggressive AI crawlers overwhelming open-source infrastructure, with LibreNews citing up to 97% of traffic from AI bots on some projects. <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenSource</a> <a href="https://mastodon.social/tags/TechNews" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TechNews</a> <a href="https://mastodon.social/tags/AIcrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AIcrawlers</a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bots</a> <a href="https://mastodon.social/tags/LibreNews" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LibreNews</a> <a href="https://mastodon.social/tags/DeveloperCommunity" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeveloperCommunity</a> <a href="https://mastodon.social/tags/Infrastructure" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Infrastructure</a> <a href="https://mastodon.social/tags/TechIndustry" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TechIndustry</a>

@990000@mstdn.socialAI bot nightmare <a href="https://bsky.app/profile/gergely.pragmaticengineer.com/post/3llco5l2zsk23" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://bsky.app/profile/gergely.pragmaticengineer.com/post/3llco5l2zsk23</a><a href="https://mstdn.social/tags/AI" class="mention hashtag" rel="tag">#AI</a> <a href="https://mstdn.social/tags/ArtificialIntelligence" class="mention hashtag" rel="tag">#ArtificialIntelligence</a> <a href="https://mstdn.social/tags/Meta" class="mention hashtag" rel="tag">#Meta</a> <a href="https://mstdn.social/tags/Facebook" class="mention hashtag" rel="tag">#Facebook</a> <a href="https://mstdn.social/tags/AICrawlers" class="mention hashtag" rel="tag">#AICrawlers</a>

Hacker NewsDevs say AI crawlers dominate traffic, forcing blocks on entire countries<a href="https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/</a><a href="https://mastodon.social/tags/HackerNews" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HackerNews</a> <a href="https://mastodon.social/tags/AItraffic" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AItraffic</a> <a href="https://mastodon.social/tags/AIcrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AIcrawlers</a> <a href="https://mastodon.social/tags/DevsNews" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DevsNews</a> <a href="https://mastodon.social/tags/InternetRegulation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#InternetRegulation</a> <a href="https://mastodon.social/tags/CountryBlocks" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CountryBlocks</a>

Jerry on MastodonOver the past 24-hours, <a href="https://hear-me.social/tags/Facebook" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Facebook</a> has been the most determined <a href="https://hear-me.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> crawler to scrape data from this server, by far. They never succeed. <a href="https://hear-me.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a> always blocks them for being one of the unwanted AI bots. What is interesting though is its determination to read one particular user invite. I wonder how it picks the other posts it wants to read.<a href="https://hear-me.social/tags/aicrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#aicrawlers</a>

Dr PenProtecting your blog from the dead eyed <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> crawlers. You can experiment with specific robots txt, and I also run a script in htaccess. I think there are metadata properties you can declare. None of this stops your pages being crawled but may afford some legal protection. (See the German Laion case recently). I'm doing a short blogpost on this, soon.<a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://mastodon.social/tags/aicrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#aicrawlers</a> <a href="https://mastodon.social/tags/htaccess" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#htaccess</a>

Cannot read property 'name' of undefinedI wonder if you could devise some sort of reverse LOIC for the damn AI bots crawling the webpages.95% visitors are bots and crawlers.<a href="https://social.zlatko.dev/tags/bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#bots</a> <a href="https://social.zlatko.dev/tags/aicrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#aiCrawlers</a>

Dim SimpleNever used this software, so I'm not officially endorsing, but seems interesting."This is a tarpit intended to catch web crawlers. Specifically, it's targetting crawlers that scrape data for LLM's - but really, like the plants it is named after, it'll eat just about anything that finds it's way inside."<a href="https://zadzmo.org/code/nepenthes/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://zadzmo.org/code/nepenthes/</a><a href="https://social.chinwag.org/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://social.chinwag.org/tags/AICrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AICrawlers</a> <a href="https://social.chinwag.org/tags/LLM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLM</a> <a href="https://social.chinwag.org/tags/nepenthes" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nepenthes</a>

Recent searches

Search options

Administered by:

Server stats:

#aicrawlers