minus-squarepewter@lemmy.worldtoTechnology@lemmy.world•AI crawlers cause Wikimedia(The umbrella organization of Wikipedia and a dozen or so other crowdsourced knowledge projects) Commons bandwidth demands to surge 50%.linkfedilinkEnglisharrow-up1·18 hours agoThere’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages. linkfedilink
There’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.