skilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 10 months agoX updates its Terms to prohibit crawling/scraping of its datastackdiary.comexternal-linkmessage-square44fedilinkarrow-up1360arrow-down122file-textcross-posted to: [email protected]
arrow-up1338arrow-down1external-linkX updates its Terms to prohibit crawling/scraping of its datastackdiary.comskilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 10 months agomessage-square44fedilinkfile-textcross-posted to: [email protected]
minus-squareIwasondigg@lemmy.onelinkfedilinkEnglisharrow-up6·10 months agoTook a look at their robots.txt, it appears to block all bots except Google.
Took a look at their robots.txt, it appears to block all bots except Google.