Who loves ya, baby?
-- found this on Sorcery of the Spectacle,
funny to see that subject pop up again. it was what drove me insane enough to find this sub in the first place.
at any rate, the problem is not the bots. I thought it was, but those are just part of the parasitic ecosystem.
but to get that, first we need to take a few steps back on web history, ad serving, UX, tracking technology and media advertising.
too lazy to gather links, but you know, do your googlin'.
I assume that most of you are fairly web literate here, but I'll try to go down into the bare bones as much as possible for those who aren't.
so let's start with a basic question - what is a web visitor anyway?
from the standpoint of a normal person, that would be a person browsing a given website or piece of content. from the standpoint of technology however all you know is that some device has downloaded content from your server using the http protocol. thanks to the wonderful technology of web browsers, you can plant browser cookies on a visitor - stuff that's used to remember if they logged in, what their preferences are, stuff that your service can read from the device. it also serves usually very basic telemetry like last visit time, session time, and so on.
this, over time has evolved in what we call browser fingerprinting, a convoluted bunch of technology that allows websites and web services to uniquely identify you.
it still doesn't know if you're a human or not, but from the standpoint of the web technology, you're a visitor.
now back in ye old days of the web, when the first banner ads were springing up, these were important questions. most consumers were still to be reached on traditional media channels, and ad spend would have to be justified somehow on the risky ventures of online business. so beyond traditional polls that would infer the value of visitors, websites would start tracking number of visitors, time on page and so on. these were used to milk the advertising cow so to speak, and it gave in to some funny developments like the creation of the popup ad - if I recon correctly on geocities, where they would just but the ads everywhere until some big auto company noticed that they're appearing on porn sites. so - put the ad in the popup, and you can claim it's not in the context of porn!
around this point in time the online ad business is still pretty low tech. you actually have to call a physical human being, they send you ppts and pdfs, you send back image files and excel sheets, you wire money, the ads run, and so on. this is called direct sales, and it's tracked again by counting a bunch of visitors, and telling you how much impressions and clicks your marvelous creatives and ad budget generated.
now enter google - or more precisely, a technology firm called doubleclick that was to be acquired by google. they developed a tool for automatic ad serving, later to be called programmatic advertising, that keeps the pesky sales dude out of the loop and achieves reasonable amounts of scale for a more hefty price - after all, if the sales are automated, you get a bidding war for attention between different advertisers, and you're paying for clicks.
so you can see how this was a strategic move for google - they already had the most valuable data available in this situation. they were seeing in real time what people were searching for, and using the programmatic ad serving system, you could effectively bid not just for general attention - but for attention with an intent to buy.
...and the way that google got this data is because they indexed the web, using bots. at least GoogleBot would identify itself as a site visitor, but in the meantime they developed a service for websites to comprehensively track their own visitors and where they were coming from and what they were doing on your website. incidentally, you could also put on google's ads on your webpage to earn quite a bit of money, as content relevant ads would be shown through the doubleclick system.
this kicked off two things:
one, the ability to classify your website visitors into different clusters and segments allowed businesses to start tailoring the appearance of the website or service to fit that specific audience segment, starting off the great fracture - segmentation of the web (in the sense that two people viewing the same website at the same time were not seeing the same thing)
two, it created a very strong financial incentive for people to trick google into thinking they were having actual human visitors that would click on ads, when in fact they were bots. in an even funnier twist, some of them were from browser hijackers, commonly known as malware at the time, which google cross-financed. look up download valley and crossrider.
at the cross section of the above two, you had one interesting twist: websites that would appear differently to the security bots or the compliance officers of Google as they would to fake visitors or malware jacked human beings. the former would get a benign looking website, while the latter would get bombarded with auto clicking ads.
this kicked off the billion dollar arms race called online advertising fraud.
I'm not here to shed a tear for big money corps bleeding money. the real fallout lay somewhere else, but for that you have to understand that you never really saw the real internet, you only saw your corner and the one that was personalized for you.
but if you ever had the pleasure of watching daytime TVs or off channels and witnessing the ads, you could kind of infer what kind of audience must be watching these shows generally. from quite clear rip offs to magic number lotteries and television fortune telling, these sorts of programming was aimed at the most gullible, bought for pennies, where the smallest audience portion had to be converted into a money making operation.
...and with audience segmentation and data gathering, that was now possible at unprecedented scale, automatically. so big was the scale in fact, that it gave birth to an entire new beast of an industry called affiliate marketing, where instead of a regular payroll, you'd get a cut of the sale should you figure out an angle on where to push whatever fucking bullshit the vendors were offering to whoever the fuck would be dumb enough to click on an ad and buy. (the funniest story I recall was someone pulling five figures a month because he figured out that if you buy ads on anime-hentai pages and sell PUA shit courses and e-books you'd make a killing)
at any rate, affiliate marketing brought with it the killer landing page, the thing that's supposed to hammer the nail in the coffin once you get through the banner ad. the earliest form of deceptiveness in memory comes from various pirate sites, that had fake download buttons as banner ads and virus alerts as the landing pages. but then at some point, some schmuck realized that for certain type of products, like diet pills or forex trading or whatever, the best lander is in fact a fake news page that comes packed with comments and all. that would convert like crazy, because it had the appearance of social proof.
until at least the lawsuits came raining down, and these sorts of landing pages and campaigns for being banned left right and centre on all platforms. which just launched a new arms race as the campaigns would be disguised for the bots doing the checkups, and aged facebook profiles would start selling for like 5K USD - these people were making 30-40k a day, they could afford to spend that much to continue running the shop.
speaking of facebook - it came just about the right time for the shit to brew max total. first they were unprecedented in the amount of data they were getting off of their users, and they came just in time to catch the full swing of what we call the 'responsive web' - that no user at the same time would see the same thing on their page, it was all allocated through an intricate web of recommendations, running real time, based on previously gathered and forecast behavioral data.
it also ran on one simple premise: take over the starting page position from google for most people, then they do not have to justify, ever, any ad spend that takes place on their platform, as long as it performs. furthermore, it was completely lacking any revenue share sort of scheme (save for the short period of facebook gaming, see Zynga), thus there was no incentive for the amount of bot traffic that the previous internet era had bred. instead, it came with an entirely different one - bots that would offer social proof in the way of shares and likes, but would not directly risk the business model, thus giving no incentive for facebook to fight them. (note that google didn't do much jack shit either besides indiscriminately penalizing websites it deemed suspicious when they reached critical payout thresholds)