I’m going to come right out of the gate here by using something that’s a bit of a “dirty word” around the Kasada offices. Cover your eyes, because here it comes:
WAF. Web Application Firewall.
You may have noticed we try to avoid using the term “WAF,” or refer to our Polyform product as a “WAF replacement,” even though it very clearly provides all of the same (alleged) benefits of a classic WAF. This is primarily due to our teams consisting of longtime hackers, DevOps folks, and sysadmins who have (what seems like) a lifetime experience of dealing with WAF technology.
I recall back in the Dark Age of the Internet (the mid-90s) when WAFs first began to emerge, and I was working in IT during the “Dot Com Boom” of the Early 2000s when E-Commerce exploded and the main deluge of WAFs really began to flood the market. All of the products did pretty much the same thing: block suspicious POST strings, and if an attacker was being particularly obnoxious, block their IP. A devastating response in Ye Olde Days when obtaining a new IP was a relatively arduous task.
Fast-Forward 20 years to our present utopia of 2019. Artificial Intelligence and Machine Learning have helped WAF technology to transcend existence itself, stopping attacks by leveraging Quantum Computing to simply only allow requests through that come from perfect realities where attacks no longer exist!
I am, of course, joking. Most WAF attack detection and response technology hasn’t changed much, if at all. IT enterprises are spending incredible amounts of money on appliances and support contracts on ancient methodologies that weren’t even particularly good when they were new. In the meantime, attack techniques and technologies have continued to advance, and these days bypassing a standard WAF is usually trivial.
Back in my Penetration Testing days, I used to LOVE coming across exposed SQL interfaces because I could bust out sqlmap, an automatic attack tool with some seriously amazing WAF bypass methods built right in. I’d set it, forget it, get a sandwich, and come back later to a nice local database dump, all right past the WAF’s “watchful” gaze.
Tie this to a bot that trolls around all day looking for SQL servers (or just leverages Shodan), and you’ve got…well, you’ve got a huge portion of the Internet’s “white noise” I referenced in my last blog post. What do most WAF’s do about this in our far-off future of 2019? They block requests, and they blacklist IPs. They also generate an unending deluge of false-positive alerts, but that’s a rant for a different blog post. So aside from that alert flood, and the inability to actually catch anyone who’s actually trying, this all sounds like a decent defensive plan, right?
Lying to Yourself with Blocking & Blacklisting
Blatant request blocking and IP blacklisting are flawed at their cores. Blocking (HTTP 403, no response, closing the connection, etc.) has the problematic effect of directly alerting the attacker that what they’re up to isn’t working out. It gives them an easily-identifiable prompt stop, investigate the problem, correct for it, and try again. It does barely anything to deter. Your best bet for these cases is to send a dynamically-generated page that can’t be fingerprinted for automatic identification. For example: if you’re an airline dealing with price scrapers, you could respond with an identical page (or API response) for the one requested, but with erroneous pricing data.
As for the IP Blacklisting… Just… Just don’t. Please stop. You’re dynamically generating your own unwinnable game of Whac-A-Mole, and risk inadvertently blocking potential customers and forcing them to use a competitor. Many defensive engineers are vaguely aware that attackers “use VPNs” and “hacked servers” to launch their attacks while also hiding their identities, but the rabbit hole of cloaking methodology runs surprisingly deep. The use of residential IPs as “exit nodes” for bots is exceptionally common, as attackers know you’re not going to block an entire residential ASN (or possibly even a single IP) because that’s where the majority of customers come from.
But isn’t getting your hands on thousands of residential IP addresses extremely difficult, as it would require thousands of residential ISP accounts, which is insanely expensive?
At the very least, wouldn’t it require purchasing a botnet of compromised residential machines on the Dark Web, which even then is extremely unreliable?
There are a surprising number of legitimate, legal sources leasing out residential IPs for anyone to use. Let’s take a look at Luminati.io, for example. This company is operated by the purveyors of Hola VPN, an extremely-popular freemium VPN service many use to stream TV broadcasts from other regions in their homes (you know, from their residential IPs), and the overlap between these two companies is nothing short of terrifying. If you read through Hola’s End-User License Agreement (BECAUSE YOU DO READ THOSE, RIGHT?), you’ll see that by using Hola, you agree to allow the parent company to use your VPN connection as a tunnel for OTHER traffic. Traffic that is not yours. Traffic from anyone who wants to pay Luminati for the use of your home IP. Traffic from bots.
Another notable player in this game is MonkeySocks (don’t ask). They offer an SDK that can be rolled into ANY mobile app that pays the app creators to allow connections on your device to be reused for it’s IP proxy network, which is subsequently used by bots and other malware. Of course, the fact that this is happening is once again buried deep in the apps terms of service, which you’ll never read.
The Attacker vs. The Attack
Even if we ignore all of that, blocking IPs does nothing to stop the attackers, only temporarily slow down the attack. Kasada’s Polyform telemetry system has the unique ability to differentiate between the general attack traffic and attacker troubleshooting traffic. That is, we’re able to watch the attacker remotely troubleshooting issues and coordinating new attack methodology to roll out to their bots, and what we find is generally unsurprising:
As you can see, the attackers are often operating out of completely different geolocations than the actual bots. As such, blacklisting IPs, ASNs, or even entire countries’ IP blocks is not effective, as all this does is let the attacker know they need to just use IPs from another source. An easy prospect in 2019. Combating this requires a solution that is able to stop the attacker’s automation right out of the gate, at the very first HTTP request, thus preventing them from ever developing a functional targeted automated attack framework. This not only eliminates a huge amount of the attacks that are leveraged against you, but as it is now directly affecting a human being, has the massive benefit of eventually demotivating the human from continuing their task.