Nine's web app protection blocked 96m bad requests in 2024 Olympics

Nine Entertainment blocked up to 96 million web requests from bots and potentially malicious traffic during this year’s Paris Olympics.

The media organisation's Fastly web application firewall filters 1.2 billion daily web and application requests, including on its brands the Australian Financial Review and Nine News.

However, during the Olympics, this daily number of requests increased “four-fold” with “some peaks even higher on some days such as the swimming”, Nine technology director of publishing and enterprise practices Andre Lackmann said.

“Across all of those services, we have a 70 percent CDN offload, so about 30 percent of that traffic is coming through the WAF itself, and about one-to-two percent is being blocked.

“If you do the sums there, there are millions of requests being blocked that we would have a lot of challenges managing in any other way.”

Speaking at the Fastly Xcelerate conference in Sydney, Lackmann said the scale of the traffic loads across Nine’s multiple brands led its technology team to use the Fastly's managed security service.

"It’s super challenging for us to staff and manage an engineering organisation that has internet security experts on a 24/7, 365-day basis,” he explained. “That’s where MSS really stepped in.”

Nine first began trialling Fastly in “its early phases” in 2017, before migrating from its legacy WAF to Fastly’s in 2023.

“About 20 million Australians visit one of our properties every month,” Lackmann said.

"At the end of 2023, having a bigger merged company and having two of everything in some cases, we were able to consolidate all of our WAF and internet security into a single platform with Fastly.”

Speaking about Nine’s use of Fastly’s MSS, Lackmann said the company found unusual traffic coming through its backend on the first day of this year’s State of Origin.

“We use Slack to trigger incidents, and this was no different,” Lackmann said. “We launched an incident here, and then some of us broke out and needed to speak to the MSS team about the unusual traffic coming through."

Targeting AI scrapers

Lackmann also touched on Nine’s challenges with preventing artificial intelligence bots from lifting its brands’ content for use in large language models like Perplexity.

This is especially important for Nine’s subscription-based brands like the AFR, The Age and the Sydney Morning Herald.

According to Lackmann, "AI services are increasingly able to summarise that content” for these brands.

“We’re taking a more strict stance,” Lackmann said. “We just updated our robots.txt to be able to enable blocking as much as robots.txt blocks anything.

“Beyond that, it’s about scraping at a rate-limiting level, but there’s challenges in being able to do it at a more granular level.

“On our more costly side, so the Financial Review, where the content we aim to protect is at a higher level, we have put some mitigations in place that make it more difficult to get to the whole of the content. That has been successful to a point.”

However, upon testing Perplexity’s ability to pick up an article concerning pharmaceuticals in Australia, Lackmann noticed that the AI scrapers were not as smart as initially feared.

“[Perplexity] very confidently spoke about what that article was about,” he said. “But what it actually was doing was just choosing three random ASX-listed pharmaceutical companies and largely just made up the whole of the response.

“It’s a challenging space, and one we’re reacting to,” he added.