How to Scrape Cloudflare Protected Websites with Python?

Cloudflare is a potent deterrent to malicious bots, and as such, it presents the most significant barrier to web scraping. It has several layers of mitigation that must be overcome (CAPTCHA, TLS fingerprinting, browser behavior, etc.), making it challenging to bypass—especially for advanced developers. Nevertheless, pulling data from Cloudflare-protected websites is often necessary for tasks such as market research, competitive intelligence, or simply retrieving your data from an aggregator.

In this guide, we will guide you through the process of scraping websites protected by Cloudflare with Python.We will begin with the open-source cloudscraper library (to discuss simpler projects) and then discuss the other scalable and more reliable solution, Scraping Intelligence. Also we will discuss configuration, limitations and best practices to ensure that you are scraping the data in a safe and responsible way.

How Does Cloudflare Detect Bots?

Cloudflare employs both passive detection and active detection to determine whether the user is legitimate or a bot.

• Passive detection looks at IP reputation, HTTP headers, TLS fingerprints, and request patterns.
• Active detection challenges, such as CAPTCHA or canvas fingerprinting, are used as a test to confirm human-like behavior.

Often, Cloudflare returns error codes such as 1010, 1015, or 1020, which are then delivered with a 403 Forbidden HTTP status to bots using automation to scrape.

How to Use Cloudscraper to Scrape Cloudflare Protected Websites

Method 1: Using cloudscraper in Python

For sites with Cloudflare’s easier bot protection (version 1), Cloudscraper often works fine. It’s faster than using full browser automation and relatively simple.

Step 1: Configure Your Environment
Install the required libraries under Python 3.6+:

pip install cloudscraper requests

Step 2: requests vs. cloudscraper

An introductory request call typically triggers Cloudflare’s defenses:

import requests

res = requests.get("https://www.glassdoor.com/")
print(res.status_code) # → 403 Forbidden

Whereas with Cloudscraper, you might bypass that hurdle:

import requests

res = requests.get("https://www.glassdoor.com/")
print(res.status_code) # → 403 Forbidden

You’ll be shown the page content. The page’s CSS and JS may not properly load locally, but the data is there.

Step 3: Save the Cloudflare Session Tokens
When you pass the challenge, Cloudflare will issue a cookie/on the first request, and you’ll want to reuse the cookie or token for subsequent requests. cloudscraper makes it easy to get these tokens for later use:

token = scraper.get_cookie_string("https://www.glassdoor.com/")
print(token)

Ensure that subsequent requests are made with the same token and originating IP address; otherwise, the prompts to complete the challenge may reappear.

The Limitation of Cloudscraper

• Only works reliably with Cloudflare bot protection v1.
• Fails with advanced protections like CAPTCHAs or version 2 defenses.
• Doesn’t render JavaScript-heavy pages or handle headless browser behavior.

In such cases, you’ll need a more sophisticated solution like Scraping Intelligence.

Method 2: A Strong Alternative—Scraping Intelligence

When you have complex anti-bot systems or are scaling up your scraping, then consider Scraping Intelligence a powerful, maintenance-free option. Below you can see how it is even better than Cloudscraper:

• Advantages
  IP Rotation of millions of residential, mobile, and datacenter IP addresses worldwide
• Automated CAPTCHA Solving, so you no longer have to intervene manually
• JavaScript rendering using real browser environments
• Managing Headers, cookies, and sessions to mimic real-user behavior
• A 99.99% success rate against anti-bot measures, including Cloudflare and Datadome.

Quick Example: Scraping example with Scraping Intelligence

Here’s how to fetch a blog post, returned in markdown via Scraping Intelligence:

import requests

payload = {
'api_key': 'YOUR_API_KEY',
'url': 'https://example.com/some-article',
'output_format': 'markdown'
}

response = requests.get('https://api.scrapingintelligence.com/', params=payload)
content = response.text

with open('article.md', 'w', encoding='utf-8') as f:
f.write(content)

It will abstract retry, anti-bot defenses, and rendering requirements without hassle.

Comparison: Cloudscraper vs. Scraping Intelligence