Peak Season? Time to Scale

-25% OFF Residential Proxies
–15% OFF Residential Light

Peak Season? Time to Scale

Stock up on traffic for less!

 

-25% OFF Residential Proxies
–15% OFF Residential Light
GET STARTED

How to Scrape Bing Search Results: Data Fields, Pagination, and Proxies

How to Scrape Bing Search Results: Data Fields, Pagination, and Proxies

Quick Answer

Download Bing’s HTML with Requests and pass that response to Beautiful Soup. Extract one record per organic listing, keeping its title, destination, and snippet together and recording where it appeared. Attach the query and capture details to the saved page. Then follow the returned Next link and check for new destinations before adding another page to the export.

The example below does this with ordinary HTML responses. It also accepts a proxy connection for collections that need a chosen network exit. Rendering, geographic settings, and the limits of the retrieved sample are covered where they affect the output.

Key Takeaways

  • A Bing tracking address can conceal a destination you have already collected. Compare the decoded addresses.
  • Count organic listings independently of advertisements, answer panels, and sitelinks.
  • Treat pagination offsets as requested settings. They do not establish the ranks actually retrieved.
  • Retain repeated appearances when studying rankings, but count unique destinations separately when building a URL list.
  • Report where collection stopped, including repeated pages and unverified responses.

Which Bing Data Source Should You Use?

Choose between collecting permitted page responses and a data supplier that explicitly supports Bing. Before integrating a supplier, ask for a response to one of your own queries and a later page from the same search. Inspect the returned fields and market, then check what the agreement lets you retain.

For managed collection, the engine list on MangoProxy’s SERP Scraper API page includes Bing. The published request example targets Google, so confirm the Bing-specific inputs, supported fields, and pagination behavior before choosing that route. The example in this article instead shows a collector that reads Bing page responses directly.

An older tutorial may direct you to Microsoft’s discontinued Bing Search APIs. Their retirement date was August 11, 2025. When evaluating a current third-party tool, inspect its sample response instead of assuming the former Microsoft field names still apply.

Establish permission for collection, storage, and the intended use before scheduling a run. In particular, the Microsoft Services Agreement forbids circumventing access restrictions and includes impermissible scraping in that prohibition. The example below assumes that permission is in place and stops on a failed request.

Define the output narrowly. A list of organic destinations is a different deliverable from a complete rendering of the search page, with advertisements, maps, related questions, and generated answers. The example below collects organic listings only.

What Fields Belong in a Bing Search Dataset?

Store search context once per captured page and attach its reference to every result. This keeps the circumstances of a search available without confusing them with facts about the linked website.

LevelFieldWhat to retain
SearchQueryExact submitted text; keep any displayed correction separately.
SearchRequested contextCountry or market setting, interface language, filters, and client configuration.
PageCapture referencePage identifier, retrieval time in UTC, requested URL, and final response URL.
PageOutcomeAccepted, repeated, unreadable, denied, or stopped at the run limit.
ResultOrganic positionOrder among organic listings on this captured page.
ResultTitle and snippetText shown in the selected listing; an absent snippet stays null.
ResultOriginal linkThe headline’s href, preserved as source evidence.
ResultDestination URLA direct link or a destination extracted from a recognized Bing wrapper.
ResultOptional detailsDisplayed breadcrumb, date label, and sitelinks, where separately extracted.

The breadcrumb is a display label. A shortened path cannot supply the full destination address, so use the headline’s actual link. Dates deserve a similar check: record the label you found in the search result, and leave publication date unknown until its meaning has been verified.

Keep organic position local to the captured page. If the page contains six organic listings, save positions 1 through 6. Do not fill the missing four positions merely because you requested ten results. For a combined export, a collection sequence can number the observations you accepted, but label it accordingly.

How Do You Extract Organic Listings With Python?

Extract each row from a single organic listing. In the saved response used for this article, the first listing was “Welcome to Python.org.” Its headline, link, and snippet belonged to the same li.b_algo container. Keeping that container as the boundary for extraction ensures the exported description stays with its headline.

The parser finds the listings with #b_results > li.b_algo. It then reads h2 a and .b_caption p inside each one. These are the locations found in the saved pages. If a later export loses its snippets, look for the descriptions in a retained response before replacing the selector.

Install Requests and Beautiful Soup for the example.

bash
python -m pip install requests beautifulsoup4

The Beautiful Soup documentation is the reference for adjusting these selection rules. After reading the listings, the function checks for a Next control and returns its link with the rows.

python
import base64
import binascii
from urllib.parse import parse_qs, urljoin, urlsplit

from bs4 import BeautifulSoup


def destination_url(href):
    parts = urlsplit(href)
    host = (parts.hostname or "").lower()
    is_bing = host == "bing.com" or host.endswith(".bing.com")
    if is_bing and parts.path.rstrip("/") == "/ck/a":
        token = parse_qs(parts.query).get("u", [""])[0]
        if not token.startswith("a1"):
            return None
        try:
            value = token[2:]
            value += "=" * (-len(value) % 4)
            href = base64.urlsafe_b64decode(value).decode("utf-8")
        except (ValueError, UnicodeError, binascii.Error):
            return None
        parts = urlsplit(href)
    if parts.scheme not in {"http", "https"} or not parts.hostname:
        return None
    host = (parts.hostname or "").lower()
    if host == "bing.com" or host.endswith(".bing.com"):
        return None
    return href


def parse_page(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    rows = []
    for position, block in enumerate(
        soup.select("#b_results > li.b_algo"), start=1
    ):
        link = block.select_one("h2 a[href]")
        if link is None or not link.get_text(" ", strip=True):
            raise ValueError("Organic container has no usable headline")
        href = urljoin(page_url, link["href"])
        snippet = block.select_one(".b_caption p, p.b_algoSlug")
        rows.append({
            "organic_position": position,
            "title": link.get_text(" ", strip=True),
            "result_href": href,
            "destination_url": destination_url(href),
            "snippet": snippet.get_text(" ", strip=True) if snippet else None,
        })
    next_link = soup.select_one("li.b_pag a.sb_pagN[href]")
    next_url = urljoin(page_url, next_link["href"]) if next_link else None
    return rows, next_url

The Python result needed one more step before its address could be compared with another result. Its headline used Bing’s /ck/a tracking link. Inside that link, the u value encoded the destination with an a1 prefix and URL-safe Base64. Removing the prefix and decoding the value yielded the Python home-page address. No request to Python’s website was needed.

The saved record keeps both addresses: the original Bing link under result_href and the destination under destination_url. An unrecognized format produces no destination, so the loop stops before accepting that page. The example also leaves Bing-hosted destinations unresolved. Keep any later download of an extracted address subject to the downloader’s own checks.

A listing can have no snippet and still enter the export. Without a usable headline, the parser raises an error and the collector adds none of that page’s rows.

Before changing the request client, check whether the missing field appears in the rendered page at all. Use the Python Web Scraping guide to choose an extraction approach for the content you find. Rendering is relevant when it supplies that missing content.

How Do You Check That Bing Pagination Is Advancing?

Check the addresses after following Next. A page that contains only destinations already seen has added nothing to a URL discovery run, even if its request URL carries a different first value.

The retained HTTP responses show why this matters. On November 5, 2026, the query python web scraping returned ten organic destinations at first=1. Requests for first=11 and first=21 returned those same destinations, in the same order. All three requests shared one cookie-preserving session and used no proxy or browser rendering. That is one observed collection outcome, not a ten-result limit established for Bing as a whole.

Build the repeat check from extracted destinations. A full-HTML comparison can be distracted by changing tokens, and tracking URLs can change while still pointing to the same website page.

The collection loop belongs after the parser. It allows up to three page attempts and keeps the cookies between requests. Each accepted page goes into the JSON report. A response with no new destination ends the loop, and the report records why it stopped.

python
import json
import os
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlencode

import requests

query = "python web scraping"
settings = {"q": query, "setlang": "en", "cc": "us"}
url = "https://www.bing.com/search?" + urlencode(settings)
report = {"query": query, "requested_settings": settings, "pages": []}
seen = set()
stop_reason = "page_limit"

with requests.Session() as session:
    session.trust_env = False
    session.headers.update({
        "User-Agent": "BingCollectionExample/1.0",
        "Accept-Language": "en-US,en;q=0.9",
    })
    proxy = os.environ.get("BING_PROXY_URL")
    if proxy:
        session.proxies.update({"http": proxy, "https": proxy})
    for attempt in range(1, 4):
        requested_url = url
        try:
            response = session.get(url, timeout=(10, 30))
            response.raise_for_status()
            rows, next_url = parse_page(response.text, response.url)
        except (requests.RequestException, ValueError):
            stop_reason = "request_or_parse_error"
            break
        if not rows:
            stop_reason = "no_organic_rows_unverified"
            break
        if any(row["destination_url"] is None for row in rows):
            stop_reason = "unresolved_destination"
            break
        keys = {row["destination_url"].split("#", 1)[0] for row in rows}
        if not keys - seen:
            stop_reason = "no_new_destinations"
            break
        seen.update(keys)
        report["pages"].append({
            "page_attempt": attempt,
            "requested_url": requested_url,
            "response_url": response.url,
            "retrieved_at": datetime.now(timezone.utc).isoformat(),
            "results": rows,
        })
        if not next_url:
            stop_reason = "no_next_link_observed"
            break
        target = urlsplit(next_url)
        if target.scheme != "https" or target.hostname != "www.bing.com":
            stop_reason = "unexpected_next_host"
            break
        url = next_url
        if attempt < 3:
            time.sleep(3)

report["stop_reason"] = stop_reason
report["unique_destinations"] = len(seen)
Path("bing-results.json").write_text(
    json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8"
)

The code in this article is provided for illustration. It may behave differently or fail to run depending on your environment settings, software and library versions, or changes to the services and APIs it uses. Before using the examples, test them in your own environment and adapt them as needed.

The requested cc and setlang values are recorded as inputs, not proof of the country or language Bing used. For an actual collection, inspect the returned context and keep the settings consistent across the searches being compared.

This conservative example stops if any destination is unresolved. It also groups request and parsing errors into one stopping category. A scheduled collector should log the specific response status or parser error, retrieval timing, and a permitted input reference separately, with credentials removed. Keep those failed attempts outside the result rows.

Three seconds is simply the delay chosen for this example, not a Bing request quota. A denied or rate-limited request ends the loop. Before scheduling a further attempt, check the response and any waiting instructions it provides.

What Can Proxies Change in a Bing Collection?

A proxy changes the network exit used by the configured client. For a regional comparison, use that exit as one of the settings for the run.

Bing also uses information from the client and session. Its explanation of how Bing delivers search results describes the roles of language, location, search history, and device information. Record the requested market alongside the session and client settings. A US proxy exit alone does not establish that every returned listing belongs to a uniform US search context.

Keep the same intended exit policy and cookie state while following one query’s pages. Rotate between independent jobs when the study calls for independent routes. For example, if page two uses another country and a fresh cookie state, its different results cannot be attributed to pagination alone.

There is one optional proxy input in the example: BING_PROXY_URL. Copy the provider’s complete connection URL into that variable in the environment used to start Python. It carries the gateway, port, and authentication details. Encode reserved characters within the credentials: for instance, an @ inside a password becomes %40, so it cannot be confused with the separator before the hostname.

Use the https key for the request to Bing even when the proxy URL starts with http://. The key refers to the destination request; the URL prefix refers to the connection with the proxy gateway. The Requests proxy documentation gives an example of this mapping.

When BING_PROXY_URL has no value, this code adds nothing to the proxy dictionary. Setting trust_env to False prevents it from picking up inherited environment settings instead. Browser proxy settings are not a substitute for configuring this Python client. SOCKS support requires the appropriate Requests dependency and a matching proxy scheme.

Verify the exit through an approved IP-echo endpoint using the same configured Python session. A browser IP-check page describes that browser’s request. With rotating proxies, even a check in the same session describes only that request unless the provider’s allocation settings retain the exit. Neither result establishes that Bing accepts the connection.

When comparing residential and datacenter options, check which provides the geography and exit policy your run requires. The proxy infrastructure for web scraping guide explains the pool and routing choices in more detail.

Practical Example: Auditing a Misleading Export

Here is an invented run that might be reported as twenty-four results if every extracted row were simply appended to a file.

The first request returns eight organic listings. The second also returns eight, but five destinations have already appeared on the first page. It adds three destinations to the discovery list.

The third request produces another eight listings, all pointing to destinations seen earlier. At that point the collector stops. This example illustrates the counting rules; it is separate from the live HTTP check described above.

There are sixteen accepted result appearances and eleven unique destinations. The last response contributes no accepted page. Keep its failed-progress outcome in the run log instead of labelling its rows as ranks 21 through 28.

The five repeated destinations on the second accepted page still describe appearances on that page. Preserve their page references and local organic positions for an audit. A separate discovery table can contain just the eleven unique URLs. This avoids forcing ranking observations and a deduplicated URL list into one count.

Three page attempts with 8, 3 and 0 new destinations yield 16 accepted appearances and 11 unique destinations; collection stops on page 3.

If you need absolute ranks, establish that the source supplies the requested interval and that its position convention matches your report. Adding ten to the second page’s local positions is insufficient evidence when pages overlap or return fewer results than expected.

For CSV export, flatten each accepted page into result rows with the query, capture reference, time, and local position repeated beside each result. Choose a documented CSV convention for missing values, such as an empty snippet cell, when converting JSON nulls. Sitelinks need their own rows linked to the parent result if you collect them. They do not increase the count of organic listings.

Match the Connection to Your Search Study

Write down which searches need a shared exit, which need independent routes, and which geographic settings must be held constant. Then select connection settings against that specification.

For a run that needs country targeting and residential exits, consider MangoProxy Residential proxies. The product offers geographic selection and rotation. Start with a short query list and inspect the pages returned through your chosen settings. This article’s HTTP check used no proxy, so it supplies no test result for MangoProxy on Bing.

Glossary

SERP (search engine results page): the page being parsed. Organic position: a listing’s order within its organic results.

HTML (HyperText Markup Language): the document structure. CSS (Cascading Style Sheets) selectors: rules used here to find elements in that structure.

HTTP (Hypertext Transfer Protocol): the protocol for web requests. HTTPS: its secure form, which encrypts the connection.

API (application programming interface): a software interface. JSON (JavaScript Object Notation): the report format used here. CSV (comma-separated values): a format for tabular text.

IP address (Internet Protocol address): a network address. The exit IP is the source address seen for the routed connection. Cookies retain client state between eligible requests.

UTC, Coordinated Universal Time, is the reference used for capture times. Base64 encodes bytes as text and appears in the inspected tracking links. SOCKS is an alternative proxy protocol supported through an optional Requests dependency.

Final Thoughts

A useful Bing export lets another reader trace a row to its captured search page and understand its position. Retain the context and stopping reason alongside the results, so limited retrieval cannot quietly become a claim of complete coverage.

Frequently asked questions

Here we answered the most frequently asked questions.

Ask a question

Can Grounding with Bing Search replace a search-results export?

Not as an interchangeable source of raw result rows. Microsoft’s current Grounding with Bing Search documentation describes search used to ground an agent’s response and states that developers do not receive the raw returned content. Check the available output against your dataset requirements rather than converting answer citations into assumed organic ranks.

Should I store the snippet as the destination page's description?

For research into what a searcher saw, use the snippet saved from Bing and keep its query and capture time. For the destination page’s metadata, make a separate permitted retrieval and date that observation. Store its description alongside the saved search snippet, preserving the wording from each source.

How should I report a URL that the run did not find?

Report it as unobserved in that run. The export covers the pages and context successfully collected. A URL outside that sample, or beyond a failed pagination step, has no confirmed position in that run. Investigating index status requires separate evidence.

Leave Comment

Your email address will not be published. Required fields are marked *