How to Scrape Web Data & Map Spatial Coordinates using Python

1. Introduction: From Web Tables to GIS Maps

Learning How to Scrape Web Data Map Spatial Coordinates Python allows you to tap into valuable online geographic information, including lists of schools, hospitals, and administrative locations. . Much of this information is published as HTML tables or ordinary text on websites rather than as ready-to-use GIS datasets.

For a spatial analyst, the challenge is turning this unstructured or semi-structured information into a dataset that can be mapped and analyzed.

Consider a website listing public facilities with their names, addresses, and categories. The table may look like this:

Facility Name	Address	Category
Central Clinic	Example Street, Nairobi	Healthcare
Green Valley School	Example Road, Nairobi	Education
Riverside Market	Example Avenue, Nairobi	Market

Although the table contains useful information, it cannot immediately support point mapping because the addresses have not yet been converted into geographic coordinates.

Python provides a practical solution. By combining web scraping, data cleaning, geocoding, and spatial data processing, you can transform an ordinary webpage into a vector dataset that opens in QGIS or ArcGIS Pro.

This tutorial uses five main libraries:

• Requests: Retrieves webpage content over HTTP.
• BeautifulSoup: Extracts information from HTML documents.
• pandas: Cleans and organizes tabular data.
• geopy: Connects Python to geocoding services such as Nominatim.
• GeoPandas: Converts coordinates into spatial geometries and exports GIS files.

The workflow is:

Webpage containing location data
              |
              v
       Requests + HTML
              |
              v
          BeautifulSoup
              |
              v
       pandas DataFrame
              |
              v
     Clean address columns
              |
              v
    geopy address geocoding
              |
              v
       Latitude + longitude
              |
              v
       GeoPandas GeoDataFrame
              |
              v
      GeoJSON / Shapefile
              |
              v
        QGIS / ArcGIS Pro

The objective is to build a reproducible Python web scraping GIS workflow that you can adapt to university assignments, environmental studies, and spatial data science projects.

2. Tools and Libraries Needed

2.1 Install the required libraries

If you are using a Python environment with pip, install the libraries with:

pip install requests beautifulsoup4 geopy pandas geopandas

For beginners using Conda, GeoPandas’ official installation guide also recommends the conda-forge ecosystem for managing its geospatial dependencies. See the GeoPandas installation guide.

2.2 Understand each library’s role

Library	Main function	Application in this tutorial
requests	HTTP requests	Downloads webpage HTML
BeautifulSoup	HTML parsing	Extracts table rows and cells
pandas	Tabular data manipulation	Cleans text and manages records
geopy	Geocoding interface	Converts addresses into coordinates
geopandas	Spatial data processing	Creates geometries and exports GIS files

You do not need to install QGIS or ArcGIS Pro to execute the Python workflow. However, either application is useful for inspecting the resulting spatial data.

3. Step 1: Extract Address or Location Data from a Webpage

The first task is retrieving the HTML content of a webpage and locating the table containing the geographic information.

3.1 Understand the HTML structure

A typical HTML table contains rows (tr), header cells (th), and data cells (td).

For example:

<table class="location-table">
    <tr>
        <th>Facility Name</th>
        <th>Street Address</th>
        <th>Category</th>
    </tr>
    <tr>
        <td>Central Clinic</td>
        <td>Example Street, Nairobi</td>
        <td>Healthcare</td>
    </tr>
</table>

BeautifulSoup can locate the table and extract its contents.

Real webpages have different HTML structures. The code below assumes the target website contains a table with the class location-table. You must replace the example URL and table selector with values that match a website you are permitted to access.

3.2 Retrieve the webpage

import requests
from bs4 import BeautifulSoup

# Replace with a webpage you are authorized to scrape
URL = "https://your-permitted-site.example/locations"

# Identify your application in the request
headers = {
    "User-Agent": (
        "StudentGISProject/1.0 "
        "(contact: your-email@example.com)"
    )
}

# Download the HTML with a timeout
response = requests.get(
    URL,
    headers=headers,
    timeout=20
)

# Stop if the server returned an unsuccessful HTTP status
response.raise_for_status()

# Parse the HTML document
soup = BeautifulSoup(response.text, "html.parser")

print("Page downloaded successfully.")
print("Page title:", soup.title.get_text(strip=True)
      if soup.title else "No title found")

How the code works

requests.get() sends an HTTP request to the webpage. The timeout parameter helps prevent your script from waiting indefinitely for a response.

response.raise_for_status() raises an error when the server returns an unsuccessful HTTP status. This is important because receiving an HTML response does not necessarily mean the request succeeded. See the Requests Quickstart.

The BeautifulSoup() constructor then parses the response using Python’s HTML parser.

The custom User-Agent identifies your application. Replace the example contact address with a suitable contact for your project.

3.3 Extract the table rows

Now that the HTML is available, locate the target table and convert its rows into Python dictionaries.

import pandas as pd

# Find the table using its CSS class
table = soup.find("table", class_="location-table")

if table is None:
    raise ValueError(
        "Location table not found. "
        "Check the webpage URL and HTML selector."
    )

# Retrieve the table rows
rows = table.find_all("tr")

if not rows:
    raise ValueError("The table contains no rows.")

# Extract the column headings from the first row
headers = [
    cell.get_text(" ", strip=True)
    for cell in rows[0].find_all(["th", "td"])
]

records = []

# Process each remaining row
for row in rows[1:]:
    cells = [
        cell.get_text(" ", strip=True)
        for cell in row.find_all(["td", "th"])
    ]

    # Skip rows whose structure does not match the headers
    if len(cells) != len(headers):
        continue

    records.append(dict(zip(headers, cells)))

# Convert the extracted records into a DataFrame
df = pd.DataFrame(records)

print(df.head())
print("Records extracted:", len(df))

BeautifulSoup’s find_all() method retrieves matching HTML elements, while get_text() extracts the readable text from those elements. More examples are available in the BeautifulSoup documentation.

The final DataFrame should contain one row per extracted facility and columns corresponding to the table headings.

Common beginner error: A webpage may display a table in your browser but return no table in the HTML downloaded by Requests. This often happens when JavaScript loads the table dynamically. In that situation, check whether the website offers an official API, CSV download, or public data service. Do not attempt to bypass access controls or anti-bot restrictions.

4. Step 2: Clean and Format Text Columns in pandas

Scraped data is rarely ready for spatial analysis immediately. It may contain extra spaces, inconsistent capitalization, missing addresses, duplicate records, or inconsistent column headings.

Cleaning the data before geocoding improves consistency and can reduce the number of unnecessary requests to the geocoding service.

4.1 Standardize the column names

The following example assumes the table includes columns named Facility Name, Street Address, and Category.

# Remove surrounding spaces from column headings
df.columns = df.columns.str.strip()

# Standardize the column names
df = df.rename(columns={
    "Facility Name": "name",
    "Street Address": "address",
    "Category": "category"
})

# Verify that required columns exist
required_columns = {"name", "address"}

missing_columns = required_columns - set(df.columns)

if missing_columns:
    raise ValueError(
        f"Required columns are missing: {missing_columns}"
    )

print(df.columns.tolist())

Renaming columns makes subsequent code easier to understand. If your source uses different headings, update the mapping accordingly.

4.2 Clean the text

# Remove extra whitespace and normalize repeated spaces
for column in ["name", "address"]:
    df[column] = (
        df[column]
        .astype("string")
        .str.replace(r"\s+", " ", regex=True)
        .str.strip()
    )

# Remove records with missing or empty addresses
df = df.dropna(subset=["address"]).copy()

df = df[df["address"].ne("")].copy()

# Remove duplicate records with the same name and address
df = df.drop_duplicates(
    subset=["name", "address"]
).reset_index(drop=True)

print(df.head())
print("Clean records:", len(df))

The .str.replace() expression replaces repeated whitespace with a single space. .str.strip() removes spaces at the beginning and end of each value.

dropna() removes rows whose address is missing, while drop_duplicates() removes repeated facility records according to the specified fields.

Do not remove duplicates based only on address if several legitimate facilities may occupy the same building.

4.3 Prepare the geocoding query

Addresses are often ambiguous. For example, a street name may occur in several towns or countries.

Adding geographic context can improve the chance of finding the intended location.

# Replace this with the appropriate country or region
COUNTRY_OR_REGION = "Kenya"

df["query"] = (
    df["address"] + ", " + COUNTRY_OR_REGION
)

print(df[["name", "address", "query"]].head())

This creates a query such as:

Example Street, Nairobi, Kenya

The geographic context should match your study area. If your addresses already include the full city and country, adjust this step to avoid unnecessary repetition.

5. Step 3: Geocode Addresses to Latitude and Longitude

Geocoding converts a place name or address into geographic coordinates. These coordinates usually comprise latitude and longitude, which can be used to create point features.

For this tutorial, we will use geopy with the Nominatim geocoding service, which uses OpenStreetMap data.

The geopy library is an interface to geocoding services rather than a geocoding database itself. Each service has separate terms, quotas, and restrictions. See the geopy documentation.

5.1 Create a geocoder

from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter

# Identify your application with a meaningful User-Agent
geolocator = Nominatim(
    user_agent=(
        "student_spatial_research/1.0 "
        "(contact: your-email@example.com)"
    ),
    timeout=10
)

# Limit requests and allow controlled retries
geocode = RateLimiter(
    geolocator.geocode,
    min_delay_seconds=1.1,
    max_retries=2,
    error_wait_seconds=5,
    swallow_exceptions=False
)

The Nominatim() object configures the geocoding service. The RateLimiter adds a delay between calls and allows a limited number of retries.

The public Nominatim service requires an identifying User-Agent and sets a maximum of one request per second. It also discourages large-scale and recurring bulk geocoding. The code uses a slightly longer delay to remain below that request rate. Always check the official Nominatim usage policy before using the service.

Important: Public Nominatim is suitable only for small, carefully limited examples under its policy. If your thesis involves hundreds or thousands of addresses, recurring geocoding, or regular bulk updates, use a provider that explicitly permits your workload or set up an appropriate service of your own.

5.2 Geocode unique addresses and cache the results

A common beginner mistake is geocoding every row repeatedly. Identical addresses create unnecessary requests, and restarting the script can repeat previously completed work.

A local cache allows successful results and genuine no-match results to be reused. This example stores them in geocode_cache.csv.

from pathlib import Path
import pandas as pd
from geopy.exc import GeocoderServiceError

# Local cache file
cache_path = Path("geocode_cache.csv")

cache_columns = [
    "query",
    "latitude",
    "longitude",
    "matched_address"
]

# Load previously saved results, if available
if cache_path.exists():
    cache = pd.read_csv(cache_path)
else:
    cache = pd.DataFrame(columns=cache_columns)

# Record queries already in the cache
known_queries = set(
    cache["query"].dropna().astype(str)
)

new_results = []

# Geocode each unique query only once
for query in df["query"].dropna().drop_duplicates():

    if query in known_queries:
        continue

    try:
        location = geocode(query)

        if location is None:
            print("No result found:", query)

            new_results.append({
                "query": query,
                "latitude": None,
                "longitude": None,
                "matched_address": None
            })

        else:
            new_results.append({
                "query": query,
                "latitude": location.latitude,
                "longitude": location.longitude,
                "matched_address": location.address
            })

            print(
                f"Geocoded: {query} -> "
                f"{location.latitude}, {location.longitude}"
            )

    except GeocoderServiceError as error:
        # Do not cache temporary service errors as no-match results
        print(f"Service error for {query}: {error}")

# Append new results to the existing cache
if new_results:
    cache = pd.concat(
        [cache, pd.DataFrame(new_results)],
        ignore_index=True
    )

    cache = cache.drop_duplicates(
        subset=["query"],
        keep="last"
    )

    cache.to_csv(cache_path, index=False)

# Join cached coordinates back to every source row
df = df.merge(
    cache[cache_columns],
    on="query",
    how="left",
    validate="many_to_one"
)

print(
    df[[
        "name",
        "query",
        "latitude",
        "longitude"
    ]].head()
)

5.3 Understand the result

A successful geocoding operation provides fields similar to these:

name	latitude	longitude
Central Clinic	-1.28	36.82
Green Valley School	-1.30	36.84

These coordinates are illustrative and are not verified locations for the example facilities.

The latitude and longitude columns now contain the numeric values needed to create spatial point features. The matched_address column stores the description returned by the geocoder so you can review questionable matches.

Notice that not every input address is guaranteed to produce a result. Some may be incomplete, ambiguous, misspelled, or absent from the geocoder’s underlying data.

5.4 Common geocoding errors

Error: Too Many Requests (HTTP 429)

You may be sending queries too quickly or exceeding a service quota. Reduce the request rate, cache completed results, and respect the provider’s policy. Do not respond by rotating identities or distributing requests across machines to bypass limits.

Error: No location found

Check spelling, add a town or region, and inspect the original address. Review the returned match instead of assuming every result is correct.

Error: Timeout or service unavailable

The geocoding server may be temporarily unavailable. The script reports these service failures without saving them as permanent no-match results. Retry later within the service’s permitted limits.

Error: Repeated duplicate results

Identical addresses should not require a separate geocoding request for each row. The unique-query loop and local cache reduce repeat requests.

Before mapping a dataset, manually inspect a sample of the geocoded coordinates using a reference map or reliable address information. A technically valid coordinate can still point to the wrong building, street, or town.

6. Step 4: Convert the DataFrame into a GeoDataFrame

At this stage, your cleaned DataFrame contains attributes, addresses, and geographic coordinates. GeoPandas can convert these records into actual spatial features.

A regular pandas DataFrame stores tabular information. A GeoDataFrame adds an active geometry column and coordinate reference system (CRS).

6.1 Create the point geometries

import geopandas as gpd

# Keep only rows with valid coordinate values
geocoded = df.dropna(
    subset=["latitude", "longitude"]
).copy()

# Convert coordinates to numeric values
geocoded["latitude"] = pd.to_numeric(
    geocoded["latitude"],
    errors="coerce"
)

geocoded["longitude"] = pd.to_numeric(
    geocoded["longitude"],
    errors="coerce"
)

# Remove any invalid or missing coordinates
geocoded = geocoded.dropna(
    subset=["latitude", "longitude"]
).copy()

# Check that coordinates fall within valid global ranges
valid_coordinates = (
    geocoded["latitude"].between(-90, 90)
    & geocoded["longitude"].between(-180, 180)
)

geocoded = geocoded[valid_coordinates].copy()

# Build the GeoDataFrame
gdf = gpd.GeoDataFrame(
    geocoded,
    geometry=gpd.points_from_xy(
        geocoded["longitude"],
        geocoded["latitude"]
    ),
    crs="EPSG:4326"
)

print(gdf.head())
print("Coordinate reference system:", gdf.crs)
print("Mapped features:", len(gdf))

Why does this work?

The dropna() operation removes records without coordinates, while pd.to_numeric() makes sure the coordinate fields can be interpreted as numbers.

The range checks identify obvious invalid latitude or longitude values.

The critical function is:

gpd.points_from_xy(
    geocoded["longitude"],
    geocoded["latitude"]
)

The x-coordinate must be longitude and the y-coordinate must be latitude. Reversing these fields is a common GIS programming error.

The crs="EPSG:4326" parameter identifies the coordinate system used by ordinary latitude-longitude geocoding results, provided those results are returned in WGS 84 coordinates. It does not transform coordinates; it records their reference system.

GeoPandas documents these spatial data structures and CRS operations in its user guide.

6.2 Inspect the mapped locations

You can produce a basic exploratory plot without opening a separate GIS application:

gdf.plot(
    markersize=20,
    figsize=(8, 6)
)

This creates a point plot using the geometries in the GeoDataFrame. It is a simple spatial visualization, not a full basemap or cartographic layout.

For distance calculations, buffering, and area analysis, reproject the data to an appropriate local projected CRS with suitable units first. EPSG:4326 uses angular coordinates, so its coordinate units are not metres.

7. Step 5: Export to GeoJSON or Shapefile

Once the GeoDataFrame is ready, you can save it as a vector dataset and open it in QGIS or ArcGIS Pro.

GeoPandas supports formats such as GeoJSON, ESRI Shapefile, and GeoPackage through its file-writing functionality. See the official GeoPandas reading and writing guide.

7.1 Export to GeoJSON

GeoJSON is a convenient format for exchanging geographic data, especially when you want the output to be human-readable or use it in a web mapping workflow.

from pathlib import Path

# Create an output directory
output_dir = Path("outputs")
output_dir.mkdir(parents=True, exist_ok=True)

# Export the GeoDataFrame
geojson_path = output_dir / "scraped_locations.geojson"

gdf.to_file(
    geojson_path,
    driver="GeoJSON"
)

print("GeoJSON saved to:", geojson_path)

This writes the geometries and associated attributes into scraped_locations.geojson.

Because the GeoDataFrame uses EPSG:4326, its coordinates are in longitude-latitude degrees, which matches the standard coordinate system expected by RFC 7946 GeoJSON.

7.2 Export to Shapefile

Shapefile remains widely supported by desktop GIS applications, including QGIS and ArcGIS Pro.

# Export the same features as an ESRI Shapefile
shapefile_path = output_dir / "scraped_locations.shp"

gdf.to_file(
    shapefile_path,
    driver="ESRI Shapefile",
    encoding="UTF-8"
)

print("Shapefile saved to:", shapefile_path)

A Shapefile is not a single file. It normally consists of several related files, including .shp, .shx, and .dbf, with a .prj commonly providing projection information. Keep these files together.

Shapefile also has limitations, including short field-name constraints and less flexible attribute types. GeoJSON or GeoPackage may be preferable when preserving descriptive field names and attribute information is important.

7.3 Open the exported data in QGIS or ArcGIS Pro

In QGIS, use Layer → Add Layer → Add Vector Layer, then browse to the GeoJSON or Shapefile.

In ArcGIS Pro, add the exported file as a data layer using the map’s data-addition tools.

Once loaded, inspect the attribute table, verify the coordinate system, and compare a sample of locations with trusted reference information.

For a research project, this validation step is essential. The process of scraping and geocoding can introduce errors even when your Python code completes successfully.

8. Legal and Ethical Considerations

Automated collection of webpage content and geocoding of addresses should be carried out responsibly. Technical access does not automatically imply permission to collect, reuse, or redistribute data.

Respect robots.txt and website terms

Before scraping a website, inspect its robots.txt file, terms of service, licensing conditions, and any published API documentation.

The Robots Exclusion Protocol describes rules that website operators use to indicate which paths automated crawlers are requested to access or avoid. However, robots.txt is not an authorization mechanism and does not override a site’s terms or applicable law. See RFC 9309.

Prefer official open-data portals, published APIs, and downloadable datasets whenever they meet your needs.

Respect rate limits

Do not send rapid, repeated requests to a website or geocoding service. Implement suitable timeouts, rate limiting, and caching.

For the public OpenStreetMap Nominatim service, read the official usage policy before making requests. Its restrictions include a maximum of one request per second, an identifying User-Agent, attribution, and limitations on bulk geocoding.

Protect privacy

Avoid collecting sensitive personal information unnecessarily. Do not send confidential information or personal data to a geocoding provider unless you have a lawful basis and the provider’s terms permit that use.

Preserve attribution and licensing information

Record the original source URL, retrieval date, applicable data licence, and geocoding provider. If a map uses OpenStreetMap-derived data, include the attribution required by the provider’s terms.

For a university thesis, describing these details in your data collection methodology improves transparency and reproducibility.

9. Conclusion: From Web Scraping to Spatial Data Science

The workflow in this tutorial demonstrates how Python can transform ordinary web content into a GIS-ready dataset.

Requests retrieves the webpage, BeautifulSoup extracts the records, pandas cleans and organizes the text, geopy geocodes the addresses, and GeoPandas converts the resulting coordinates into spatial features.

Those features can then be exported as GeoJSON or Shapefile and inspected in QGIS or ArcGIS Pro.

The most important lesson is that web scraping alone does not produce a reliable spatial dataset. You must also validate addresses, manage coordinate reference systems, handle unmatched geocoding results, respect service restrictions, and verify the geographic accuracy of the final features.

Suggested student project

Choose a permitted public webpage containing a modest table of facilities in a defined study area. Extract the records, geocode a small sample, and export the verified locations to GeoJSON. Open the result in QGIS and create a map showing the facility distribution.

Document the source, processing steps, geocoding method, coordinate reference system, limitations, and validation results.

This exercise gives you a practical introduction to web scraping, Python data cleaning, geocoding, and GIS automation—and a reproducible workflow you can adapt for future spatial analysis projects.