Bulk WHOIS lookup: tools, scripts, and API methods

Querying WHOIS or RDAP for a list of domains is not trivial. The WHOIS protocol imposes strict rate limits per IP, exceed them and your address gets blocked for an hour or more. The server varies by TLD, so a list of 100 domains across different extensions can require querying 20 different servers. And the plain-text output is difficult to parse reliably. RDAP solves the last two problems but still has its own rate limits. This article covers three approaches in order of complexity: a web tool, a bash script with rate limiting, and a Python RDAP script. The right choice depends on your volume and whether you need the results once or on an ongoing basis.

The rate limiting problem

Before reaching for any solution, understand why a naive loop does not work:

• Most WHOIS servers limit to 1-5 requests per second per IP. Exceeding this triggers a temporary block, typically 1 hour, sometimes 24 hours.
• There is no standard for how a blocked client should behave. Some servers return an error message; others just stop responding.
• RDAP handles rate limiting more cleanly: servers return HTTP 429 with a Retry-After header specifying exactly how long to wait. But the limit is still real.
• A list mixing .com, .net, .io, .fr, and .de domains requires querying five different WHOIS servers, each with its own limits.

The practical rule: stay at 1 request per 2 seconds for WHOIS, and respect every HTTP 429 response you receive from RDAP servers.

Rate limit hit? RDAP servers must return HTTP 429 with a Retry-After header. Respect it, or your IP gets blocked.

Method 1: Web tools for bulk lookup

For non-developers who want to upload a list and get results without writing any code:

Tool	Free tier limit	Output format	Notes
WhoisXMLAPI	500 credits/month	JSON, CSV	Good API coverage
ViewDNS.info	~100/day	HTML, CSV	Basic, no API
DomainTools	Minimal free	CSV, JSON	Most complete, expensive
Who.is	Very limited	HTML only	Manual only

The limitation of all these tools: they cap bulk queries at reasonable volumes for free, and pricing escalates quickly beyond a few hundred domains per month. They are workable for occasional, moderate-volume needs. For anything recurring or above 500 domains, a script-based approach is more cost-effective.

Method 2: Bash script with rate limiting

For Linux and macOS users who want a simple, dependency-free solution. Save this as bulk-whois.sh:

#!/bin/bash
# bulk-whois.sh: query WHOIS for a list of domains
# Usage: ./bulk-whois.sh domains.txt

DELAY=2  # seconds between requests, stay under rate limits

INPUT="$1"
if [[ -z "$INPUT" ]]; then
  echo "Usage: $0 <domains-file>"
  exit 1
fi

while IFS= read -r domain; do
  # skip empty lines and comments
  [[ -z "$domain" || "$domain" == \#* ]] && continue
  echo "=== $domain ==="
  whois "$domain" 2>/dev/null \
    | grep -E "Registrar:|Expir|Name Server|Domain Status"
  sleep "$DELAY"
done < "$INPUT"

The grep -E filter pulls out only the useful lines, without it, raw WHOIS output is 50-100 lines per domain. Create a domains.txt file with one domain per line:

github.com
stripe.com
# this line is ignored
vercel.com
netlify.com

Extracting to CSV

For integration with spreadsheets or other tools, here is a version that writes a CSV:

#!/bin/bash
OUTPUT="output.csv"
echo "domain,registrar,expiry,status" > "$OUTPUT"

while IFS= read -r domain; do
  [[ -z "$domain" || "$domain" == \#* ]] && continue
  
  raw=$(whois "$domain" 2>/dev/null)
  registrar=$(echo "$raw" | grep -i "^Registrar:" \
    | head -1 | cut -d: -f2- | xargs)
  expiry=$(echo "$raw" | grep -iE "Expiry|Expiration Date" \
    | head -1 | cut -d: -f2- | xargs)
  status=$(echo "$raw" | grep -i "^Domain Status:" \
    | head -1 | cut -d: -f2- | xargs)
  
  echo "$domain,$registrar,$expiry,$status" | tee -a "$OUTPUT"
  sleep 2
done < domains.txt

One important caveat: grep field names differ between registries. Expiry Date works for Verisign; Expiration Date is what you get from some registrars. The grep -iE "Expiry|Expiration Date" pattern covers both but may miss others. This is the fundamental fragility of WHOIS text parsing, it is why RDAP exists.

Method 3: RDAP queries via script

The recommended approach for developers. RDAP gives structured JSON, so parsing is reliable and the field names are consistent regardless of which registry serves the response.

Finding the right RDAP server per TLD

The IANA bootstrap file maps TLDs to RDAP server URLs. Fetch it once; it changes infrequently:

import json
import urllib.request
import time

# Load IANA bootstrap once at startup
with urllib.request.urlopen("https://data.iana.org/rdap/dns.json") as r:
    bootstrap = json.load(r)

def get_rdap_server(tld: str) -> str | None:
    for entry in bootstrap["services"]:
        if tld.lower() in [t.lower() for t in entry[0]]:
            return entry[1][0]
    return None

def rdap_lookup(domain: str) -> dict:
    tld = domain.split(".")[-1]
    server = get_rdap_server(tld)
    if not server:
        return {"error": f"No RDAP server found for .{tld}"}
    
    url = f"{server.rstrip('/')}/domain/{domain}"
    try:
        with urllib.request.urlopen(url, timeout=10) as r:
            return json.load(r)
    except urllib.error.HTTPError as e:
        return {"error": f"HTTP {e.code}", "retry_after": e.headers.get("Retry-After")}
    except Exception as e:
        return {"error": str(e)}

domains = ["github.com", "stripe.com", "vercel.com", "netlify.com"]

for domain in domains:
    data = rdap_lookup(domain)
    
    if "error" in data:
        print(f"{domain}: ERROR, {data['error']}")
        if data.get("retry_after"):
            wait = int(data["retry_after"])
            print(f"  Rate limited. Waiting {wait}s...")
            time.sleep(wait)
        continue
    
    # Extract expiry date from events array
    expiry = next(
        (e["eventDate"] for e in data.get("events", [])
         if e["eventAction"] == "expiration"),
        "N/A"
    )
    registrar = next(
        (e.get("vcardArray", [[]])[1] for e in data.get("entities", [])
         if "registrar" in e.get("roles", [])),
        "N/A"
    )
    status = ", ".join(data.get("status", []))
    
    print(f"{domain}: expires {expiry} | registrar: {registrar} | status: {status}")
    time.sleep(1)  # 1 req/sec, conservative for RDAP

This script is minimal. For production use, add a local cache of the bootstrap file (it only needs refreshing weekly), exponential backoff on retries, and logging. The script above prints to stdout; redirect to a file or pipe through jq as needed.

Using Domain Sentinel's API

For production workloads where you do not want to maintain RDAP bootstrapping, rate limiting logic, and WHOIS fallback yourself, Domain Sentinel exposes an API that handles all of this. A single authenticated request returns structured data regardless of whether the TLD is served by RDAP or WHOIS. See the Domain Sentinel API documentation for endpoint details and authentication.

Monitoring a domain portfolio continuously

There is an important distinction between bulk lookup and continuous monitoring:

Bulk lookup gives you a snapshot of current data at the moment of the query. Run it once, get the results, move on.

Continuous monitoring means checking each domain regularly and alerting you when something changes, expiry date approaching, nameservers changed, EPP status shifted.

Bulk lookup gives you a snapshot. Domain Sentinel gives you a continuous feed with alerts.

For a portfolio of 50 domains spread across multiple businesses or clients, the operational workflow is: import the list into Domain Sentinel, configure alert thresholds (60 days, 30 days, 7 days before expiry), and receive notifications per domain rather than running scripts manually. Each detected change is logged with a timestamp, so you also get an audit trail.

The decision comes down to volume and frequency: for 10-50 domains checked occasionally, a web tool or bash script is fine. For 50-500 domains checked regularly, the Python RDAP script is practical. For 500 or more domains, or any portfolio that needs continuous monitoring rather than one-off checks, a dedicated service is the right answer.