How to Build a SEC Form 4 Scraper with Python Requests

```html

Python's requests library can pull insider trading data directly from the SEC's EDGAR database. No authentication required. No API wrapper needed. Just HTTP requests, XML parsing, and a 10-requests-per-second rate limit you need to respect. This guide walks through CIK lookup to transaction extraction — the specific code you need to turn Form 4 filings into structured data.

Before You Start

This guide assumes you understand HTTP requests, can read XML structure, and have written basic Python scripts. You need to know what a CIK number is — the SEC's unique identifier for every public company. Form 4 filings report transactions by insiders: executives, board members, major shareholders. The SEC provides these as XML, which means you parse them, not scrape HTML tables.

A single Form 4 can contain multiple transactions. Your scraper needs to handle nested XML structures where one filing reports five different trades on the same day. If you only need occasional manual lookups, the SEC's web interface is faster than building automation.

What You Need

• Python 3.8 or later installed on your system
• requests library (install via pip install requests)
• xml.etree.ElementTree (included in Python standard library)
• A valid email address to include in your User-Agent header — SEC requirement, not optional
• Company CIK numbers for the stocks you want to track
• Basic understanding of XML element paths

Step 1: Look Up the Company CIK Number

Every public company has a Central Index Key. Navigate to the SEC EDGAR Company Search and enter the company name or ticker. The CIK appears as a 10-digit number. Microsoft: 0000789019. Apple: 0000320193. Write it down with leading zeros intact.

Ticker symbols don't work with EDGAR's backend. The system expects CIK numbers for programmatic access. No CIK, no data.

Step 2: Set Up Required Request Headers

The SEC requires all automated requests to identify who is making them. Create a Python dictionary:

headers = {'User-Agent': 'YourCompanyName your.email@example.com'}

Replace the placeholder with your actual organization and email. The SEC monitors this header. Generic values like "Python-requests" get blocked. Proper identification protects the public database from abuse while allowing legitimate access. Skip this step and your IP gets rate-limited within minutes.

Step 3: Construct the Form 4 Index Request URL

The SEC provides a browseable index of all filings for each company:

https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000789019&type=4&dateb=&owner=only&count=100

Parameters: CIK is the company identifier. type=4 filters for Form 4 only. owner=only shows insider transactions. count=100 returns up to 100 recent filings. This request returns HTML. You'll parse it to extract document URLs for individual Form 4 XML files.

Step 4: Fetch and Parse the Filing Index Page

Use requests to retrieve the index page, then parse the HTML for Form 4 document URLs:

response = requests.get(index_url, headers=headers)
html_content = response.text

Each Form 4 has a unique accession number: 0001234567-26-000123. Search the HTML for document links containing "/Archives/edgar/data/" followed by the CIK and accession number. These link to the HTML version. You need the XML version — replace the -index.html suffix with the actual XML filename, typically ending in .xml. XML contains structured transaction data. HTML tables do not.

Step 5: Download Individual Form 4 XML Files

For each accession number you extracted, construct the direct XML file URL:

xml_url = f"https://www.sec.gov/Archives/edgar/data/{cik}/{accession_no}/form4.xml"

The actual XML filename varies. Some filings use doc4.xml. Others use company-specific patterns. Check the index page's file listing to get the exact name. Download the XML:

xml_response = requests.get(xml_url, headers=headers)
xml_data = xml_response.content

Store the raw XML bytes in memory or save to disk. Each XML file represents one insider's transactions on a specific date. A company might have five Form 4 filings on the same day from different executives.

Step 6: Parse Transaction Data from XML

Use Python's xml.etree.ElementTree to parse the Form 4 structure. Transaction data lives under <nonDerivativeTable> or <derivativeTable>:

import xml.etree.ElementTree as ET
root = ET.fromstring(xml_data)
transactions = root.findall('.//nonDerivativeTransaction')

For each transaction element, extract: <transactionDate>, <transactionShares>, <transactionPricePerShare>, and <transactionAcquiredDisposedCode>. The last one shows A for acquired, D for disposed. These map to the columns you'd see in a spreadsheet of insider trades. Handle missing values — some filings omit price data for equity grants or conversions.

Step 7: Implement Rate Limiting and Error Handling

The SEC allows 10 requests per second maximum from any IP. Add a delay:

import time
time.sleep(0.15) # 150ms delay = ~6-7 requests/second

Wrap all HTTP requests in try-except blocks. The EDGAR system returns 503 errors during maintenance windows. Implement exponential backoff retry logic. Log failed requests with their URLs so you can investigate parsing problems without re-running the entire scraper. Include the HTTP status code in your error logs.

Step 8: Structure and Export the Transaction Data

Store parsed transactions in a pandas DataFrame or database. Essential columns: filing_date, transaction_date, reporter_name, reporter_title, transaction_type, shares, price_per_share, total_value. Calculate total value by multiplying shares by price — it's not always in the XML.

Export to CSV or insert into SQL. Include the original accession number and XML URL so you can verify data accuracy by referencing the source filing. When tracking multiple companies, add a ticker_symbol column even though the SEC uses CIK numbers. Makes the data readable for financial analysis. For market context on why insider transactions matter, see how external events influence trading behavior.

What Most Guides Miss: The Derivative Problem

Stock options and other derivatives have a different XML structure under <derivativeTable>. Elements like <conversionOrExercisePrice> don't exist in the non-derivative section. Your parser will fail silently if you don't account for this.

Most scrapers handle only non-derivative transactions to keep complexity manageable. That's fine for basic insider tracking. But if you're analyzing executive compensation patterns, you're missing half the story. Options grants tell you what management expects the stock to do. Open-market purchases tell you what they believe right now. The distinction matters.

Common Problems

XML parsing fails with namespace errors: Form 4 documents use XML namespaces that ElementTree requires you to specify explicitly. Extract the namespace from the root element's xmlns attribute and prepend it to all element paths, or use the .// descendant search which ignores namespaces.

Some filings return 404 errors: Not all Form 4 filings follow the standard naming convention. Check the filing's detail page for the exact XML filename — some use patterns like wf-form4_123456789.xml instead of form4.xml.

Derivative transactions parse incorrectly: You need separate parsing logic for derivatives versus common stock transactions. Many scrapers handle only non-derivative transactions to keep complexity manageable.

Best Practices

• Cache downloaded XML files locally with the accession number as filename — re-parsing cached files is faster than re-requesting them and reduces SEC server load
• Validate extracted data ranges by checking that share counts are positive integers and prices fall within reasonable bounds for the stock's trading range
• Track filing amendments by monitoring version numbers — insiders sometimes file corrected versions of Form 4s
• Maintain a request log with timestamps to prove SEC rate limit compliance if your access gets questioned
• Compare your parsed results against the SEC's HTML rendering of the same filing as a quality check — discrepancies indicate parsing bugs

When Not to Use This

If you need real-time alerts within minutes of filing, a scraper running on a schedule introduces latency. Third-party data vendors offer paid APIs with instant webhooks. For one-time research projects covering fewer than 50 filings, manually downloading XML files from SEC.gov is faster than writing and debugging scraping code. This approach also breaks down for historical analysis spanning decades — the SEC's bulk download archives are more efficient for large-scale historical datasets. If you lack programming experience, start with simpler EDGAR API approaches before building a custom scraper.

FAQ

How do I download SEC Form 4 filings automatically every day?

Set up a scheduled task — cron job on Linux, Task Scheduler on Windows — that runs your scraper script once daily after market close. The SEC updates EDGAR continuously, but most Form 4 filings appear within two business days of the transaction date. Check for new accession numbers by comparing today's index page against yesterday's cached version. Only download XML files you haven't already processed.

Can I parse SEC Form 4 XML with Python's built-in libraries?

Yes. xml.etree.ElementTree from Python's standard library handles Form 4 XML parsing without external dependencies. For more complex XML manipulation or XPath queries, consider the lxml library, but ElementTree is sufficient for extracting transaction tables. Both work with the same parsing logic shown in this guide.

How do I automate insider trading data collection across multiple companies?

Maintain a list of CIK numbers for your target companies and loop through them with your scraper, respecting the 10 requests per second rate limit. Store results in a single database table with a company_cik column to keep all insider transactions queryable together. Schedule the scraper to run overnight when market data isn't changing. Implement resume logic so a mid-run failure doesn't force you to restart from the beginning.

What's the difference between Form 4 and Form 5 insider filings?

Form 4 reports most insider transactions within two business days of the trade. Form 5 is an annual filing covering certain exempt transactions and deferred reporting situations. For real-time insider trading monitoring, focus on Form 4. The XML structure and parsing approach work identically for both form types — just change type=4 to type=5 in your index URL.

```