How to Fetch Insider Transactions Using SEC Edgar API

```html

The SEC's Edgar system lets you pull insider transaction data programmatically — no authentication, no API key, just HTTP requests and XML parsing. This guide walks through building a Python script that retrieves recent Form 4 filings for any ticker and extracts buy/sell details. Rate limit: 10 requests per second. Cost: zero.

Before You Start

The SEC provides free public access to Edgar filing data. The catch: you must identify your requests with a real contact email in the User-Agent header. Violate rate limits or fail to declare your identity, and your IP gets blocked — usually for several minutes, sometimes longer. This guide assumes you understand HTTP requests, JSON parsing, and basic Python scripting. No API key. No account. Just compliance with SEC technical guidelines.

What You Need

• Python 3.8 or later
• The requests library (pip install requests)
• A valid email address for your User-Agent header
• Basic familiarity with SEC Form 4 — the filing that reports insider transactions within two business days
• Optional: pandas for structuring output (pip install pandas)

Step 1: Configure Your Request Headers

The SEC blocks requests that don't include a User-Agent header with your company name and email. Define your headers at the top of your script. This isn't bureaucracy — it's how the SEC contacts you if your script creates performance issues.

headers = {
    "User-Agent": "YourCompanyName yourname@example.com"
}

Replace YourCompanyName and yourname@example.com with real values. Generic User-Agents like "Python-requests/2.28" will be blocked. The SEC requires human-identifiable contact information.

Step 2: Map Ticker Symbol to CIK Identifier

Edgar organizes filings by CIK (Central Index Key), not ticker symbols. Use the Company Tickers JSON endpoint to retrieve the CIK for your target company.

import requests
import json

url = "https://www.sec.gov/files/company_tickers.json"
response = requests.get(url, headers=headers)
tickers = response.json()

# Find CIK for a specific ticker
ticker_target = "MSFT"
cik = None
for entry in tickers.values():
    if entry['ticker'] == ticker_target:
        cik = str(entry['cik_str']).zfill(10)
        break

print(f"CIK for {ticker_target}: {cik}")

The zfill(10) method pads the CIK to 10 digits with leading zeros, matching the format used in Edgar URLs. For $MSFT, the CIK is 0000789019.

Step 3: Query Recent Submissions for Form 4 Filings

Once you have the CIK, request the company's recent submission history. This JSON feed includes metadata for all recent filings — form type, filing date, accession number, primary document filename.

submissions_url = f"https://data.sec.gov/submissions/CIK{cik}.json"
response = requests.get(submissions_url, headers=headers)
submissions = response.json()

recent_filings = submissions['filings']['recent']
form4_indices = [i for i, form in enumerate(recent_filings['form']) if form == '4']

The recent_filings object contains parallel arrays. Filter for entries where form == '4' to isolate insider transaction reports. The interesting part: not every Form 4 contains an actual trade.

Step 4: Construct URLs for Form 4 Documents

Each Form 4 filing has an accession number (format: 0001234567-12-345678) and a primary document filename (typically wf-form4_123456789.xml). Combine these with the CIK to build the full document URL.

for idx in form4_indices[:5]: # Get the 5 most recent Form 4s
    accession = recent_filings['accessionNumber'][idx].replace('-', '')
    primary_doc = recent_filings['primaryDocument'][idx]
    filing_date = recent_filings['filingDate'][idx]
    
    doc_url = f"https://www.sec.gov/cgi-bin/viewer?action=view&cik={cik}&accession_number={accession}&xbrl_type=v"
    print(f"{filing_date}: {doc_url}")

These URLs point to the SEC's online document viewer. For programmatic parsing, construct direct XML file URLs by replacing the viewer link format with the archive path structure.

Step 5: Download and Parse Form 4 XML

Form 4 documents are structured XML files following the SEC's ownership schema. Download the raw XML and extract transaction details using Python's built-in XML parser.

from xml.etree import ElementTree as ET

# Construct direct XML URL
xml_url = f"https://www.sec.gov/Archives/edgar/data/{cik}/{accession}/{primary_doc}"
response = requests.get(xml_url, headers=headers)
root = ET.fromstring(response.content)

# Parse transaction details
for txn in root.findall('.//nonDerivativeTransaction'):
    txn_date = txn.find('.//transactionDate/value').text
    txn_code = txn.find('.//transactionCode').text
    shares = txn.find('.//transactionShares/value').text
    price = txn.find('.//transactionPricePerShare/value').text
    print(f"{txn_date}: {txn_code} {shares} shares at ${price}")

Transaction codes: P = purchase, S = sale, A = grant/award, M = exercise of options. This parsing approach works for non-derivative transactions (common stock). Derivative transactions (options, warrants) require additional parsing logic for strike prices and expiration dates.

Step 6: Implement Rate Limiting

The SEC enforces a 10-request-per-second limit across all endpoints. Add a delay between requests to stay compliant.

import time

for idx in form4_indices:
    # ... your request logic ...
    time.sleep(0.11) # Ensures under 10 requests/sec

Hit the rate limit and you get a 403 Forbidden response. The block lasts several minutes. Production scripts should implement exponential backoff and retry logic — the SEC doesn't provide rate-limit headers, so you track this client-side.

Common Problems

403 Forbidden errors despite correct headers: Verify your User-Agent includes both a company name and a valid email address. Single-word User-Agents or generic library names will be blocked.

Missing transaction data in parsed XML: Not all Form 4 filings include actual transactions. Some report no activity or correct previous submissions. Check for the presence of <nonDerivativeTransaction> or <derivativeTransaction> elements before parsing. Empty filings have <noSecuritiesOwned> tags instead.

Accession number format mismatches: The submissions JSON uses hyphenated accession numbers (0001234567-12-345678), but document archive URLs require hyphens removed. Always call .replace('-', '') when constructing file paths, or you'll get 404 Not Found.

What Most Developers Miss

The deeper question isn't whether you can parse Form 4 filings — it's whether the data means what you think it means. Transaction code P indicates a purchase, but it doesn't distinguish between a CEO buying $5 million of stock on conviction and an employee buying $1,200 through automatic dividend reinvestment. Form 4 reports the transaction. It doesn't report intent.

If you're building an insider buying screen, filter for transaction code P and exclude transactions under $10,000 total value. Small transactions often represent automatic plans rather than conviction buys. For portfolio-level ownership analysis, you'll need to maintain a running ledger across all filings — Form 4 reports individual transactions, but it doesn't aggregate beneficial ownership or show the insider's total position after the trade.

Best Practices

• Cache the company_tickers.json file locally and refresh it weekly rather than fetching it on every script run — the SEC updates this file infrequently
• Store raw XML files locally before parsing, especially for batch processing — this lets you re-parse without re-downloading if you discover parsing errors later
• Log your request timestamps and response codes to detect when you're approaching rate limits
• For production use, query the submissions endpoint once and cache results for the day — insider filings are time-stamped but not updated retroactively
• When screening for insider buying patterns, watch for clusters: multiple insiders buying within the same week signals more than one person buying alone

When Not to Use This

The SEC Edgar API is not designed for real-time alerting. Form 4 filings are legally required within two business days of a transaction, but companies often file at the end of that window. Need same-day insider transaction alerts? Use a commercial data provider that ingests Edgar filings continuously and sends push notifications.

The API also doesn't provide historical bulk downloads. To analyze insider patterns over multiple years, you'll need to iterate through quarterly submission archives or use the SEC's bulk data sets available via Financial Statement Data Sets.

This approach also breaks down for companies with complex ownership structures. For portfolio-level ownership analysis, consider using Form 13F data instead, which reports institutional holdings quarterly.

FAQ

How do I parse SEC Form 4 with Python if the XML structure varies?

Form 4 XML files follow a consistent schema published by the SEC, but optional fields can be omitted. Use defensive parsing with .find() checks before accessing .text attributes. Wrap your parsing logic in try-except blocks to handle missing elements gracefully. The SEC provides the official Ownership XML Schema for reference when building robust parsers.

Can I automate insider trading alerts using this API?

Yes, but you'll need to poll the submissions endpoint at regular intervals — every 15–30 minutes — and compare results against your previous query to detect new filings. Store the most recent accession number for each tracked company in a database, then alert when a new Form 4 appears. This polling approach introduces latency. The fastest commercial services monitor Edgar directly via FTP and can alert within seconds of filing.

How do I download Form 4 filings programmatically in bulk?

For historical analysis, use the SEC's daily index files available at https://www.sec.gov/Archives/edgar/daily-index/. These tab-delimited text files list all filings submitted each day, including Form 4s. Download the indices for your date range, filter for Form 4 entries, then construct document URLs using the CIK and accession number columns. This approach is more efficient than querying individual company submission feeds when analyzing hundreds of companies.

What's the difference between Form 3, Form 4, and Form 5 for insider tracking?

Form 3 reports initial beneficial ownership when someone becomes an insider (director, officer, or 10% shareholder). Form 4 reports transactions and must be filed within two business days. Form 5 is an annual summary that includes small transactions exempt from Form 4 reporting — it's filed once per year, typically in February. For real-time insider activity monitoring, focus exclusively on Form 4 filings.

The next thing to watch: whether your parsed transactions match the totals reported in the filing's footer section. Form 4 includes a "Shares Owned Following Transaction" field that serves as a built-in validation check — if your running total after applying all parsed transactions doesn't match that number, your parser missed something or handled derivative securities incorrectly.

```