Data Engineering Project
I developed a custom Python data pipeline that collects, processes, and organizes historical investor-relations content from the Internet Archive’s Wayback Machine. The system was designed to move beyond simple webpage downloading by discovering relevant archived pages, following high-value disclosure links, extracting structured information, and storing results for later financial research.
Company websites contain investor disclosures such as earnings releases, annual reports, SEC filings, presentations, and press announcements. Historical versions of these pages are available through the Wayback Machine, but the content is distributed across changing URLs, archived page structures, intermediary download links, and multiple file formats.
The goal of this project was to create a repeatable pipeline that could locate archived investor pages, download relevant captures, identify important linked content, and organize the results into a structured research database.
The pipeline begins with configured company domains and searches a range of historical investor-relations URL patterns. It supports older website structures, modern investor subdomains, and company root pages.
The system queries the Wayback Machine CDX API and identifies archived captures for relevant investor pages. Captures are categorized into page types such as investor homepages, earnings indexes, SEC filing pages, annual report pages, and press-release archives.
Archived pages and linked documents are downloaded using a multithreaded workflow. Download status, timestamps, errors, file paths, source type, and capture metadata are recorded in SQLite.
Downloaded pages are parsed with Beautiful Soup. The parser extracts page titles, visible text, links, disclosure headlines, dates, and linked document URLs while filtering navigation text, unrelated content, and duplicate records.
High-value links are scored and added to a depth-two crawl queue. Priority is given to content such as earnings results, annual reports, SEC filings, presentations, transcripts, proxy statements, and press releases.
The pipeline recognizes older intermediary URLs, including links that route through pages such as SEC filing or download handlers. It follows these pages to locate the final archived document when possible.
A reporting layer summarizes capture counts, parsing status, disclosure items, crawl depth, failures, intermediary resolutions, and results by company and year.
The SQLite database separates the workflow into four main tables:
The current database includes archived investor content for Microsoft, NVIDIA, and JPMorgan Chase, with disclosure records covering multiple historical years.
This project strengthened my understanding of data engineering workflows that must operate against inconsistent external data. Historical websites frequently change their URL structures, page layouts, and document-linking methods, so the pipeline required flexible discovery rules, persistent storage, duplicate handling, error recovery, and repeated parsing cycles.
It also gave me experience designing a system in stages rather than as one large script. Discovery, downloading, parsing, link queuing, progress monitoring, and analysis are handled by separate components coordinated by a main pipeline runner.