Research Replication and Data Engineering
I developed a Python-based research pipeline that reproduces and extends the website-disclosure measurement process from the Journal of Accounting Research study Company Websites: A New Measure of Disclosure. The pipeline retrieves historical website metadata from the Internet Archive, reconstructs website-size and disclosure-topic measures, and produces structured outputs that can be compared with the original research workflow.
Public companies distribute information through their websites in addition to traditional financial filings. Historical versions of these websites can be used to study how companies changed their online disclosure practices over time.
This project focused on reproducing the published measurement process, validating the recreated results, and translating the original workflow into a cleaner and more configurable Python pipeline.
The pipeline begins with a CSV containing company identifiers and website domains. Command-line options allow the workflow to run on the complete panel or on smaller batches for testing and validation.
The system queries the Internet Archive CDX API for archived captures associated with each company domain. Query settings such as year range, match type, collapse rules, row limits, retries, and worker counts can be configured from the command line.
Raw CDX JSON files are converted into Pandas DataFrames. The pipeline validates timestamps, removes unsuitable HTTP status codes, converts capture lengths to numeric values, and creates quarterly and annual period variables.
Capture lengths are averaged at the URL-period level and then summed to create company website-size measures. The pipeline also records URL counts, capture counts, newly observed URLs, and logarithmic website size.
Archived resources are classified by MIME type, including HTML and text content, images, video, audio, PDFs, Word documents, spreadsheets, presentations, compressed files, and other application formats.
Topic dictionaries are matched against words contained in archived URL paths. The workflow calculates annual topic-size measures for categories such as investor relations, product-market content, geographic information, and human-resources information.
An optional second stage downloads selected archived HTML captures, stores the original webpages locally, extracts visible text, and creates a webpage cache index for later textual analysis.
Generated website-size and topic outputs can be compared against the authors’ original workflow. The pipeline also records company-level and total runtime information to evaluate performance as the company panel expands.
The pipeline generates multiple datasets for inspection and downstream research:
The recreation extends the basic research calculations with features that make the workflow easier to test, monitor, and scale.
The webpage-processing stage uses a custom HTML parser to extract human-visible text while ignoring hidden or non-content elements such as scripts, styles, templates, SVG elements, and canvas elements.
The extracted text is stored separately from the original HTML. This allows future research measures to use webpage content without repeatedly downloading or reparsing the archived pages.
I tested the recreated workflow against outputs from the original research implementation. The validation process compared website-size measures, annual capture and URL counts, topic totals, and disclosure-topic proportions.
This process helped identify differences caused by URL classification, filtering rules, aggregation order, and the treatment of individual archived captures. The comparison was used to refine the recreation so that it followed the published methodology more closely.