A cache-aware crawler for UWI web domains. It extracts page metadata, classifies pages by campus, category and type, and lists everything in a searchable, filterable directory with inline editing and scheduled crawl jobs.
How it knows: The crawler honours robots.txt including Crawl-Delay, sends ETag and If-Modified-Since conditional requests, and caps concurrency per domain with a semaphore.
Nicholas Smith - all work