Generating end-of-life static archives of CMS-based websites

Most of our attention regarding websites is around creating new ones, but we cannot escape the reality of having to deal with retiring old or existing websites as well. After facing this scenario a few times at Miranj we built a tool to handle the end-of-life stage for websites. This tool attempts to crawl all publicly accessible URLs for a site and create a static archive, much like the Internet Archive’s Wayback Machine. It has a few built-in smarts to detect and capture the many different ways assets are referenced in modern markup. It generates a folder structure that retains clean URLs, as well as intelligently handles URLs with query parameters.

While the tool is not yet publicly available, we have been using it internally and some of the static archives generated by the tool of erstwhile CMS-based sites have been running smoothly in the wild for years.

email hidden; JavaScript is required if you’re interested in archiving a content-heavy site of your own. Always happy to help or exchange ideas around digital archival¹.