Issue 61 contains seven articles that we believe will help continue Code4Lib Journal\u2019s mission \u201cto foster community and share information among those interested in the intersection of libraries, technology, and the future.\u201d [2]<\/a> In no particular order, they are:<\/p>\n We hope you enjoy these articles and find them useful. Thanks again for supporting Code4Lib Journal! <\/p>\n [1]<\/a> “Mission,” Code4Lib Journal, accessed October 10, 2025, https:\/\/journal.code4lib.org\/mission<\/a>.<\/p>\n [2]<\/a> “Mission,” Code4Lib Journal.<\/p>\n","protected":false},"excerpt":{"rendered":" Welcome to the 61st issue of Code4Lib Journal. We hope that you enjoy the variety articles published in this issue.<\/p>\n","protected":false},"author":18,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[132],"tags":[],"class_list":["post-18575","post","type-post","status-publish","format-standard","hentry","category-issue61"],"_links":{"self":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18575","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/comments?post=18575"}],"version-history":[{"count":1,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18575\/revisions"}],"predecessor-version":[{"id":18583,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18575\/revisions\/18583"}],"wp:attachment":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/media?parent=18575"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/categories?post=18575"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/tags?post=18575"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}},{"id":18489,"date":"2025-10-21T15:58:58","date_gmt":"2025-10-21T19:58:58","guid":{"rendered":"https:\/\/journal.code4lib.org\/?p=18489"},"modified":"2025-10-22T09:18:29","modified_gmt":"2025-10-22T13:18:29","slug":"mitigating-aggressive-crawler-traffic-in-the-age-of-generative-ai-a-collaborative-approach-from-the-university-of-north-carolina-at-chapel-hill-libraries","status":"publish","type":"post","link":"https:\/\/journal.code4lib.org\/articles\/18489","title":{"rendered":"Mitigating Aggressive Crawler Traffic in the Age of Generative AI: A Collaborative Approach from the University of North Carolina at Chapel Hill Libraries"},"content":{"rendered":" By Jason Casden, David Romani, Tim Shearer, and Jeff Campbell<\/p>\n Since the release of multiple competing generative AI platforms, the University of North Carolina at Chapel Hill University Libraries has seen increasingly overwhelming and service-disrupting traffic from web crawlers, including those from previously well-behaved hosts such as Google and Microsoft. This traffic consists of an unprecedented flood of evasive and extremely aggressive requests, which caused multiple consecutive days of intermittent service disruption for our public catalog interface (and later to various digital collections services). Although most of our peers in libraries, archives, and the broader open web have also been acutely impacted[1a<\/a>][2a<\/a>], our institution was among the first to experience severe service disruptions from this activity. This article will describe these impacts, as well as our evolving and escalating attempts to mitigate the denial-of-service caused by these automated crawls. We will discuss our novel facet-based bot detection and blocking technique, practices identified by other institutions, and opportunities for collaborative detection and blocking of “bad bots” (i.e., bots that aggressively query our services without regard for robots.txt or other policies) among libraries and archives.<\/p>\n When UNC Libraries first began to see our systems under an unprecedented and unsustainable wave of traffic in the Spring of 2024, we began to check in across our professional networks. Within traditional IT academic communities (i.e., campus schools and departments), no systems were adversely affected. However, within library and archival IT communities nationwide, it was clear many systems were under strain. This excessive traffic was successfully mitigated by UNC Libraries software and system engineers in fairly short order, ensuring systems were available to serve researchers. When our systems detected a second, and seismic, wave of automated web crawls in late fall 2024, we found, mostly through informal communications, the same national patterns, but far more intense, almost exclusively in memory institutions. These informal communications took a number of forms. As a group was moving into a meeting agenda or winding one down with informal chat, one or two people would mention that their systems were receiving traffic that seemed similar to distributed denial-of-service (DDoS) attacks, and others would chime in with similar experiences. For example, LYRASIS-hosted Fedora Repository Program governance calls in 2024 had so many side conversations about bot activity that it spun off an interest group that is meeting regularly and has begun to compile resources. In the Code4lib community Slack instance[3<\/a>], a new messaging channel was established, but not publicized, as a forum for an international community of technical libraries and archives colleagues to discuss their attempts to mitigate crawler-driven downtime. Additionally, as our colleagues shared our internal communications with their professional networks, we started receiving requests for consultations with other institutions.<\/p>\n The resulting information sharing is organic, inconsistent, and clearly not reaching all affected institutions. This lack of comprehensive awareness and coordinated response is exacerbated by the fact that systems are not predictably affected. Staff may only need and seek out a shared understanding and response after their own system is affected. Additionally, unaware of the global pattern, individuals often attempt to resolve the issue locally and only come to understand the broader nature of the attacks after spending weeks or months trying to restore access to their impacted systems. In many cases, this is the first time IT staff in libraries and archives have experienced a significant and coordinated security event.<\/p>\n As mentioned, there are efforts to bridge these gaps. We conducted an informal survey in late March 2025, and we are aware of at least one other similar survey. Our goal was to begin documenting the extent of the impact, better understand which systems are affected, identify solutions that seem to be working, and gauge interest in building a community around the topic. We found that roughly a third of respondents were from outside academic libraries and a quarter were from outside the U.S., hinting at the global impact of this bot traffic. Sixty-five percent of respondents are actively battling bot traffic or experiencing degraded services. The Engelberg Center on Innovation Law & Policy and the Confederation of Open Access Repositories (COAR) asked similar questions in surveys conducted in April 2025 and found that 74.4% and 75.5% of respondents, respectively, are actively engaging in bot mitigation[1b<\/a>][2b<\/a>]. More than 95% of respondents to our survey are interested in coming together collectively to share and learn from each other. The LYRASIS-hosted group is actively exploring ways to further this goal.<\/p>\n As the same software and systems engineers tackled the second wave of activity in the fall of 2024, they found it was clearly different from the first wave. The apparent source of the traffic was masked, and behavior evolved to outflank each effort to block it. We soon realized we needed help and began sharing data and ideas with our regional academic library IT colleagues. We also initiated talks with our university IT security professionals, leading to a call in early 2025 between Libraries IT staff and both security and networking staff from the university IT division. As we described the problem and expressed uncertainty about whether this qualified as a DDoS attack, a security colleague responded with a clarifying message that “authorial intent does not matter” and both offices generously offered their help. Over the next week, a cross-departmental group with representatives of Libraries IT, the campus security office, and the campus networking office determined that implementing additional network infrastructure was the most promising intervention. The security office was also able to confirm that, in their professional networks, the issues were not widespread across academic institution systems, but were specifically affecting memory institutions located within those campuses. Our central campus networking office referred us to a Web Application Firewall (WAF) vendor and worked closely with us to configure and deploy a cloud-based load balancer and WAF. We were fortunate to have available funding and support from the University Libraries Administration to pursue this vendor-based approach.<\/p>\n Our initial response to early reports of sluggish responses from the catalog was to focus on scaling and optimization. We increased server resources, tuned web server configurations, profiled the application, and implemented targeted performance improvements. Although we did not have a load balancer in place at the time, colleagues at other institutions scaled more dramatically behind load balancers, adding more and more servers as the traffic worsened. Our attempts to control this overwhelming traffic, with significant contributions from many of our colleagues, began with more familiar blocking methods and evolved into a layered suite of mitigation methods.<\/p>\n We have always been willing to block IP addresses in our local host-based firewalls when we observe malicious requests. These typically involve repeated attempts to break into the web server or na\u00efve denial-of-service attacks. While we have automation in place to deploy configuration changes to the firewall, this was typically a one-off manual process, triggered when the activity crossed a threshold and generated alerts in our monitoring system. Other sources of blocks include IP address and user agent lists shared by our colleagues at NCSU Libraries and Princeton University Libraries and lists compiled from other sources beyond libraries[4<\/a>][5<\/a>].<\/p>\n In the spring of 2024, we noticed a new kind of traffic: very high-volume requests from a single IP address or a narrow range of addresses, initially targeting full-text resources and eventually expanding to images and other audiovisual content. The traffic originated from commercial cloud providers, often Amazon AWS or Microsoft Azure, making it unlikely to be human-generated. If we banned one IP address, the traffic would often shift to an adjacent one. The scale of this activity quickly overwhelmed our capacity for manual intervention, and if left unchecked, it would have exhausted our server resources. This traffic also included user agents tied to commercial AI crawlers, such as Anthropic’s “ClaudeBot.” This led us to our first attempt at an automated response: fail2ban[6<\/a>] continuously monitors log files and, when it detects a user-defined pattern, inserts a firewall block that is later removed after a configurable duration. It also supports escalating penalties for repeat offender IPs. Recognizing that user-agent strings are a soft identifier, we used available lists to build a fail2ban “jail” for known AI crawler user agents while we developed a strategy for providing data to AI clients. Short bans of just a few hours proved insufficient, so we adopted a progressive ban cycle for multiple offenses: 1 day, 7 days, 1 month, 6 months, and 1 year. After discussions among internal stakeholders, this ban of known AI user agents was extended to all our public-facing websites and continues to identify addresses to re-ban or ban for the first time.<\/p>\n We began to see a new kind of web crawling traffic in the late fall of 2024. It first showed up on our catalog discovery layer rather than sites with full text or other similar resources, and it was more aggressive than previous crawlers. It sidestepped our fail2ban user agent solution, as it utilized the common, generic user agent “Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/136.0.0.0 Safari\/537.36 Edg\/136.0.0.0.” It also no longer originated from commercial cloud providers but from mainstream ISPs such as AT&T, Comcast, Spectrum, and Verizon-Home.<\/p>\n Again, the volume of these requests overwhelmed server resources. It defeated our attempts to tune application performance and consumed any new server resources. Analysis of our logs showed several odd characteristics. There were identical simultaneous requests from multiple disparate IP addresses. This indicated a deep pool of addresses was available and we suspected a residential proxy network was in use. Most of these hundreds of thousands of clients were making something like ~10 requests per minute in bursts of 3-5 minutes, effectively functioning as a DDoS attack. When we attempted to block at 80 requests per 5 minutes, and later at 60 requests per 5 minutes, we experienced brief respites before new bots would return with slightly slower streams of requests.<\/p>\n After standard IP blocks and request throttling both failed, a deeper examination of the logs revealed that these bots were requesting huge numbers of very large and unusual combinations of facets, producing costly and unusual requests. These types of queries can be very resource-intensive and slow to return, contributing to a vicious cycle of slow responses, large request queues, and repeated query attempts. Many of these queries involved subject headings that were far outside our collecting strengths. To illustrate: in the entire month of November, we received 15 searches using the terms “Finnish” and “Music.” On December 4th, 2024 alone, we received 11,329 of these searches. More importantly, we also discerned that many, if not most, of these requests used a far higher number of facets than we would expect from a human user. For example, in all of November 2024, we received 294,398 requests with 15 or more facets applied (many of which were likely also bots). On December 4th alone, we received 140,104. Also on December 4th, we received 17,527 requests with 20 or more facets applied and 203 requests with 25 or more facets applied. That day’s 648,861 requests came from 22,861 distinct IP addresses with no clear relationship between them. These numbers do not include any requests from the millions of IP addresses we had already blocked in the previous phase.<\/p>\n Less than two weeks later, Google published faceted navigation SEO best practices[7<\/a>], stating that “faceted navigation is by far the most common source of overcrawl issues site owners report to us.” We later read about the development of “bot traps,”[8<\/a>] and realized that libraries may have unwittingly created bot traps by creating broadly and deeply filterable discovery interfaces. We began to explore the idea of using faceted interfaces as a kind of security honeypot[9<\/a>] where we could detect signs of non-human interaction and establish bans that would propagate to other systems where that traffic may be more difficult to detect.<\/p>\n We developed a regular expression filter for fail2ban to isolate requests with more than a certain number of facets. We collaborated with our User Experience department and catalog committee to determine the ideal facet threshold, settling on a range of 8 to 12, depending on the severity of traffic, while allowing all queries from on-campus and VPN networks. After the facet-based bot blocking method went live on the afternoon of December 5th, 2024, we almost immediately saw large numbers of IP addresses being banned, 700 in the first 30 minutes, over 10,000 after 24 hours, and 16,000 after 48 hours. This reflects, in part, the very deep pool of IP addresses that the attackers were able to draw on. The 3,604,924 requests from the week before we deployed the facet-based blocks (November 28th-December 4th) originated from 207,351 distinct IP addresses. In contrast, the week after the facet-based blocks were deployed (December 6th-December 12th) saw 673,031 requests (an over 81% drop) from 220,110 distinct IP addresses. The distinct IP address count was likely sustained due to the ability to submit a successful “bad” query before being banned. We began sharing the details of our method with other institutions the next day.<\/p>\n We also saw adaptive behavior based on the length of the original ban, which started at just 2 minutes and began after the second “bad” faceted query. The initial ban eventually grew to 12 hours after the first offense, with longer bans for repeat offenses. Although this solution was sufficient for nearly a month, the crawlers eventually managed to evade the bans by limiting the number of facets in their queries and drawing from larger sets of IP addresses. We have continued to find this practice helpful as a complement to the other measures described below, as we experienced renewed outages when it was removed. Additionally, in the five months since this change, we have received only one user report of a problem with this limit.<\/p>\n After several weeks of stability, we began to struggle under the weight of huge numbers of neighboring IP addresses. We modified our facet-based ban rules from blocking just the offending IP address to blocking the whole \/24 subnet; that is, the pool of 256 IP addresses in the last dotted quad of the address, for an offense by any IP in that subnet. We also lowered the facet limit and started banning clients after a single flagged query. We were seeking to expand our protection to stabilize our servers with the least possible collateral damage to human users.<\/p>\n Finally, the traffic moved offshore to East Asia, notably the People’s Republic of China (PRC). While this allowed us to programmatically ban large networks, we quickly found ourselves playing a game of whack-a-mole again. That is, once again managing an ongoing sequence of emergent problems, each requiring immediate, but likely temporary, resolution. After discussions with stakeholders, we decided to briefly ban all addresses from the PRC. We reasoned that the catalog was down anyway, and this might provide some temporary breathing room for us to develop a new solution. This had the surprising effect of almost immediately moving traffic to Brazil, and then to various locations in South America and Europe when we blocked Brazil. At this point, we were unable to restrict queries any further without significantly impacting human users. We knew we needed a non-local solution that could sit between the traffic and our server and examine the traffic to either block bots or prioritize on-campus and VPN users.<\/p>\n We approached our campus networking and security colleagues for assistance. They worked through several possible options and then introduced us to their Web Application Firewall (WAF) vendor. A Web Application Firewall (WAF) examines and filters traffic to a web application based on the source, content, or other attributes of a request before the request reaches the application server. We worked in close partnership with our campus networking unit and the vendor to rapidly implement a trial of a cloud-based, GDPR-compliant WAF for the catalog server. On several occasions, this required involving senior engineers at the vendor, with one apologizing for smiling because “it’s just interesting that I haven’t seen something like this before.”<\/p>\n Once the WAF was implemented and configured, we started to see some relief from the traffic. The WAF included a bot filtering mechanism based on ratings of client “trustworthiness,” and also allowed us to more easily monitor and block traffic from troublesome sources. For the fourth time, we spent weeks feeling that we had found a solution. We licensed the WAF for a year, and we started planning to move other services behind it. After nearly a month of stability, however, large numbers of bots began finding their way through the WAF.<\/p>\n We worked with the WAF vendor to evaluate their “AI-driven” bot detection system that can be included as a component of the WAF. This seemed to solve the problem again, but the cost of this system would more than double the WAF price. This was prohibitively expensive, so we began investigating ways to prioritize traffic regionally (e.g., rate-limiting sources of traffic outside the campus and our VPN). The cloud WAF infrastructure made it impossible to include the VPN in these rules since the WAF was located at a non-campus IP address. We began discussing VPN exceptions with campus networking and security colleagues, while also evaluating a recent publication by Jonathan Rochkind[10<\/a>], which described the use of the Cloudflare Turnstile CAPTCHA-style system to detect and block bot traffic.<\/p>\n One common comment from our Libraries colleagues as we shared updates on the situation was something along the lines of, “Oh, no, please, I hope we don’t need a CAPTCHA\u2026” Unfortunately, despite legitimate concerns regarding accessibility and user annoyance, the severity of the crawler situation forced us to explore this option. Fortunately, newer technologies like the free-to-use Cloudflare Turnstile offered a promising middle path. Unlike traditional CAPTCHAs that explicitly challenge users to identify traffic lights or crosswalks, Turnstile operates automatically for most users through a combination of browser fingerprinting, behavioral analysis, and lightweight challenges that typically complete automatically without user intervention. Cloudflare also asserts that Turnstile is WCAG 2.1 Level AA compliant[11<\/a>].<\/p>\n We implemented Turnstile selectively, focusing initially on the most resource-intensive endpoints in our catalog, particularly those involving complex faceted searches that had proven vulnerable to crawler abuse. This targeted approach minimized the impact on typical user workflows while providing robust protection for our most susceptible service points. We configured the system to exempt on-campus and VPN users entirely, ensuring that many of our primary users experienced no change in service. We did not issue challenges for record views or searches without facets, and we only challenge a client once per day.<\/p>\n Results were immediate and dramatic. Server load returned to normal levels within hours of implementation, and we experienced very few further outages on protected endpoints. Those outages were eliminated by restoring our facet-based ban mechanism. In the extremely rare cases where legitimate users encountered service disruptions due to Turnstile (i.e., two incidents in the first four months of use), our cross-departmental response team quickly guided users to solutions.<\/p>\n Our sustained efforts, along with those of the wider libraries and archives community, to counter the evolving practices of these crawlers have yielded several valuable lessons. These insights, developed through a continuous cycle of implementing defenses and observing crawler adaptations, reveal the distinct challenges posed by this modern bot activity and the inherent limitations of conventional security and traffic management measures.<\/p>\n The traffic we have observed originates from a vast number of unrelated IP addresses, often from consumer internet service providers rather than from recognized corporate servers. This distributed network of bots has proven to be adept at evading our mitigation measures, as it undermines the traditional approach of banning attributable blocks of IP addresses. It appears that these requests were coming from a residential proxy network, essentially a legitimized botnet. These are services that can be hired to distribute network traffic over a wide range of (sometimes tens of millions of) unrelated individual internet addresses. These proxy nodes can be established in various ways, but one method is to conceal them within mobile apps, browser extensions, or other software[12<\/a>]. Others include paying users to run this software on their computers and building massive collections of proxied addresses in data centers.<\/p>\n We have found that static, rule-based defenses are insufficiently flexible to respond to crawlers that exhibit a kind of intelligent adaptation. As we implemented countermeasures, we observed crawler behavior rapidly evolving to circumvent our defenses. When we blocked based on user agents or specific request patterns, the crawlers quickly adapted by modifying these identifiers. Unlike traditional web crawlers that follow predictable paths through sites, these new crawlers exhibited what appeared to be machine learning-based adaptability. They would probe different access vectors, analyze our responses, and modify their approach accordingly.<\/p>\n While we are aware of efforts to implement complex bot fingerprinting in libraries, our attempts to identify patterns in request headers have proved largely unsuccessful, as the crawlers routinely rotate or randomize header values. Even when we identified consistent patterns in behavior, these would shift within days or sometimes hours of implementing blocking measures. This high level of adaptation suggests sophisticated orchestration beyond typical web crawling tools, pointing to well-funded operations with significant resources at their disposal.<\/p>\n The crawlers producing this massive wave of traffic are as inefficient as they are evasive. Rather than following responsible web crawling practices, such as respecting robots.txt, utilizing sitemaps, or implementing reasonable rate limits, these crawlers appear to be designed to extract maximum data, regardless of the computational or infrastructure costs imposed on libraries and archives. We observed crawlers repeatedly retrieving the same resources tens of thousands of times per day, ignoring cache headers, and generating needlessly expensive (i.e., slow) queries applying exhaustive combinations of facets. This “brute force” approach suggests either a lack of technical sophistication or, more likely, a business model where the value of comprehensive data extraction outweighs any concern for efficiency or ethical crawling standards. These inefficient patterns, combined with the highly sophisticated adaptive and evasive behavior, support our hypothesis that the entities behind these crawlers prioritize rapid and comprehensive data acquisition over efficient resource utilization, likely because the current commercial value of training AI models outweighs the costs of infrastructure. The contrast in sophistication between evasive and harvesting techniques also suggests the pairing of advanced, massively distributed, and evasive network traffic infrastructure with immature crawling technology. A common refrain from our colleagues emphasized that we would love to share our data with all comers, in any preferred format, if it meant avoiding this abuse.<\/p>\n Our initial response focused on scaling infrastructure and optimizing application performance. However, we quickly discovered fundamental limits to this approach. Even after implementing increasing server resources, optimizing the catalog application, and carefully tuning server configurations, the sheer volume of traffic overwhelmed our systems. Although institutions with cloud-based infrastructure were able to scale much more aggressively by adding servers behind a load balancer, they eventually reported similar experiences as well as concerns related to the fees associated with cloud-based scaling. With on-premises infrastructure, we faced physical hardware limitations that couldn’t be quickly addressed. While organizations across various sectors have experienced increased crawler activity, libraries appear uniquely vulnerable to these aggressive bots. Our discovery interfaces combine deep faceting capabilities, highly structured metadata, extensive collections of unique assets of profound research value, and typically lower access barriers than commercial entities, which creates an ideal target for data harvesting operations. This vulnerability is further compounded by libraries’ commitment to the open access to information, a principle that can sometimes conflict with the need for robust security and business continuity measures.<\/p>\n The Code4Lib Slack channel quickly became a valuable resource for those aware of its existence, while other critical information sharing occurred informally at the margins of unrelated community calls and events. However, we discovered a significant information gap as we communicated with peer institutions: many were completely unaware of this community resource despite experiencing identical issues. We found ourselves repeatedly directing colleagues from other institutions to this channel, highlighting the uneven distribution of community knowledge. When information travels primarily through informal networks, some institutions, particularly smaller ones, are most likely to be left without effective practices. One promising approach involves implementing lightweight “proof of work” checks. Unlike traditional CAPTCHA-style mechanisms, these checks operate invisibly to users while requiring browsers to complete small computational tasks that are trivial for individual sessions but may become prohibitively expensive at crawler scale. Cloudflare Turnstile includes proof-of-work checks as one component of its verification suite. One prominent open-source example is Xe Iaso\u2019s Anubis software[17<\/a>], which “weighs the soul of your connection using a proof-of-work challenge in order to protect upstream resources from scraper bots.” This approach may provide an improved balance between accessibility and security. The computational requirements can be dynamically adjusted based on system load, increasing the barrier during suspected crawler activity while remaining negligible during normal operations. Several academic libraries and archives have recently implemented Anubis to control bot traffic[18<\/a>][19<\/a>][20<\/a>][21<\/a>], a process documented by Sean Aery at Duke University Libraries[22<\/a>].<\/p>\n Our experiments with geographic blocking, while effective in the short term, highlighted the need for more nuanced regional service prioritization. Rather than outright blocking entire countries or regions, future infrastructure could implement tiered access channels that prioritize traffic from the institution’s primary service areas during periods of high load.<\/p>\n These sorts of tiered access models conflict with our professional commitment to open access to information[23<\/a>], but the intensity of these “attacks” has prompted surprising conversations along these lines. Implementation would require careful consideration of equity and access principles, potentially including allowlisting mechanisms for known international research and academic partners, as well as transparent communication about resource allocation during periods of peak demand.<\/p>\n It is also important to note the effect this may have on web analytics reporting. Although bot traffic filtering provided by analytics platforms has always been imperfect, the volume of undetectable machine-driven traffic is likely to massively distort any reporting for periods before the implementation of effective mitigation methods.<\/p>\n The unprecedented surge in aggressive crawler traffic, coinciding with the release of commercial generative AI models, has forced libraries to rapidly evolve our approach to network and application security. While the timing strongly suggests connections to AI training, we cannot definitively attribute these activities to specific organizations or purposes. Regardless of intent, the impact on library services has been substantial, requiring significant financial and human resources to maintain stable access for our human users.<\/p>\n The libraries and archives community has developed a multi-layered defense strategy that has proven, to date, to be somewhat effective against even the most aggressive and evasive crawlers. We have continued to experience intermittent service disruptions related to this traffic, and some of our peers now require logins for catalog access and block access to increasingly broad sets of IP addresses, sometimes representing entire countries. Although client checks such as Cloudflare Turnstile or Anubis are currently the most effective layer of the strategy, the facet-aware detection method introduced in this article represents a novel approach designed explicitly for library discovery interfaces. We strive to transform a potential vulnerability into a detection opportunity by identifying non-human interaction patterns unique to faceted interfaces.<\/p>\n As the generative AI landscape continues to evolve, we anticipate ongoing adaptation from both crawlers and our institutions. By establishing structured channels for sharing detection patterns and mitigation strategies, libraries and archives can collectively build a more resilient infrastructure that maintains our commitment to open access to information while ensuring service stability for our communities. As we look to the future, the international community that formed out of necessity and desperation may serve as a valuable model for a rapidly changing technological landscape.<\/p>\n This work would not have been possible without the cross-departmental team at UNC-Chapel Hill that responded to these incidents over the past year. In addition to the authors of this article, this team includes Sidney Stafford, Alex Everett, Dean Farrell, Jamie McGarty, Joseph Moran, Zachary Tewell, Emily Brassell, Benjamin Pennell, Chad Haefele, and others, with invaluable strategic support from Mar\u00ed<\/span>a Estorino and Michael Barker. We also send our thanks to our regional colleagues at the Triangle Research Libraries Network, and specifically to Adam Constabaris at NCSU Libraries for introducing us to residential proxy networks and to Sean Aery for sharing Duke University Libraries’ experience with the Anubis system.<\/p>\n We are grateful for the Code4Lib Slack #bots channel community for its curiosity, creativity, and generosity. We are particularly appreciative of Jonathan Rochkind’s work demonstrating the integration of Cloudflare Turnstile in a Blacklight application.<\/p>\n Jason Casden<\/em> is the Head of Software Development at the University of North Carolina at Chapel Hill University Libraries. He leads engineers working on a diverse portfolio of projects dedicated to building, maintaining, and innovating the Libraries’ digital services, including digital collections and repositories, archival information systems, digital preservation systems, and the public catalog discovery layer.<\/p>\n David Romani<\/em> is the recently retired Senior Linux System Administrator at the University of North Carolina at Chapel Hill University Libraries. David’s professional focus at UNC was system architecture and automation.<\/p>\n Tim Shearer<\/em> is the Associate University Librarian for Digital Strategies and IT at the University of North Carolina at Chapel Hill University Libraries. In this role he leads the Library Information Technology division and has responsibility for commodity computing, infrastructure, user experience, software development, repository development, discovery, digitization, and information security. Tim is interested in how technology can play a transformative role in supporting discovery, delivery, use, and preservation of information. He holds an MSLS from UNC Chapel Hill\u2019s School of Information and Library Science.<\/p>\n Jeff Campbell<\/em> is the Head of Infrastructure Management Services at the University of North Carolina at Chapel Hill University Libraries. He leads a skilled team of systems professionals responsible for the strategic planning and management of the Libraries\u2019 IT infrastructure, including oversight of server environments, network architecture, cybersecurity measures, and disaster recovery planning.<\/p>\n [1a<\/a>][1b<\/a>] Weinberg M. Are AI Bots Knocking Cultural Heritage Offline? Engelberg Center on Innovation Law & Policy; 2025. https:\/\/glamelab.org\/products\/are-ai-bots-knocking-cultural-heritage-offline\/<\/a><\/p>\n [2a<\/a>][2b<\/a>] Shearer K, Walk P. The impact of AI bots and crawlers on open repositories: Results of a COAR survey, April 2025. COAR: Confederation of Open Access Repositories; 2025. https:\/\/coar-repositories.org\/news-updates\/open-repositories-are-being-profoundly-impacted-by-ai-bots-and-other-crawlers-results-of-a-coar-survey\/<\/a><\/p>\n [3<\/a>] Code4Lib Slack page. Code4Lib. [accessed 2025 May 22]. http:\/\/code4lib.org\/irc\/#slack<\/a><\/p>\n [4<\/a>] networksdb.io: Company to IP, IP address owners, Reverse Whois + DNS, free tools & API. [accessed 2025 May 22]. https:\/\/networksdb.io\/<\/a><\/p>\n [5<\/a>] Dark Visitors – Track the AI Agents and Bots Crawling Your Website. [accessed 2025 May 22]. https:\/\/darkvisitors.com\/<\/a><\/p>\n [6<\/a>] fail2ban\/fail2ban: Daemon to ban hosts that cause multiple authentication errors. 2025 [accessed 2025 May 22]. https:\/\/github.com\/fail2ban\/fail2ban<\/a><\/p>\n [7<\/a>] Illyes G. Crawling December: Faceted navigation | Google Search Central Blog. Google for Developers. [accessed 2025 May 22]. https:\/\/developers.google.com\/search\/blog\/2024\/12\/crawling-december-faceted-nav<\/a><\/p>\n [8<\/a>] Koebler J. Developer Creates Infinite Maze That Traps AI Training Bots. 404 Media. 2025 Jan 23 [accessed 2025 May 19]. https:\/\/www.404media.co\/developer-creates-infinite-maze-to-trap-ai-crawlers-in\/<\/a><\/p>\n [9<\/a>] Provos N, Holz T. Virtual honeypots: from botnet tracking to intrusion detection. Addison-Wesley; 2010.<\/p>\n [10<\/a>] Rochkind J. Using CloudFlare Turnstile to protect certain pages on a Rails app \u2013 Bibliographic Wilderness. [accessed 2025 May 19]. https:\/\/bibwild.wordpress.com\/2025\/01\/16\/using-cloudflare-turnstile-to-protect-certain-pages-on-a-rails-app\/<\/a><\/p>\n [11<\/a>] FAQ \u00b7 Cloudflare Turnstile docs. [accessed 2025 May 21]. https:\/\/developers.cloudflare.com\/turnstile\/frequently-asked-questions\/<\/a><\/p>\n [12<\/a>] Mi X et al. Resident Evil: Understanding Residential IP Proxy as a Dark Service. In: 2019 IEEE Symposium on Security and Privacy (SP). 2019. p 1185\u20131201. https:\/\/ieeexplore.ieee.org\/abstract\/document\/8835239<\/a>. https:\/\/doi.org\/10.1109\/SP.2019.00011<\/a><\/p>\n [13<\/a>] Rochkind J. samvera-labs\/bot_challenge_page: Show a bot challenge interstitial for Rails, usually using Cloudflare Turnstile. [accessed 2025 May 22]. https:\/\/github.com\/samvera-labs\/bot_challenge_page\/<\/a><\/p>\n [14<\/a>] Corall J. libops\/captcha-protect: Traefik middleware to add an anti-bot challenge to individual IPs in a subnet when traffic spikes are detected from that subnet. [accessed 2025 May 23]. https:\/\/github.com\/libops\/captcha-protect<\/a><\/p>\n [15<\/a>] Cliff D. cerberus\/config\/initializers\/zoo_rack_attack.rb at support\/1.x \u00b7 NEU-Libraries\/cerberus. [accessed 2025 May 22]. https:\/\/github.com\/NEU-Libraries\/cerberus\/blob\/support\/1.x\/config\/initializers\/zoo_rack_attack.rb<\/a><\/p>\n [16<\/a>] ai-robots-txt\/ai.robots.txt: A list of AI agents and robots to block. 2025 [accessed 2025 May 22]. https:\/\/github.com\/ai-robots-txt\/ai.robots.txt<\/a><\/p>\n [17<\/a>] Iaso X. TecharoHQ\/anubis: Weighs the soul of incoming HTTP requests to stop AI crawlers. 2025 [accessed 2025 May 22]. https:\/\/github.com\/TecharoHQ\/anubis<\/a><\/p>\n [18<\/a>] Archives & Manuscripts at Duke University Libraries. David M. Rubenstein Rare Book & Manuscript Library. [accessed 2025 Jun 22]. https:\/\/archives.lib.duke.edu\/<\/a><\/p>\n [19<\/a>] Connecticut\u2019s Archives Online – Repositories. Western Connecticut State University Archives. [accessed 2025 Jun 22]. https:\/\/archives.library.wcsu.edu\/arclight\/caoSearch\/repositories\/<\/a><\/p>\n [20<\/a>] UCLA Library Digital Collections. [accessed 2025 Jun 22]. https:\/\/digital.library.ucla.edu\/<\/a><\/p>\n [21<\/a>] Columbia University Libraries Finding Aids. [accessed 2025 Jun 22]. https:\/\/findingaids.library.columbia.edu\/<\/a><\/p>\n [22<\/a>] Aery S. Anubis Pilot Project Report – June 2025. 2025 Jun 26 [accessed 2025 Jul 8]. https:\/\/hdl.handle.net\/10161\/32990<\/a><\/p>\n [23<\/a>] Hellman E. AI bots are destroying Open Access. Go To Hellman. [accessed 2025 May 20]. https:\/\/go-to-hellman.blogspot.com\/2025\/03\/ai-bots-are-destroying-open-access.html<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" The rise of aggressive, adaptive, and evasive web crawlers is a significant challenge for libraries and archives, causing service disruptions and overwhelming institutional resources. This article details the experiences of the University of North Carolina at Chapel Hill University Libraries in combating an unprecedented flood of crawler traffic. It describes the escalating mitigation efforts, from traditional client blocking to the implementation of more advanced techniques such as request throttling, regional traffic prioritization, novel facet-based bot detection, commercial Web Application Firewalls (WAFs), and ultimately, in-browser client verification with Cloudflare Turnstile. The article highlights the adaptive nature of these crawlers, the limitations of isolated institutional responses, and the critical lessons learned from mitigation efforts, including the issues introduced by residential proxy networks and the extreme scale of the traffic. Our experiences demonstrate the effectiveness of a multi-layered defense strategy that includes both commercial and library-specific solutions, such as facet-based bot detection. The article emphasizes the importance of community-wide collaboration, proposing future directions such as formalized knowledge sharing and the ongoing development of best practices to collectively address this evolving threat to open access and the stability of digital library services.<\/p>\n","protected":false},"author":202,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[132],"tags":[],"class_list":["post-18489","post","type-post","status-publish","format-standard","hentry","category-issue61"],"_links":{"self":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18489","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/users\/202"}],"replies":[{"embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/comments?post=18489"}],"version-history":[{"count":2,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18489\/revisions"}],"predecessor-version":[{"id":18585,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18489\/revisions\/18585"}],"wp:attachment":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/media?parent=18489"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/categories?post=18489"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/tags?post=18489"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}},{"id":18462,"date":"2025-10-21T15:58:57","date_gmt":"2025-10-21T19:58:57","guid":{"rendered":"https:\/\/journal.code4lib.org\/?p=18462"},"modified":"2025-10-21T16:00:00","modified_gmt":"2025-10-21T20:00:00","slug":"liberation-of-lms-siloed-instructional-data","status":"publish","type":"post","link":"https:\/\/journal.code4lib.org\/articles\/18462","title":{"rendered":"Liberation of LMS-siloed Instructional Data"},"content":{"rendered":" By Hyung Wook Choi, Jonathan Wheeler, Weimao Ke, Lei Wang, Jane Greenberg, and Mat Kelly<\/p>\n Data liberation is fundamental to enabling data reuse. The concepts of data reuse, data sharing, and open data have gained significant attention in academic and research communities, with the goal of making research outputs more widely accessible for purposes such as education, business, and further scientific inquiry [1<\/a>][2<\/a>]. By promoting open access, researchers can ensure that valuable data sets contribute well beyond their original scope of analysis [3<\/a>]. However, several barriers hinder effective data reuse, including technical difficulties [4<\/a>], the absence of standardized practices and supporting infrastructure [5<\/a>], and concerns about privacy [6<\/a>].<\/p>\n The advancement of Information Technology (IT) has significantly influenced the growth of e-learning systems, enabling more flexible and accessible opportunities for education across various domains. These systems have supported not only formal education but also cultural engagement and both professional and personal skill development, thereby shaping modern educational infrastructures [7<\/a>]. Since much of the educational content has become available online, especially post-COVID [8<\/a>], sharing and reusing educational content holds potential to enhance teaching and learning experiences, promote collaboration among educators, learners, and instructional designers, and ultimately improve educational outcomes. The reuse of educational resources can be adapted and customized into different contexts and strategies to meet specific needs in the fields. Despite the significant benefits, challenges such as copyright concerns, quality assurance, and technical interoperability continue to hinder the sharing and reuse of educational resources and must be systematically addressed [9<\/a>]. For instance, Poole pointed out the short supply of faculty and the lack of shared and sustainable national infrastructure, including datasets, tools, techniques, and instructional demonstrations, for supporting data science education in the Library and Information Science (LIS) field [10<\/a>]. The study emphasized that promoting a national digital platform requires coordinated tools and technologies across the educational lifecycle.<\/p>\n In this paper, we describe a preliminary process of liberating instructional material from the Blackboard learning management system (LMS), representative of efforts from grant-funded, multi-year boot camps that occurred in June of 2021, 2022, and 2023. These boot camps preceded 6-month internships by each fellow at institutions in the United States including university libraries, non-profit organizations, and data science-driven collectives, among others. The instructional material was developed and interactively presented by 10 teaching, tenure-track, and tenured professors at the lead university and attended by 68 fellows over three years of the project (24, 26, and 18 fellows in each of the respective years). The topics covered in the boot camp included Data Management, Cleaning, and Mining; Information Retrieval; Data Curation, Preservation, Metadata, and Ontologies Search, and Machine Learning and ChatGPT among others. The purpose of this data liberation effort is to both widely share the extensive, well-informed instructional material beyond the original cohorts of fellows but also to explore the nuances that need to be considered beyond the technical efforts, like those of privacy, rights restrictions, and the need to disseminate the material in an LMS-agnostic, sustainable, usable platform.<\/p>\n Several studies have been conducted to associate digital archiving with learning management systems (LMS). Vitaglione et al. proposed a new software model for archiving slide-based presentations from the Internet, built around the Lecture Object format designed for storing lectures online [11<\/a>]. This project was developed in collaboration with the CERN HR Division Training and the University of Michigan, aiming to respond to the growing integration of the World Wide Web in educational contexts. Building on this initiative, Bousdira et al. documented the Web Lecture Archiving Project (WLAP), which involved the implementation of web-based archiving technologies to record a series of content-rich presentations at CERN [12<\/a>]. This was accomplished using Sync-O-Matic, a tool that generates slide-based web lectures viewable through a standard web browser. They introduced a new concept of Learning Object which refers to a standard format for archiving web lectures and for exchanging them between different archives. Herr et al. published another study on lecture archiving at CERN and the University of Michigan as a part of the University of Michigan ATLAS Collaboratory Project (UMACP) [13<\/a>]. They updated the work that the new system adopting the Lecture Object architecture would be implemented with the goal of a very long-term archive of rich, high-quality media and associated metadata.<\/p>\n Chu et al. examined the advantages and disadvantages of learning management systems (LMS) and lecture capture (LC) technologies by introducing a blended learning model called the START program, which integrated e-learning with virtual mentorship and was implemented at Stanford University [14<\/a>]. Moodle was selected as the LMS platform for this initiative. Developed by the Anesthesia Informatics and Media Laboratory in the Department of Anesthesia, the program aimed to enhance the preparedness of interns transitioning into anesthesia residency. Their findings indicated that online educational programs can be effective but require substantial technical expertise during the initial setup phase. Jamieson and Verhaart presented three case studies documenting the transition from Blackboard to Moodle [15<\/a>]. Their analysis revealed that Moodle\u2019s features were less specific compared to Blackboard, leading to issues such as errors during content import. More recently, Santoso et al. proposed an archival system, implemented using Docker, to store and manage historical course content and activities, with the goal of alleviating storage demands on the Moodle server [16<\/a>]. Following implementation, they conducted three separate evaluation tests. The results demonstrated that variations in course size significantly impacted the time required to restore backup files.<\/p>\n Given that the primary goal of this effort is to archive content for future reuse, we draw on research in web archiving, particularly efforts related to preserving content behind authentication barriers. While Kelly et al. focused on archiving private and personal content [17<\/a>], their earlier work explored tool development for ad hoc approaches to small-scale data extraction [18<\/a>], which closely aligns with the scope of our project. More recent initiatives, such as the Webrecorder project [19<\/a>], have further democratized personal web archiving, making it more accessible regardless of privacy constraints. This body of work has informed our data extraction methodology and will continue to guide our approach as the archiving project evolves.<\/p>\n \n While we had direct, practical familiarity with the contents of the bootcamp Blackboard instance, we needed to liberate the contents from the authentically-protected system in a manner that maintained the file and course structure as well as the metadata to organize iteration of the bootcamp. This required a process of data extraction, inventory, and auditing for assurance of the process, which we describe in this section. We began this process in the Fall of 2023 and the effort is currently underway (see below timeline).<\/p>\n We exported course data using the functionality built into Blackboard and took inventory of the contents of the export using the DROID file format identification tool from the UK National Archives [20<\/a>]. We then parsed the CSV output of the scan using an R script to generate a count of files by type, with the intent of getting an overview of the number of scripts, notebooks, PDFs, and binary types. The export included files in PowerPoint and Word formats, as well as operating system specific but otherwise contextually unnecessary files like *_MACOS* directories. Given the large number of files in the export, the DROID report also served to confirm assumptions about the structure of the data: one top-level directory containing structural XML files and a small number of subdirectories, primarily used for Blackboard configuration. Among these was a single dedicated subdirectory serving as the content store, which contained all of the embedded course materials.<\/p>\n Since all files within the course export are named according to a schema that is internal to a single Blackboard course instance, with site-specific identifiers replacing human-readable file names, our next step was to mine the XML for filenames. Within the content store directory, each content file was represented by two files: the content in its original format and an XML file that included system information including the original file name. The shared identifier between the two files was the system-generated filename. We developed a Python script to read the XML files and output a lookup table with two columns, the system filename, and the corresponding original filename.<\/p>\n The initial process produced a list of content files using their original filenames. However, conducting a thorough audit required matching these files with the corresponding lesson descriptions within each module. One potential method was to set up a sandboxed Blackboard instance, import the exported data, and compare it with the live version of the course while it remained accessible online.<\/p>\n Instead, a scripted solution was developed. This script parsed the XML files to generate a weekly overview of course content, aligning the schedule with specific XML data to map course materials to each week. To streamline this process, lookup tables were also used rather than relying solely on the raw XML files.<\/p>\n The institution running the Blackboard system uses Kaltura as a long-term video hosting solution, rather than relying on Zoom for video storage. While Zoom recordings are deleted after 18 weeks [21<\/a>], an automated process transfers videos created through the institution\u2019s Zoom account into Kaltura. These videos can then be embedded within Blackboard without being physically copied. Instructors reference the videos via Blackboard, but the actual files remain hosted on Kaltura.<\/p>\n In December 2023, the grant-funded institution hosting the Blackboard instance notified all instructors that the prior embedded instances of Kaltura videos would no longer function and need to be reset in Blackboard. Given all instances of the bootcamp had since completed, this meant that a manual process of adapting the embeds might need to be performed to allow the videos to be accessible and playable. We mitigate this effort by extracting the URLs of the videos in the import rather than relying on the embed markup itself to allow the video to be usable in its original form, i.e., within the Blackboard interface.<\/p>\n As the process of data extraction and migration progresses, it is essential to critically consider the broader implications associated with the dissemination of the resulting materials. Chief among these concerns is the issue of instructional authorization: the participation of instructors in the boot camp events does not inherently imply consent for public distribution of their contributions. Accordingly, explicit permission from each instructor must be obtained prior to dissemination.<\/p>\n Furthermore, a comprehensive review and analysis of the extracted content is required to ensure that copyrighted materials, whether included inadvertently or permitted within the credential-protected Blackboard environment, are not redistributed without proper authorization. For materials that are publicly available but still under copyright, our dissemination strategy includes verifying their presence in the Internet Archive\u2019s repository and providing external links rather than hosting the content directly.<\/p>\n For resources for which reuse permissions have been granted, particularly interactive elements such as Jupyter notebooks, we intend to integrate and adapt these materials into existing Library Carpentry workshop formats. This will support their continued use and extension by broader audiences, contingent upon the prior acquisition of all necessary permissions from the original content creators.<\/p>\n At the time of this writing, we have performed a preliminary audit of boot camp materials that includes instructional videos recordings of the instructors\u2019 lectures, PowerPoint\/PDF presentation files, and Python-based notebooks. While some of the notebooks are Jupyter-based and stored in the Blackboard shell itself, some instructors opted for notebooks on other platforms like Google Colab. While the offerings of these two mechanisms for executing Python scripts using notebooks is similar, there is no guarantee that one notebook will be portable to other systems due to the availability of modules and resources on each respective platform. We plan to evaluate both the notebooks and their linked versions (e.g., in Colab), and manually verify that the execution results match expectations.<\/p>\n Dissemination of the bootcamp material need not maintain the original web-based representation, that is, organized by structure and style of the Blackboard template. We are looking into effective means of organizing the content, both from the context of the multiple iterations of the boot camp to make the content available but also to facilitate accessibility. Our intention is for the content to be widely available and permissively licensed for reuse. We are considering using platforms like GitHub Pages to accomplish this with external linking to captures at Internet Archive for copyrighted, publicly accessible, web-based materials.<\/p>\n Our initial attempts at data liberation have been somewhat ad hoc, relying primarily on the built-in capabilities of the system. As we have seen with other web-based platforms, data exports (“dumps”) are often incomplete. While we continue to assess how comprehensive and representative the Blackboard data dump is in capturing the full content of the boot camps, we are also exploring practitioner-oriented web archiving tools and methods to extract data in a more portable and consistent manner. We plan to use the archiving tools in the near future to implement a more systematic approach and to benchmark their performance against our current baseline. These tools will be especially valuable for capturing hard-to-preserve resources and will also support the portability of the data in its original structure, such as the Blackboard course template.<\/p>\n This study aims to liberate educational content for reuse with Blackboard data from the Summers of 2021, 2022, and 2023. Making the liberated data into reusable content required multiple steps including obtaining permission for copyright, organizing the data dumps for consistency, and assuring the quality of contents. However, if the demonstrated methods could employ a more systematic process, it would bring tremendous potential to educational workers by sharing and reusing the quality of educational contents without concerns about license.<\/p>\n For support of the 3-year grant that made both the fellowships and boot camps possible, we would like to recognize support by the Institute of Museum and Library Services (IMLS), grant #RE-246450-OLS-20. We would also like to thank the instructors at Drexel University for their cooperation and willingness to make their valuable instructional data widely available.<\/p>\n [1<\/a>] National Institutes of Health. (2023). Data Management & Sharing Policy Overview | Data Sharing. https:\/\/sharing.nih.gov\/data-management-and-sharing-policy\/about-data-management-and-sharing-policies\/data-management-and-sharing-policy-overview#after<\/a><\/p>\n [2<\/a>] National Science Foundation. (2023). Preparing Your Data Management Plan – Funding at NSF https:\/\/new.nsf.gov\/funding\/data-management-plan#nsfs-data-sharing-policy-1c8<\/a><\/p>\n [3<\/a>] Piwowar, H. A., & Vision, T. J. (2013). Data reuse and the open data citation advantage. PeerJ<\/em>, 1, e175. doi:10.7717\/peerj.175<\/a><\/p>\n [4<\/a>] Borgman, C. L. (2012). The conundrum of sharing research data. Journal of the American Society for Information Science and Technology<\/em>, 63(6), 1059-1078. doi:10.1002\/asi.22634<\/a><\/p>\n [5<\/a>] Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American psychologist<\/em>, 61(7), 726. doi:10.1037\/0003-066X.61.7.726<\/a><\/p>\n [6<\/a>] El Emam, K., Rodgers, S., & Malin, B. (2015). Anonymising and sharing individual patient data. bmj<\/em>, 350. doi:10.1136\/bmj.h1139<\/a><\/p>\n [7<\/a>] Oliveira, P. C. D., Cunha, C. J. C. D. A., & Nakayama, M. K. (2016). Learning Management Systems (LMS) and e-learning management: an integrative review and research agenda. JISTEM-Journal of Information Systems and Technology Management<\/em>, 13, 157-180. doi:10.4301\/S1807-17752016000200001<\/a><\/p>\n [8<\/a>] Camilleri, M. A., & Camilleri, A. C. (2022). The acceptance of learning management systems and video conferencing technologies: Lessons learned from COVID-19. Technology, Knowledge and Learning<\/em>, 27(4), 1311-1333. doi:10.1007\/s10758-021-09561-y<\/a><\/p>\n [9<\/a>] Hilton III, J., Wiley, D., Stein, J., & Johnson, A. (2010). The four \u2018R\u2019s of openness and ALMS analysis: frameworks for open educational resources. Open learning: The journal of open, distance and e-learning<\/em>, 25(1), 37-44. doi:10.1080\/02680510903482132<\/a><\/p>\n [10<\/a>] Poole, A. H. (2021). LEADING the way: A new model for data science education. Proceedings of the Association for Information Science and Technology, 58(1), 525-531. doi:10.1002\/pra2.491<\/a><\/p>\n [11<\/a>] Vitaglione, G., Bousdira, N., Goldfarb, S., Neal, H. A., Severance, C., & Storr, M. (2001). Lecture Object: an architecture for archiving lectures on the Web. International Journal of Modern Physics C<\/em>, 12(04), 533-547. doi:10.1142\/S012918310100219X<\/a><\/p>\n [12<\/a>] Bousdira, N., Storr, K. M., Myers, E., Goldfarb, S., Neal, H. A., Severance, C., & Vitaglione, G. (2001). WLAP the web lecture archive project: The development of a web-based archive of lectures, tutorials, meetings and events at CERN and at the University of Michigan<\/em> (No. CERN-OPEN-2001-066). https:\/\/cds.cern.ch\/record\/516632<\/a><\/p>\n [13<\/a>] Herr, J., Lougheed, R., & Neal, H. A. (2010, April). Lecture archiving on a larger scale at the University of Michigan and CERN. In Journal of Physics: Conference Series<\/em> (Vol. 219, No. 8, p. 082003). IOP Publishing. doi:10.1088\/1742-6596\/219\/8\/082003<\/p>\n [14<\/a>] Chu, L. F., Young, C. A., Ngai, L. K., Cun, T., Pearl, R. G., & Macario, A. (2010). Learning management systems and lecture capture in the medical academic environment. International Anesthesiology Clinics<\/em>, 48(3), 27-51. doi:10.1097\/AIA.0b013e3181e5c1d5<\/a><\/p>\n [15<\/a>] Jamieson J., Verhaart M. Issues Surrounding Course Content Migration: Blackboard to Moodle. In: Proceedings of the 18th annual conference of the national advisory committee on computing qualifications (NACCQ), Tauranga, New Zealand, 2005. Available at: https:\/\/citrenz.org.nz\/citrenz\/conferences\/2005\/concise\/jamieson_moodle.pdf<\/a><\/p>\n [16<\/a>] Santoso, B. J., Ijtihadie, R. M., & Millah, Z. (2023, September). A Docker Container-Based Solution for Course Archival on Moodle: Implementation and Evaluation. In 2023 8th International Conference on Electrical, Electronics and Information Engineering (ICEEIE)<\/em> (pp. 1-6). IEEE. doi:10.1109\/ICEEIE59078.2023.10334688<\/a><\/p>\n [17<\/a>] Kelly, M. and Weigle, M. C., \u201cWARCreate – Create Wayback-Consumable WARC Files from Any Webpage,\u201d In Proceedings of the ACM\/IEEE Joint Conference on Digital Libraries (JCDL)<\/em>. 2012, pp. 437\u2013438. doi:10.1145\/2232817.2232930<\/a><\/p>\n [18<\/a>] Kelly, M., Nelson, M.L., and Weigle, M.C, \u201cA Framework for Aggregating Private and Public Web Archives,\u201d In Proceedings of the ACM\/IEEE Joint Conference on Digital Libraries (JCDL)<\/em>, 2018, pp. 273\u2013282. doi:10.1145\/3197026.3197045<\/a><\/p>\n [19<\/a>] Webrecorder: web archiving for all. (n.d.). Webrecorder. Retrieved April 1, 2024, https:\/\/webrecorder.net\/<\/a><\/p>\n [20<\/a>] DROID: file format identification tool (n.d.), Retrieved April 4, 2024, https:\/\/www.nationalarchives.gov.uk\/information-management\/manage-information\/preserving-digital-records\/droid\/<\/a><\/p>\n [21<\/a>] Drexel University Information Technology, (n.d.), Frequently Asked Questions for Drexel Zoom Users, https:\/\/drexel.edu\/it\/help\/a-z\/zoom\/zoomfaq\/<\/a> Accessed: April 7, 2024<\/p>\n Hyung Wook Choi<\/em> is a PhD candidate at Drexel University\u2019s College of Computing and Informatics. She holds Master degrees in Data Science and Library and Information Science from Drexel and Ewha Womans University. Her primary research is centered on detecting and tracking semantic shifts with NLP and Information Retrieval.<\/p>\n Jon Wheeler<\/em> is a Data Curation Librarian within the University of New Mexico’s College of University Libraries and Learning Sciences. Jon’s role in the Libraries’ Data Services initiatives include the development of research data ingest, packaging and archiving work flows. Jon’s research interests include the requirements and usability of sustainable architectures for long term data preservation and the disposition of research data in response to funding requirements.<\/p>\n Weimao Ke<\/em> is an Associate Professor in Information Science at Drexel University. His research is centered on information retrieval (IR), with emphasis on computational models that support effective and intelligent human-information interaction. His work explores the application of machine learning, multi-agent systems, and information-theoretic frameworks to the modeling and analysis of complex information environments. He is also interested in simulation-based approaches for studying emergent behavior and dynamics in distributed systems, as well as the design of scalable and adaptive AI systems in information networks.<\/p>\n Lei Wang<\/em> is an Assistant Teaching Professor at Drexel University’s College of Computing and Informatics. Her work focuses on clinical data analysis, natural language processing with large language models, and biomedical optical imaging. At Drexel, she teaches graduate-level courses in data science and leads research initiatives at the intersection of healthcare and AI. Her research interests include pediatric concussion, neuroimaging, and the application of machine learning in clinical informatics. She actively contributes to academic publishing and peer review in neuroscience and biomedical engineering.<\/p>\n Jane Greenberg<\/em> is the Alice B. Kroeger Professor and Director of the Metadata Research Center at the College of Computing & Informatics, Drexel University. Her research activities focus on metadata, knowledge organization\/semantics, linked data, data science, and information economics. Her research has been funded by the NSF, NIH, IMLS, NEH, Microsoft Research, GlaxoSmithKline, Santander Bank, Library of Congress, as well as other agencies and organizations.<\/p>\n Mat Kelly<\/em> is an Assistant Professor in the Department of Information Science at Drexel University\u2019s College of Computing & Informatics. He earned his Ph.D. in Computer Science from Old Dominion University. Dr. Kelly’s research centers on digital preservation, with a particular emphasis on the archiving and retrieval of private web content. His work has been supported by the Institute of Museum and Library Services and the National Science Foundation.<\/p>\n","protected":false},"excerpt":{"rendered":" This paper presents an initiative to extract and repurpose instructional content from a series of Blackboard course shells associated with IMLS-funded boot camp events conducted in June of 2021, 2022, and 2023. These events, facilitated by ten faculty members and attended by 68 fellows, generated valuable educational materials currently confined within proprietary learning management system environments. The objective of this project is to enable broader access and reuse of these resources by migrating them to a non-siloed, static website independent of the original Blackboard infrastructure. We describe our methodology for acquiring and validating the data exports, outline the auditing procedures implemented to ensure content completeness and integrity, and discuss the challenges encountered throughout the process. Finally, we report on the current status of this ongoing effort and its implications for future dissemination and reuse of educational materials.<\/p>\n","protected":false},"author":508,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[132],"tags":[],"class_list":["post-18462","post","type-post","status-publish","format-standard","hentry","category-issue61"],"_links":{"self":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18462","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/users\/508"}],"replies":[{"embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/comments?post=18462"}],"version-history":[{"count":0,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/posts\/18462\/revisions"}],"wp:attachment":[{"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/media?parent=18462"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/categories?post=18462"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/journal.code4lib.org\/wp-json\/wp\/v2\/tags?post=18462"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}},{"id":18510,"date":"2025-10-21T15:58:56","date_gmt":"2025-10-21T19:58:56","guid":{"rendered":"https:\/\/journal.code4lib.org\/?p=18510"},"modified":"2025-10-21T16:00:10","modified_gmt":"2025-10-21T20:00:10","slug":"extracting-a-large-corpus-from-the-internet-archive-a-case-study","status":"publish","type":"post","link":"https:\/\/journal.code4lib.org\/articles\/18510","title":{"rendered":"Extracting A Large Corpus from the Internet Archive, A Case Study"},"content":{"rendered":" By Eric C. Weig<\/p>\n Since 2015, The University of Kentucky Libraries have uploaded historic newspaper issues to the Internet Archive. We are now changing our newspaper program to host newspapers locally, and expanding the collection scope to include not only issues scanned from analog sources but also born digital and web harvested HTML (HyperText Markup Language) newspaper content.<\/p>\n Gathering local copies of content we placed in the Internet Archive would have been cumbersome as it would have involved accessing long term tape based archival storage systems, which meant time and effort from multiple staff. Alternatively, an automated process might be utilized to pull files back down from the Internet Archive and AI could play a role to make needed development faster. I had some experience with Python and anticipated that AI would be more successful crafting small, narrowly focused scripts than more complex code. [1<\/a>]. Curiosity played a role here as well. With so much interest growing around the use of AI, giving it a try on a small and well-defined coding project sounded like fun, with the option of abandoning it if results were not successful.<\/p>\n Initial research also uncovered that the Internet Archive encourages web crawling via its robots.txt file and offers access to an API to gather metadata and files. [7<\/a>] [4<\/a>]<\/p>\n The first step in the process involved assembling all the metadata needed to gather the Internet Archive objects from the Kentucky Digital Newspaper collection (see Figure 1).<\/p>\n At a minimal level, this requires the Internet Archive specific identifiers that were assigned to the objects. When looking at a URL for an object, the identifier is located at the very end of the URL, following the last forward slash. For example, the following object URL https:\/\/archive.org\/details\/xt74qr4nm47j has an object identifier that is \u2018xt74gr4nm47j\u2019. In the Internet Archive, various URLs relate to an object and follow specific patterns that reference the identifier value. For example, to download a .zip file containing all of the files associated with the object, including what was originally uploaded, the following URL pattern is used [5<\/a>]: https:\/\/archive.org\/compress\/xt74qr4nm47j.<\/em><\/p>\n Additional metadata was desirable to facilitate renaming and sorting the objects once they were downloaded as well as to calculate the total storage requirements for files prior to downloading. Specifically, four fields were retrieved: creator, identifier, date, and item_size. The \u2018creator\u2019 metadata values were useful in sorting downloaded content into directories organized by newspaper title. The \u2018identifier\u2019 metadata was used to construct the download URLs, and \u2018date\u2019 metadata was prepended to the downloaded .zip file names in order to add semantic information. The \u2018item_size\u2019 metadata was used to calculate how much local storage space we needed.<\/p>\n Another aspect of getting the metadata involved ensuring it was in a parsable format. Fortunately, the Internet Archive has several options for gathering metadata formatted as structured data. For collections with fewer than ten thousand items, a URL using \/scrape\/ will retrieve data (see Figure 2). [4<\/a>]<\/p>\n Figure 2.<\/strong> Sample Scrape API URL that gathers the first 10,000 metadata records the kentuckynewspapers collection.<\/p>\n Since our collection included far more than 10,000 items, this would not work for us. Instead, I used the advanced search web interface to search for what I needed, as the advanced search results page also provides the option to export results as structured data (see Figure 3). [6<\/a>]<\/p>\n I chose to use JSON as the output format (see Figure 4).<\/p>\n callback({“responseHeader”:{“status”:0,”QTime”:469,”params”:{“query”:”collection:kentuckynewspapers”,”qin”:”collection:”kentuckynewspapers””,”fields”:”creator,date,identifier,item_size”,”wt”:”json”,”sort”:”creatorSorter asc”,”rows”:1000,”json.wrf”:”callback”,”start”:0}},”response”:{“numFound”:86819,”start”:0,”docs”:[{“creator”:”Adair County news.(The)”,”date”:”1917-11-14T00:00:00Z”,”identifier”:”xt744j09xt7p”,”item_size”:13630085},{“creator”:”Adair county community voice”,”date”:”2012-10-18T00:00:00Z”,”identifier”:”kd99882j6j5s”,”item_size”:91957159},{“creator”:”Adair county community voice”,”date”:”2012-09-20T00:00:00Z”,”identifier”:”kd9tm71v5w8w”,”item_size”:93942368},{“creator”:”Adair county community voice”,”date”:”2012-10-04T00:00:00Z”,”identifier”:”kd9dz02z1d9w”,”item_size”:92310777},{“creator”:”Adair county community voice”,”date”:”2012-09-27T00:00:00Z”,”identifier”:”kd9jq0sq9446″,”item_size”:95768429},{“creator”:”Adair county community voice”,”date”:”2012-09-20T00:00:00Z”,”identifier”:”kd9pg1hh6t91″,”item_size”:93942985},{“creator”:”Adair county community voice”,”date”:”2012-09-13T00:00:00Z”,”identifier”:”kd9610vq304m”,”item_size”:95027685},{“creator”:”Adair progress (The)”,”date”:”2015-06-11T00:00:00Z”,”identifier”:”kd91n7xk8900″,”item_size”:118977672},{“creator”:”Adair progress (The)”,”date”:”2012-08-16T00:00:00Z”,”identifier”:”kd90z70v8f74″,”item_size”:83747017},{“creator”:”Adair progress (The)”,”date”:”2014-02-06T00:00:00Z”,”identifier”:”kd9x34mk6r4d”,”item_size”:89747837},{“creator”:”Adair progress (The)”,”date”:”2012-07-26T00:00:00Z”,”identifier”:”kd9d79571k58″,”item_size”:107745767},[\/text]<\/p>\n Figure 4.<\/strong> Sample JSON output generated from the Internet Archive Web Interface Advanced Search. Fields gathered included identifier, date, creator, and item_size.<\/p>\n A calculation of storage requirements was essential for planning so that an automated process could run without encountering insufficient storage space.<\/p>\n Calculating the storage space needed was relatively simple as all the information needed was represented in the JSON output, in the item_size metadata element..<\/p>\n As I approached the use of AI for small script code generation, articles on effective prompt engineering were helpful. I chose to use the free ChatGPT (GPT-4o mini) for its ease of use and established accuracy with Python coding: A 2024 study published on Python code generation and AI found that \u201cof all the models tested, GPT-4 exhibited the highest proficiency in code generation tasks, achieving a success rate of 86.23% on the small subset that was tested. GPT-based models performed the best compared to the other two models, Bard and Claude.\u201d [2<\/a>] My approach was to start very simple with prompts initially aimed at producing code to perform simple processing tasks, then branching out to create more sophisticated prompts that incorporated multiple processes. I also spent time experimenting with prompts and then reviewing the resulting code, using few-shot learning by providing example elements within the prompt while sometimes performing multiple iterations with different phrasing and level of granularity as well as striving to cover all necessary details with as few words as possible. See Figure 5 for the completed prompt used to generate code. [9<\/a>] [3<\/a>]<\/p>\n Write a Python script that runs from the commandline and processes a .json file at a full path specified in the code. The JSON file contains data wrapped in a JavaScript-style callback function, like callback({…}).<\/em> The script should extract the JSON content, parse it, and sum the values of the item_size field found in the list located at response.docs. The script should print and write to a log file listing the total size in bytes, KB, MB, GB, and TB. The log file path should be specified in the code. Include error handling for if the file is malformed and print a message indicating when the process is completed successfully.<\/span><\/p>\n Figure 5.<\/strong> Figure 5. show the ChatGPT (Chat Generative Pre-trained Transformer) prompt that generated storage.py.[14<\/a>] The italicized text indicates prompt content added after the initial prompt was conducted. The second prompt was then submitted in its entirety.<\/p>\n One quick fix I needed to make in the resulting code was to make sure that the file paths were set as raw strings in order to ignore escape character sequences. [10<\/a>] I made this change directly to the code after it was generated.<\/p>\n Now that I had all the URLs to download the objects, I began considering a responsible approach to retrieving the data. I intended to heed the guidelines established in the Internet Archive\u2019s robots.txt file. [2<\/a>] Incorporating a 10 to 30 second pause between each download seemed reasonable. [11<\/a>]<\/p>\n Initial Prompt:<\/strong><\/span> Refining Prompt:<\/strong><\/span> Refining Prompt:<\/strong><\/span> Figure 7.<\/strong> ChatGPT prompt and refining prompts that generated download_zips.py.<\/p>\n Lessons learned from earlier prompts led me to including the example callback function\u2014formatted like callback({…})\u2014 as a few-shot prompt so the script would parse the JSON effectively. Post code generation, I again set file paths as raw strings. Additionally, the prompt specified grouping downloads in manageable sets within sequentially ordered subdirectories and include validation for downloaded .zip files.<\/p>\n Tests were conducted utilizing small sections of the JSON file. One error that occurred during some downloads was that the Python script could not create a valid directory name. Values in the creator metadata contained characters not valid for directory naming. A quick manual adjustment to the Python code to sanitize the creator metadata, addressed this issue.<\/p>\n Figure 8.<\/strong> Manual fix to AI generated code to sanitize text for use in naming Windows file system directories.<\/p>\n Additional tests with more small subsections of the JSON file were successful and the script was then run in earnest (see Figure 9).<\/p>\n The download_zips.py script ran intermittently for nearly two months. There were three interruptions due to local machine reboots for Windows updates. There were 6,489 failed downloads due to temporary network connection issues and what appeared to be corrupt files.<\/p>\n Downloaded files are being reviewed one title at a time. Any items missing due to unsuccessful downloads are downloaded a second time. The Internet archive uses a file manifest that includes checksum values for all the files contained within the .zip archive, and this was useful for reviewing files and assessing whether or not they were corrupted.<\/p>\n Downloading our previously uploaded content from the Internet Archive via automation was straightforward and easy to accomplish with some careful planning. The Internet Archive is tailored to allow for harvesting content. URLs can be formulated using the Web interface advanced search options and these URLs gather results formatted as structured data such as JSON and XML. Utilizing AI to generate small Python scripts that were narrow in focus, ChatGPT yielded well-formed Python that accomplished development milestones, allowing for quick testing and only minor problems to fix via debugging or inputting refining prompts.<\/p>\n [\n
Notes<\/h2>\n
Introduction<\/h2>\n
Community Context<\/h2>\n
Strategic Partnership<\/h2>\n
Escalation of the Crawling and Mitigation<\/h2>\n
User agent and IP blocking<\/h3>\n
Request throttling<\/h3>\n
Facet blocking<\/h3>\n
Subnet-based blocks<\/h3>\n
Regional bans<\/h3>\n
Commercial Web Application Firewall<\/h3>\n
Commercial AI-driven bot detection<\/h3>\n
In-browser client verification (i.e., Cloudflare Turnstile)<\/h3>\n
Lessons Learned<\/h2>\n
Banning blocks of IP addresses is insufficient<\/h3>\n
Rule-based controls are insufficient<\/h3>\n
We are dramatically out-resourced<\/h3>\n
Scaling and performance optimization have limited impacts<\/h3>\n
\nThis experience has demonstrated that technical resource optimization alone, although helpful, cannot solve the problem of aggressive crawler traffic. Comprehensively addressing this issue requires fundamentally rethinking how we prioritize human access to our systems.<\/p>\nLibraries and archives may be uniquely vulnerable<\/h3>\n
Future Directions and Collaboration<\/h2>\n
Community knowledge sharing<\/h3>\n
\nThe ad hoc nature of our collective response highlights the need for more formalized knowledge-sharing mechanisms that can reach libraries of all sizes and technical capacities. While the Code4Lib Slack channel represents an important first step, expanding communication channels and developing shared repositories of code (of which there are several recent examples[13<\/a>][14<\/a>][15<\/a>][16<\/a>]) and documentation specifically designed for institutions with limited technical resources would strengthen our community’s collective resilience against these evolving threats.<\/p>\nProof-of-work checks<\/h3>\n
Regional service prioritization<\/h3>\n
Impact on web analytics<\/h3>\n
Conclusion<\/h2>\n
Acknowledgments<\/h2>\n
About the authors<\/h2>\n
References<\/h2>\n
Introduction<\/h2>\n
Related Work<\/h2>\n
![](\"\/media\/issue61\/choi\/figure1.png\")
\nFigure 1.<\/strong> The Bootcamp data was organized in Blackboard by-day in the leftmost navigation. Within each day, the respective instructors posted their contents for the fellows. The contents of this image have been masked to remove identifying information.\n<\/p>\nMethodology<\/h2>\n
\n
Data Extraction and Inventory<\/h3>\n
Traversing the Course Structure<\/h3>\n
Kaltura Caveat<\/h3>\n
Discussion and Future Work<\/h2>\n
Audit Redux<\/h3>\n
Presentation Medium<\/h3>\n
Evaluation of Our Approach cf. Web Archiving \/ Auto-crawling Tools<\/h3>\n
Conclusion<\/h2>\n
Acknowledgements<\/h2>\n
References<\/h2>\n
About the authors<\/h2>\n
Background<\/h2>\n
Gathering Metadata for Objects<\/h2>\n
![](\"\/media\/issue61\/weig\/figure_01.png\")
\nFigure 1.<\/strong> Collection landing page for Kentucky Digital Newspapers on the Internet Archive. Shows results for 86,819 items.<\/p>\nhttps:\/\/archive.org\/services\/search\/v1\/scrape?fields=identifier,item_size,creator,date&q=collection%3Akentuckynewspapers<\/pre>\n
![](\"\/media\/issue61\/weig\/figure_03.png\")
\nFigure 3.<\/strong> Internet Archive Web Interface Advanced Search GUI with field inputs used for generating the JSON formatted metadata output for all 86,819 items.<\/p>\nCalculating Storage Needs<\/h2>\n
![](\"\/media\/issue61\/weig\/figure_06.png\")
\nFigure 6.<\/strong> Command-line output of the storage.py script. Shows calculated storage size in bytes, KB, MB, GB and TB.<\/p>\nDownloading Objects<\/h2>\n
\nWrite a Python script that loads a JSON file located at a path specified in the code. The JSON file contains data wrapped in a JavaScript-style callback function, like callback({…}). The script should extract the JSON content, parse it, and read the values of identifier, creator, and date. for each identifier construct a download URL in the format https:\/\/archive.org\/compress\/{identifier}. Create a subdirectory under a base path specified in the code and named after the creator with characters that are invalid or problematic replaced with underscores and leading\/trailing whitespace stripped. Download the ZIP file corresponding to the identifier and save it as {date}_{identifier}.zip in the appropriate subdirectory. Wait for a random interval between 10 and 30 seconds before proceeding to the next download. If a download fails, log the failed URL to a log file specified in the code.<\/span><\/p>\n
\nAdd code that groups the downloads into sets of 1000 and creates subdirectories in the downloads directory that are named 0001, 0002, etc.<\/span><\/p>\n
\nAdd Python code so that after each .zip file is downloaded, it is verified using zipfile.is_zipfile() to ensure it is a valid ZIP archive. If is_zipfile() returns False, an error message should be recorded in a log file (e.g., download.log) indicating that the file is not a valid ZIP. If the file passes this check, it should then be opened with zipfile.ZipFile() and tested using the testzip() method. If testzip() returns a file name, indicating a corrupted file inside the archive, this should also be logged with an appropriate warning or error message<\/span><\/p>\nORIGINAL CODE:<\/strong> \r\n# Subdirectory under group: creator name save_dir = os.path.join(group_dir, creator) <\/span>\r\n\r\nUPDATED CODE:<\/strong> \r\n# Sanitize creator sanitized_creator = re.sub(r\u2019[^A-Za-z0-9]\u2019,\u2019_\u2019, creator) \r\n# Subdirectory under group: creator name save_dir = os.path.join(group_dir, sanitized_creator<\/span><\/pre>\n![](\"\/media\/issue61\/weig\/figure_09.png\")
\nFigure 9.<\/strong> Command-line output from the download_zips.py script.<\/p>\nResults<\/h2>\n
![](\"\/media\/issue61\/weig\/figure_10.png\")
\nFigure 10.<\/strong> Calculation of failure rate for downloads.<\/p>\n![](\"\/media\/issue61\/weig\/figure_11.png\")
\nFigure 11.<\/strong> An XML manifest file is present within each .zip download. This file includes checksum values for each file within the .zip archive.<\/p>\nConclusion<\/h2>\n
References<\/h2>\n