Drop #628 (2025-03-26): Web-Slinging Wensday

hmmfetch; headrs; ESM Or Bust

Yep. I snuck another R package into a Drop. And, yes, I will continue to advocate for dropping the completely useless ‘de’ in ‘Wednesday’.

A caveat for the first two sections: please do not abuse websites. Ars just dropped “Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries”, and even our global sensor network catches AI crawlers (so they are literally just hoovering up any web content at even any rando IP address). Header randomization — especially in conjunction with residential proxy servers — is more often used for evil than good, but there are legitimate uses for this techniue, too.

Type your email…

Subscribe

TL;DR

(This is an AI-generated summary of today’s Drop using Ollama + llama 3.2 and a custom prompt.)

• hmmfetch: A JavaScript package that wraps fetch() to add realistic, randomized HTTP headers, mimicking browser behavior to avoid detection as automated traffic (https://github.com/willswire/hmmfetch/tree/main)
• headrs: An R package inspired by hmmfetch, offering similar functionality for generating realistic HTTP headers and integrating with R’s {httr} and {httr2} libraries (https://codeberg.org/hrbrmstr/headrs)
• ESM Or Bust: Anthony Fu advocates for transitioning to ESM-only packages, discussing the growth in ESM adoption and the benefits of moving away from dual CJS/ESM formats (https://antfu.me/posts/move-on-to-esm-only)

hmmfetch

hmmfetch is a lightweight JavaScript package that wraps the standard fetch() function but automatically attaches realistic, randomized HTTP headers to each request. These headers mimic those that a browser like Chrome, Firefox, or Safari would typically send, including User-Agent, Accept-Language, and others. We can use hmmfetch the same way we’d use the native fetch, but also have the option to override or specify certain values, such as browser type (chrome, firefox, etc.), operating system (windows, mac, linux), or language preferences. There’s also a generateHeaders() function for generating these headers separately, without sending a request.

The purpose of randomizing headers is to make outbound HTTP requests look more like they’re coming from a human using a browser, rather than from a script or bot. This can help avoid detection or throttling from services that flag non-browser traffic. Many sites inspect headers to detect automation—for example, looking for missing or uniform User-Agent strings, or unrealistic Accept and Accept-Language combinations. By rotating plausible header sets, hmmfetch can blend in with typical browser behavior, which is useful for scraping, testing, or research where a lower profile is helpful.

headrs

I had jankier R snippets that I’ve used to do most of what hmmfetch does, but decided to riff from it and make a {headrs} R package (knot). It offers the same basic functionality: generating realistic HTTP headers that mimic browser behavior to avoid detection when making requests. Where hmmfetch wraps JavaScript’s native fetch(), {headrs} integrates with R’s httr and httr2 HTTP client libraries, providing functions that both generate browser-like headers (generate_headers()) and use them to perform GET requests (hmmfetch() for {httr}, hmmfetch2() for {httr2}).

Like hmmfetch, this package randomizes headers like User-Agent, Accept-Language, and the sec-* family of headers to resemble those sent by real browsers—Chrome, Firefox, Safari, or Edge—on various operating systems including Windows, macOS, and Linux. It includes accurate user-agent strings and browser-specific headers such as sec-ch-ua or sec-ch-ua-platform. You can lock to specific browsers or OSes, or let it choose randomly. The randomness is meant to reduce the chance of requests being flagged as automated.

str(  generate_headers())## List of 12##  $ Accept                   : chr "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/si"| __truncated__##  $ Accept-Language          : chr "ko-KR,ko;q=0.9"##  $ Cache-Control            : chr "max-age=0"##  $ Sec-Fetch-Dest           : chr "document"##  $ Sec-Fetch-Mode           : chr "navigate"##  $ Sec-Fetch-Site           : chr "none"##  $ Sec-Fetch-User           : chr "?1"##  $ Upgrade-Insecure-Requests: chr "1"##  $ User-Agent               : chr "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.1823.58"##  $ sec-ch-ua                : chr "\"Chromium\";v=\"117\", \"Not:A-Brand\";v=\"24\", \"Microsoft Edge\";v=\"117\""##  $ sec-ch-ua-platform       : chr "\"Linux\""##  $ sec-ch-ua-mobile         : chr "?0"

hmmfetch(  "https://httpbin.org/headers",  options = list(    headers = list("X-Custom-Header" = "custom-value")  )) |>   httr::content(    as = "text",    encoding = "UTF-8"  ) |>   writeLines()## {##   "headers": {##     "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", ##     "Accept-Encoding": "deflate, gzip", ##     "Accept-Language": "en-GB,en;q=0.9", ##     "Cache-Control": "max-age=0", ##     "Host": "httpbin.org", ##     "Sec-Fetch-Dest": "document", ##     "Sec-Fetch-Mode": "navigate", ##     "Sec-Fetch-Site": "none", ##     "Sec-Fetch-User": "?1", ##     "Upgrade-Insecure-Requests": "1", ##     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15", ##     "X-Amzn-Trace-Id": "Root=1-67e3c625-1717b8685fa2fe456aa4f7aa", ##     "X-Custom-Header": "custom-value"##   }## }

hmmfetch2(  "https://httpbin.org/headers",  options = list(    headers = list("X-Custom-Header" = "custom-value")  )) |>   httr2::resp_raw()## HTTP/1.1 200 OK## date: Wed, 26 Mar 2025 09:17:25 GMT## content-type: application/json## content-length: 926## server: gunicorn/19.9.0## access-control-allow-origin: *## access-control-allow-credentials: true## ## {##   "headers": {##     "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", ##     "Accept-Encoding": "deflate, gzip", ##     "Accept-Language": "en-US,en;q=0.9", ##     "Cache-Control": "no-cache", ##     "Host": "httpbin.org", ##     "Sec-Ch-Ua": "\"Chromium\";v=\"122\", \"Not:A-Brand\";v=\"24\", \"Microsoft Edge\";v=\"122\"", ##     "Sec-Ch-Ua-Mobile": "?0", ##     "Sec-Ch-Ua-Platform": "\"macOS\"", ##     "Sec-Fetch-Dest": "document", ##     "Sec-Fetch-Mode": "navigate", ##     "Sec-Fetch-Site": "none", ##     "Sec-Fetch-User": "?1", ##     "Upgrade-Insecure-Requests": "1", ##     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.1823.58", ##     "X-Amzn-Trace-Id": "Root=1-67e3c625-6678ec4b17bb39e0017d2112", ##     "X-Custom-Header": "custom-value"##   }## }

I’ll likely be updating this as I use it and discover where it is too brittle. PRs are most welcome, as well.

ESM Or Bust

In “Move on to ESM-only”, Anthony Fu discusses the evolution of JavaScript module systems and advocates for transitioning to ESM-only packages, and is also a reflection on the author’s changing perspective since writing about dual CJS/ESM formats three years prior.

Fu notes that ESM adoption has grown significantly, with ESM packages on npm increasing from 7.8% in 2021 to 25.8% by the end of 2024. This growth has been supported by modern tools like Vite, which treats ESM as a first-class citizen, and testing libraries like Vitest that were designed for ESM from the beginning. CLI tools such as tsx and jiti have also simplified the development process by enabling seamless execution of TypeScript and ESM code.

Fu highlights two approaches to ESM adoption: bottom-up, exemplified by Sindre Sorhus migrating low-level packages to ESM-only in 2021, and top-down, where high-level frameworks and tools lead the transition. Fu argues that the top-down approach is more effective for smooth adoption, as it’s easier for ESM packages to depend on CJS packages than vice versa.

A significant milestone in ESM adoption is Node.js’s ability to require() ESM modules, a feature recently unflagged and backported to Node.js v22. This capability allows ESM-only packages to be consumed by CJS codebases with minimal modifications, enabling what Fu calls a “middle-out” approach to migration.

Fu discusses several challenges with maintaining dual CJS/ESM formats, including interop issues between the different module systems, dependency resolution complications, and increased package size. These issues make a compelling case for transitioning to ESM-only.

The article provides guidance on when to move to ESM-only, recommending it for new packages, browser-targeted packages, standalone CLI tools, and packages targeting evergreen Node.js versions. Fu emphasizes the importance of understanding consumers’ requirements before making the transition.

To help track ESM adoption, Fu introduces the Node Modules Inspector, a visualization tool for analyzing package dependencies and identifying potential migration issues. The article concludes with Fu’s plan to gradually transition his maintained packages to ESM-only and his hope for a more portable, resilient, and optimized JavaScript/TypeScript ecosystem.

Anthony is def worth a Bsky follow and RSS feed pin, too.

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:

• Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev
• Bluesky via https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy

Also, refer to:

• this post, and
• this post

to see how to access a regularly updated database of all the Drops with extracted links, and full-text search capability.