mstdn.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A general-purpose Mastodon server with a 500 character limit. All languages are welcome.

Administered by:

Server stats:

8.8K
active users

#billionfilefs

0 posts0 participants0 posts today
James Young<p>Surely someone's looked into this: if I wanted to store millions or billions of files on a filesystem, I wouldn't store them in one single subdirectory / folder. I'd split them up into nested folders, so each folder held, say, 100 or 1000 or n files or folders. What's the optimum n for filesystems, for performance or space? <br>I've idly pondered how to experimentally gather some crude statistics, but it feels like I'm just forgetting to search some obvious keywords. <br><a href="https://mefi.social/tags/BillionFileFS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BillionFileFS</span></a> <a href="https://mefi.social/tags/linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>linux</span></a> <a href="https://mefi.social/tags/filesystems" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>filesystems</span></a> <a href="https://mefi.social/tags/optimization" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>optimization</span></a> <a href="https://mefi.social/tags/benchmarking" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>benchmarking</span></a></p>
James Young<p>I retried this with XFS recently, and I got it working, albeit slowly, after repeated crashes. <a href="https://mefi.social/tags/BillionFileFS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BillionFileFS</span></a></p>
James Young<p>Ooh, someone else compared filesystems on how well they managed having a billion files - not just the empty files I tested, and on more filesystems: EXT4, XFS, BtrFS, F2FS, ZFS <a href="https://arxiv.org/html/2408.01805v1" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">arxiv.org/html/2408.01805v1</span><span class="invisible"></span></a> <a href="https://mefi.social/tags/BillionFileFS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BillionFileFS</span></a></p>
James Young<p>More fun with way too many files: archiving and compressing. So, much, compression.</p><p><a href="https://pronoiac.org/misc/2024/09/file-systems-with-a-billion-files-archiving-and-compression/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">pronoiac.org/misc/2024/09/file</span><span class="invisible">-systems-with-a-billion-files-archiving-and-compression/</span></a></p><p><a href="https://mefi.social/tags/BillionFileFS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BillionFileFS</span></a></p>
James Young<p>The "easier blog post" I wrote about: I managed to populate a file system with a billion empty files, in under seven hours, a new personal best. <a href="https://pronoiac.org/misc/2024/06/file-systems-with-a-billion-files-making-forests-parallel-multitouch/" rel="nofollow noopener" target="_blank"><span class="invisible">https://</span><span class="ellipsis">pronoiac.org/misc/2024/06/file</span><span class="invisible">-systems-with-a-billion-files-making-forests-parallel-multitouch/</span></a> <a href="https://mefi.social/tags/BillionFileFS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BillionFileFS</span></a></p>