What ever happened to chunkfs?
Jul 02, 2009, 12:02 (1 Talkback[s])
(Other stories by Valerie Aurora)
Full Text Search: The Key to Better Natural Language Queries for NoSQL in Node.js
"Before we describe chunkfs, we should first talk about the
problems that led to its creation. Chunkfs was motivated by the
growing difficulty of reading and processing all the data necessary
to check an entire file system. As the capacity of storage grows,
the capacity of the rest of the system to read and check that data
is not keeping pace with that growth. As a result, the time to
check a "normal" file system grows, rather than staying constant as
it would if the memory, bandwidth, seek time, etc. of the system
grew in proportion to its storage capacity. These differential
rates of growth in hardware - like the differences between RAM
capacity, bandwidth, and latency - are part of what keeps systems
"To understand the change in time to check and repair a file
system, a useful analogy is to compare the growth of my library
(yes, some people still own books) with the growth of my ability to
read, organize, and find the books in the library. As a child, my
books fit all on one shelf. I could find the book I wanted in a
matter of seconds and read every single one of my books in a week.
As an adult, my books take up several bookshelves (in addition to
collecting on any flat surface). I can read many times faster than
I could when I was a kid, and organize my books better, but finding
a book can take several minutes, and reading them all would take me
a year or more. If I was trying to find a twenty dollar bill I left
in a book, it would take me several hours to riffle through all the
books to find it. Even though I am a better reader now, my library
grew faster than my ability to read or search my library.
Similarly, computers are faster than ever, but storage capacity
grew even faster."