* File System

I had another one of those merger ideas. I would like to create a merger of ReiserFS, BitTorrent, Coda, OCFS and git to create a distributed, clustered, versioned, atomic file system.

In ReiserFS, writes to the file system are done atomically like a database, so that it is impossible to leave the file system in an inconsistent state. Each one of these atomic changes can be considered like a commit in git. So, if the full transaction history is available, then I could seek back in time to any previous version of a file. git replicates commits between machines via its push and pull commands and tries to merge them. Two commits that are unrelated have no problems merging. So, combined, a file system where the merging of their logs could allow file systems on different physical disks be synchronised even if both are active.

Of course, that isn't very useful on a single computer. This is where the BitTorrent enters the picture. Each file system can be considered a peer that knows a globally unique identifier for the file system. Most likely a URL, it could identify a tracker that helps peers find each other. As a peer makes changes to file system, it announces that it has new commits and the new commits are replicated by its peers. If a peer comes on-line, it can review its log and determine the last commit it received and then start updating its replica (where newly minted commits have higher priority than helping old replicas fast-forward). A particularly smart peer would also put commits for blocks that applications are requesting on higher priority.

Coda has the nice feature of disconnected operation and git has a similar concept. If a client cannot make contact with others, that shouldn't necessarily prevent access to the file system. When writing, the commits are simply stored locally. Once connected to others, the file system can begin pulling changes. Any commits that do not create conflicts can be pushed out and any that do must be resolved by the user on his system first. A rate-limited replication would also be useful for clients over slow links. A laptop that is normally connected to its peers over a high-speed Ethernet can detect that it is on a distant Internet connection and try to replicate only high-priority commits with capped speed.

A laptop would pose some significant problems on two levels. The first is the size of its disk. If a group of servers held a file system in the terabyte range, replicating it to laptops and even desktop would have insufficient space to hold a replica. Rather than move to a pure client model like Coda's, extended attributes could be used to assign wants to certain directory trees on a per peer basis. A tree with a want of 1 would be always replicated to a peer's disk and a tree with a want of 0 would never be replicated. Each user would then make sure that on their computer, their home directory had a want of 1 and, perhaps, a shared documents directory a want of 0.5 and the other user directories have a want of 0. Their system will then only bother to replicate information they care about into a more limited space. A potential problem arises if all replicates set the want of a certain tree to 0. It would effectively evaporate from the file system yet still be present when all disks fill with more desirable files. A smart peer would also cache relative wants based on access patterns. So, one of two files with equivalent want should be preferred to be stored if it is accessed more often and/or recently.

The second significant problem with laptops is trust. If an organisation like a university put all the user's home directories into the file system, administrators control the machines on campus and establish trust relationships, but students wanting to connect from home would cause a nightmare. The replication should obey the permissions of the files in the file system. When the tracker is set up, public keys for the trusted clients could be created with root-level access. As a student, I should be able to use my user account when I access the tracker to get a limited key that is capable of replicating only data I would have permission to read. This way, even a hacked version of the software would be prevented from extracting data from the file system that the operating system would deny him from reading if he were on a peer with root-level access. Hacked peers would only have the opportunity to steal data from other hacked peers.

Disk space might become a concern in the servers. Obviously, having a master server connected to large amount of redundant disk designed to keep a copy of everything is going to be costly. It would be desirable to have redundancy, but that would require not only another server but another disk array. A clustering feature should allow to peers to share physical disk over a SAN. Each server would be independent, but aware that the other is using the same disk. In normal operation, they will effectively load balance for each other and if one fails, the other should transparently fail-over. Interestingly, other peers can be entirely unaware of this. There would simply be two peers that are equally up-to-date and, when one goes away, it is equivalent to any peer crashing, and, when it comes back, it is a new peer that is suddenly up-to-date. This is where the OCFS enters the picture.

Okay, some one build it.