Академический Документы
Профессиональный Документы
Культура Документы
Imagine, if you can, a network file system so realiable that the clients keep working even if the server falls over. Imagine using it at home without dialling up, or connecting once a day for a minute to syncronise with the server. That's Coda; not vapourware, but something that exists now. I'll discuss the Coda logo later.
Understanding Coda
Should I use it? How does it work? How do I set it up? How do I manage it? After reading this, I hope you can answer the question `Should I use Coda?' And to help you answer it, I'll try to explain what it does and how it works, what's involved in setting it up, and what's involved in keeping it going. I'll begin with an overview of Coda, continue with a deeper look at certain points, and then give a summary and an evaluation of Coda and its current state of development and usefulness.
Kinds of machine
Clients File and Data Servers System Control Machine (SCM) Backup machines A cell contains several kinds of machine, including, of course, the clients. A client is any machine which can see a cell's Coda file system. In addition to clients, there is another kind of machine which we may call a file and data server, since Coda deals with a lot of data besides just files. One of the file and data servers plays a special role in a cell: the System Control Machine, or SCM. It is the master server, and needs to be installed first. And if you plan to backup your Coda system, you will need a backup machine. A single machine may perform all four functions, so that you can have a cell with only one machine in it. Servers do not need to be clients in a Coda cell, nor do backup machines. But you will need more than one file and data server to take advantage of Coda's best features. The organisation of a Coda cell has some implications for security: you need to protect the integrity of your file and data servers. In general, a hacker who breaks into a Coda client cannot compromise the security of the servers. Of course, if he lurks undetected, with root access, the blackguard may learn something compromising from another user of the cell.
Volumes
Virtual partition Mounted globally Quota Cloning Different kinds of volume The building block of a Coda file system is the volume. A volume is something like a virtual partition. It may exist physically on one server, or it may have several identical copies on two or more servers. It must be mounted cell-wide if it is to be read or written, but it can be backed up without being mounted. Quota is managed by setting a limit on each volume. A volume may have no quota set, in which case it can grow until one of the physical partitions where it lives fills up. Coda does something to volumes called cloning, which we need to look at more closely. Also there are several different kinds of volume.
Cloning
Like fork: copy on write Occupies same space (initially) Like a collection of hard links One copy is read-only Helps backup Used to replicate (copy) volumes Cloning does to a volume something like what fork does to a Unix process. The original volume and the clone share the same disk space, which in the clone is read-only, and in the original volume is copy-on-write. The original and its clone are rather like two identical collections of hard links to the same files. When you change a file in the volume, the link to the read-only copy of that file is broken, and a new file is created. So the clone is like a backup of a volume at the time it was taken. It requires very little disk space unless the volume changes substantially. The clone of a volume can itself serve as a limited backup of the volume. In one of our cells, we clone all user volumes every night, and if you accidentally delete or mangle a file, you can read or copy the file from the clone without bothering the system administrator. When you backup a Coda system, you don't need to stop using it; instead, you clone each volume to be backed up, and you dump the clone to tape, or to a staging disk from which you can do something else with it. Cloning was used to produce read-only replicated volumes, but Coda seems to be dropping support for these.
Types of volume
Simple (read-write) Read-only (e.g., backup) Replicated (read-only) Replicated (read-write) (Coda only) We have already seen two types of volume, a simple read-write volume, and a read-only volume such as a clone. When you create a full dump of a Coda volume, you can restore this as a readonly volume. Read-only volumes are also useful because you can make identical copies on several servers, and access to these read-only replicas may be much more efficient than access to a readwrite volume. A replicated volume is fully usable as long as at least one running server has a copy of it; this may improve the availability of the data. (Coda seems to be dropping support for readonly replicas. It seems that a slight gain in efficiency is offset by the increased complexity of the code.) Besides volumes of these kinds, which all Coda's cousins support, Coda also has read-write replicas. These are convenient because they give the advantage of high availability, though writing to them may be less efficient than writing to simple read-write volumes. Coda classes some volumes as read-write replicas even though they exist on only one server. Read-write replicas, whether they live on one server or on several, are the normal kind of volume in Coda cells, and you should plan your cell structure with this in mind. Replication is something like having a highly configurable, flexible RAID system in software. It is true that there is some management overhead, but we find that hardware RAID systems can be down for a long time if the controller breaks, which has happened to us several times now.
Disconnected operation
Mobile computing Accidental disconnection High availability Deliberate disconnection Hoarding (selection and priority) Reconnecting Fixing conflicts The idea of disconnected operation is central to Coda, and was part of its design from the beginning; the earliest Coda papers speak of `mobile computing'. A client may lose contact with the cell's servers accidentally, whether through network failure or because servers go down, or deliberately, because you are removing the client, or because networking over a phone line is too expensive to do continuously. Besides files that may happen to be in the cache, Coda lets you select portions of the file system which the cache manager tries (if possible) to `hoard'; that is, to keep current. You can set priorities to guide the cache manager in this. Reconnecting is easy if you haven't changed anything. If you have, if you created or edited files while the system was disconnected, you need to tell the system to reintegrate your changes. Of course, it is possible while your system is disconnected, for two people to change the same file or directory independently. In that case, Coda provides a mechanism for comparing the different versions, harmonising them, and resolving the conflict.
Coda Logo Artist Gaich Muramatsu The Japanese artist Gaich Muramatsu has created several entertaining drawings about Coda, which illustrate its various features. The Coda logo derives from a pair of pictures which contrast AFS users, who are unhappy when they lose their network connections, and Coda users, who continue to work without a problem in similar circumstances.
Supported Platforms
Linux (all platforms?) FreeBSD NetBSD (all platforms?) Solaris (sparc and i386) Windows 95/98 NT (Coda server only) Coda runs under Linux on all platforms, though I suspect it has been well tested only on the i386 and perhaps the sparc Linux. It was developed originally on BSD systems based on the Mach microkernel, so it has been ported to FreeBSD and NetBSD; in the latter case again I suspect it hasn't been tested as thoroughly on some of the less common platforms. Recently Solaris ports were announced, for the sparc and i386 platforms; these are fairly new, and not as well-used as the others. Coda can be compiled in the CygWin environment from Cygnus, and the resulting client and server does work under Windows 95 and 98, though there are known to be some problems with this. The Coda server works under NT as well, though the Coda client does not.
Features
I'd like to look next at some of Coda's features, particularly those in which it differs significantly from NFS. These cause some slight limitations, which you need to know if you plan to use Coda.
Access Control
R Read files in this directory L Attach to and list this directory I Insert a file into this directory D Delete a file from this directory K Lock files in this directory (not Coda) W Write or append to files in this directory A Change ACLs on this directory These are the seven access permissions which may appear in an ACL. R or read access lets you read files in a directory. L, list, or lookup access lets you attach to a directory and list its contents. Coda does not support `hidden' directories, which a user can access without being able to do a directory listing. None of the other permissions actually works unless you have L access to the directory as well. I or insert access, and D or delete access let you insert or delete files in a directory respectively, or create or delete a subdirectory respectively, or mount or unmount a volume there. The K or lock access is intended to control the ability to lock files in a directory, but this has no effect in a Coda system. W or write access lets you write or append to files in a directory, and A or admin access lets you change a directory's Access Control List.
Mode Bits
Ignored for directories For files, group and other mode bits ignored R mode bit must be set to read a file R and X bits must be set to execute a file W bit must be set to write to or delete a file Suid and sgid may not do what you expect In Coda, the Unix mode bits on directories are ignored. On files, the group and other mode bits are ignored; only the user bits have any effect. To read a file you need read and lookup access to the directory through its ACLs, and the file needs its R mode bit set. (While other Unix file systems allow you to read files in `hidden' directories if you know their names, Coda does not.) To execute a file, you need read and lookup ACLs, and you need both R and X mode bits set. (While some other Unix file systems allow you to execute a file which you cannot read, Coda does not.) And you cannot write or delete a file unless its W bit is set. Coda remembers the Coda user ID of a file, the ID of the Coda user who created it. But the owner and group of a file have no influence on who can access it. The Coda system administrator can make an executable file in Coda SUID, in which case the command gets the effective Unix user ID of the file owner. Note that this does not give the program the access which that Coda user has; it does not give a running SUID program an effective Coda ID. This may cause problems if you want cron jobs to write into Coda. All files and directories in Coda appear to have the Unix group `nogroup', so SGID programs are not useful.
Warnings
Flush() does not send data to fileservers: you must close() file before data gets written. Coda is not suitable for many log files. Commands like 'tail -f' may not work. When a process on one machine is writing, it may not be visible on another machine. Besides the above-mentioned problems related to ACLs and mode bits, you should be aware that a Coda client does not send data to fileservers when a running program calls flush(). You have to close() a file before it gets sent to the fileserver, and even then it doesn't happen if you are disconnected. After disconnected operation, you can't reintegrate without authenticating first. For this reason, Coda may not be a good place to write log files. Commands like 'tail -f' don't work unless you are on the same client machine. A file being written by a process on one machine cannot be read on another machine until the writing process closes the file.
More Warnings
Databases should not have data files in Coda. Data may be lost if the writing process fails. Processes on other machines may not see data. But this warning applies only to databases managed by daemons which leave them open. For this reason, Coda is not in general a good place to store databases. If the writing process aborts, or the client reboots, data that has been written may be lost. If you intend to share data across machines, you may have problems when a file is open. Note that this applies only to data base files which get left open by their managers; Berkeley DB-style databases are usually OK, as are others which get closed after being changed.
Looking Deeper
I hope by this time you have some idea of what Coda can do and how it works. Now I'd like to look a little more deeply into system administration and disconnected operation.
Internal Data
Files: /vicepa Data about volumes (kind, where, backed up) Data about protection Data for encryption (shared keys) Data about backup Data about cell Recoverable virtual memory Coda file and data servers store several kinds of data. The data which appears on clients as files, directories, and so on is stored on the file server, of course, but not in this format. Each Coda file server has at least one physical partition, usually named '/vicepa', '/vicepb', and so forth, where some of the contents of the Coda file system are stored in a binary format. There is approximately one file per file, but they are arranged in a hash tree. The file servers store information about the cell's volumes (what kind they are, where they are, and so on), the mapping between Coda users and groups and their ID numbers, the list of groups to which each user belongs, the list of members for each group, the shared secrets which allow servers to communicate with each other, information about backup (whether each volume has been or will be backed up, when, so that you can do incremental backups); information about the cell (which machine is the master SCM server, what other servers there are, and so on). Data on its way to its final home in the vice partitions gets stored in 'recoverable virtual memory' on that file server, to protect its integrity. Normally RVM requires two raw partitions on each file server, in addition to the partition(s) used for /vicepa, (/vicepb,...). That is, if you plan to make a machine a Coda file server, you should have at least three physical partitions to be devoted to Coda: two raw partitions to be used by RVM, and one file system mounted on /vicepa to contain Coda files.
Servers
codasrv: the workhorse updateclnt and updatesrv rpc2portmap kauth2 Now let's look at the processes that run on a Coda server. A file server always has a process called `codasrv', which is the workhorse: it serves files to Coda clients on other machines, and it keeps the vice partitions upto-date. Processes called `updateclnt' and `updatesrv' transfer between the various servers in a Coda cell the other data which Coda keeps. This data is kept in one directory, /vice/db, on each server. Every 30 seconds these processes check to see whether the data is the same on all servers, and if not, they copy it across. This may be very inefficient if you have many volumes or many users, at least if this data changes frequently. The rpc2portmap daemon listens for incoming requests for Coda services and passes them on. kauth2 receives requests for Coda tokens, and passes them out when the client has a valid Kerberos ticket.
Server Security
If you are a hacker who breaks into a Coda client, you can't access protected data in Coda without a Coda token. You can, of course, read the contents of the local cache if you have root access. You can't change Coda files unless they are world writable; that is, unless the Coda group System:AnyUser has write and lookup access to that directory. On the other hand, a hacker who breaks into a server can run pdbtool to create users, delete data, fake a Coda token, and do a great deal of damage. We may say that Coda affords some protection to the clients and the data, but you must take pains to guard the Coda server against attack.
Backing Up
backup program backup.sh script Requires physical backup partition well documented (with exceptions) configured by dumplist file: 7F000002 IIIFIII coda.proj 7F000003 IFIIIII coda.user Coda comes with a backup program `backup' and a shell script `backup.sh' which runs it. Together they read a list of volumes to be backed up, and do this (if possible). Backup clones the volume to be backed up, then dumps the clone to a file. It supports both full and incremental backups, and is designed to be run daily or weekly. There is a script `backup.sh' is designed to be run by cron. It is the script used at CMU, and the installation does not attempt to edit it for your cell. The backup program must backup volumes to a physical partition, since it is intended to be used with some utility like 'dump' to copy the whole partition to tape. The backup procedure is fairly well documented, though the documents do not say that you must customise backup.sh to make it work in your cell. Here are a couple of lines from a sample dumplist file: 7F000002 IIIFIII coda.proj 7F000003 IFIIIII coda.user These mean that the volume with groupID 7F000002, named coda.proj, should have a full dump on Wednesdays and incremental dumps every other day, while the other volume has its full dump on Mondays. Dumplist is updated automatically when you create a read-write replicated volume; otherwise you need to edit it by hand.
Hoard command
hoard -f <filename> hoard clear hoard walk hoard list hoard off hoard on hoard delete <filename> The hoard command configures the way the cache manager keeps files and directories in the cache. The command hoard -f reads a hoard file and sends the commands in it to the cache manager. Venus remembers these commands and checks the cache every 10 minutes to see that these files are there. Note that you cannot tell hoard to save files or directories which you have no right to access; you must have a Coda token to run most hoard commands. `clear' tells the cache manager to forget its hoard list. `walk' forces venus to check the cache now and add files and directories if necessary. `list' prints a list of the cached objects; it reassures me that hoarding is going on. There are also `off' and `on' commands which let you switch the automatic hoard monitoring function off and back on without actually clearing the hoard list, and a `delete' command which removes a file or directory from the hoard list.
Authenticating
kclog kclog -tofile tokenfile kclog -fromfile tokenfile ctokens cunlog By `authenticating' I mean proving to Coda that I am the person who is allowed to do certain things in the file system. The Coda authentication program is `kclog'. Before you run it, you need the Kerberos tickets for a Coda user. On its own, `kclog' or `kclog [username]' checks that your Kerberos tickets are valid, and if so, it gives you a Coda token for a limited period. With the arguments `-tofile [filename]', it also writes the token to a file, so that you can use it in disconnected mode. With the arguments `-fromfile [filename]' it reads and sets the Coda token from a file which you created earlier, without trying to verify your Kerberos credentials. The command `ctokens' lists your Coda tokens, and the command `cunlog' destroys them. The default time limit for Coda tokens is 25 hours. I find this uselessly short for disconnected operation, but the only way to get a longer token is to edit the source of the kauth2 daemon and recompile it. I think there should be some way to specify the lifetime of a token. Getting a very long token, one that lasts weeks or months, is something of a security risk, as is saving it to a file. On the other hand, it is difficult to see how this can be avoided. As far as I can tell, Coda associates the user's Coda token with the Unix user ID which requested it, which is not the safest policy. It seems to me that Coda needs some idea like the Program Authentication Group (or PAG) of AFS, which lets you control this a little better.
venus.init stop There are two ways to shut down a client safely. One is to use the command `vutil shutdown', then `umount /coda'. The other is to use the supplied script `venus.init' with the `stop' argument. If you don't shut down a client properly, you risk losing data.
Reintegrate
Start venus (venus &) Get Coda token. cfs fr /coda/myfiles/mountpoint Resolve any conflicts. To reintegrate after disconnected operation, start venus as usual, and get a Coda token, since without a token, you won't be allowed to write changes to the file server. Give the command cfs fr /coda/myfiles/mountpoint (which means, Coda file system, force reintegration on this volume) specifying the mount point of each volume you wish to reintegrate. (This is a minor nuisance if you forget that you have changed some other volume.) If conflicts are reported, you need to resolve them as soon as you can. Conflicts are handled by a process called `repairing', and there are several commands that help you do this. When you begin a repair, the volume is locked, and the inconsistent files are `spread out' as distinct objects in the file system. You can then edit the files and harmonise them before ending the repair and reintegrating the file system again.
Security
Use Kerberos 5. Guard your file servers. Not for highly sensitive data possible security holes because most of Coda has not been stressed Now I'd like to make some points about security. I think you should in any case use Kerberos 5 with Coda, because the default internal authentication is so poor. You need to guard your file servers, since they are by far the most vulnerable point in the cell. And there are potential security holes because most of Coda has not been the subject of serious security testing. Despite this, I suspect that Coda may be more secure than NFS, and certainly not substantially less secure than AFS or DFS, bearing in mind its inadequate encryption. The Coda developers are well aware of the limitations of Coda's encryption, and intend to replace it with something more serious.
Installing Coda
Plan carefully; have one or two trials Many scripts may need to be edited. Join the mailing list, or find a friendly expert. Beware prepackaged binaries. Read through documentation and look at the mailing list archives. It's easier to install Coda when you have a little experience in doing so, but that's true of a lot of software. Begin by doing a one-server cell with one additional client. This will give you the feel of the software and remind you of some critical issues. Next try a small, real cell with two or three other users. To learn the backup system, get it working, then do a full backup and two incrementals on successive days, then delete the whole cell and restore it. This will be very painful. You will probably need to edit many of the scripts that come with Coda; I had to change seven of them for my use, but then I don't like to install every executable in /usr/bin. If you use Coda, you ought to join the Coda mailing list, or at least find an expert to help you over the rough spots. The mailing list is very friendly and (within reasonable bounds) very tolerant of newcomers. If I say `beware of prepackaged binaries', I don't mean that you shouldn't use them, but that they may make choices which you ought to change. I suspect most people will find a prepackaged binary is the best way to try Coda for the first time, but I don't think any of them should be used in a production cell. It is worth reading through the main documentation carefully. Also look at recent archives of the mailing list; the problems you face have probably bothered someone else, and the solutions probably exist already.
Documentation
9 manuals Coda Administration and User Manual Notes and Explanation about Coda Security man pages The Coda documentation, which incidentally is not included with the Coda source, contains 9 manuals. The most useful of these is the Coda Administration and User Manual, though you should also read the Notes and Explanations about Coda Security. The Administration and User Manual contains man pages, though they are (I think) in HTML or PostScript format rather than in troff source format. (Someone has put out a collection of plain man pages, but, the last time I looked, they were a different version from those in the Admin manual.)
(2) Disconnected operation Until recently, Coda's documents were seriously out of date. They didn't cover new commands and new options which had been added to the system in the past two years. They referred to and described at length several procedures which depended essentially on commands which no longer exist. And sometimes they were completely misleading. Since I gave the first version of this talk, this has improved substantially; a new version of the Admin manual has been produced which corrects many of these faults. For example, the Admin manual in several places recommended that you create a replicated read-only root volume for your cell, and gave details of how to do this. The procedure didn't work, and is not supported in the current version of Coda. Disconnected operation involves running `hoard walk'; this appears in the manual, but not in the man page for hoard, which is quite incomplete. Saving Coda tokens for disconnected operation does not appear in the documentation, nor are there instructions for starting venus in disconnected mode. The cfs subcommand `fr' or `forcereintegrate' does not appear in the documentation. In short, the documentation has flaws and omissions; it still needs some work.
Summary
To sum up, I shall give my evaluation of some of Coda's functionality, and suggest work I think it is ready to do, though with some cautions and qualifications. I shall also suggest possible future developments.
Core Functions
File serving Mobile operation Resilience High availability The main thing Coda does is to serve files. The Coda files works extremely well under stress and does not seem to fail. Coda is resilient, whether the network fails, the file server goes down, or you do something unexpected. It is reliable. If you are looking for high availability, Coda is certainly a serious option.
Attractive Applications
Web server Shared files: in, e.g., Beowulf cluster or across an enterprise especially where NFS is inadequate mobile/disconnected operation There are some applications which Coda is particularly suited to support. One of those is web serving. If you need to share web pages across several servers, as in a Linux Virtual Server cluster, or if you need extra security, Coda may be just what you need, and its caching could have been designed with web serving in mind. For the same reasons Coda's cousin AFS got its lifetime extended and found new customers. Coda may be a good choice wherever you need to share files across several machines, such as in a Beowulf cluster, or across a department or a larger enterprise; this is especially the case where NFS is not an appropriate solution because of its poor functionality and limited performance. Coda was designed with mobile, disconnected operation in mind, It works well with slow connections, it allows you to connect briefly and update your hoard, then to disconnect and work for hours, then to reconnect again briefly and reintegrate again.
User Issues
Few users = few problems especially for system administrators There is not much for users to learn Beware of risky acts by users on clients. The difficulties of users of Coda will depend on how many they are, and on how complex their usage is. If you have many users, especially if they want their own groups, if their file store has to be backed up, and so on, it would be quite a headache for the system administrator. Because Coda's file semantics differ slightly from those of plain Unix file systems, you might think users need to learn a lot before they are turned loose on the system. But this is not so: beginners do not need to learn everything, unless perhaps they want to use disconnected operation. The number of utilities which will have problems because of Coda's nonclassical-Unix semantics is surprisingly small.
Stored Data
Not too much hard to administer hard to restore from backups No sensitive data limited security, limited stress The problem of adapting to large scale usage affects the amount of data as well as the number of users. Coda's protection and volume databases will not work well for very large cells in their present form. But the larger problem lies in administering, backing up, and restoring all that data, and not in Coda's internal functions. Also, I have said that Coda is not suited to holding really sensitive data, because of its limited security at present; that is, because its internal encryption routines are crippled, and because things may remain in a client's cache for a long time. Moreover, Coda's constituent parts have not had a serious security review, or been exposed yet to systematic attack or exploitation attempts. On the other hand, you should probably not serve secure data using NFS or any other shared file system. Of course, you can always use encrypted loopback on Coda files on a Linux box.
Needs Work
Documentation Better administration Remove dependence on SCM Coda needs a lot of work. The documents need to be brought up to date and expanded, especially to explain things clearly to ordinary users.