Top Document: comp.arch.storage FAQ 2/2 Previous Document: [7.7] Microsoft Windows NT {Brief} Next Document: [7.9] Non-Unix Large File Systems See reader questions & answers on this topic! - Help others by sharing your knowledge From: File Systems There is now an industry group working on standardizing an API for files larger than 2 GB (the max size normally supported on most Unix systems). More info as I get it. The WWW-enabled can have a look at http://www.sas.com:80/standards/large.file and see the various proposals on the table. Note that it is VERY easy to confuse whether an OS supports _files_ larger than 2 GB or _file systems_ larger than 2 GB. My table lists some of both (thanks to ben@rex.uokhsc.edu (Benjamin Z. Goldsteen), Ed Hamrick (EdHamrick@aol.com) and Peter Poorman (poorman@convex.com) for much of this information). It is straightforward for systems with 64-bit integers to support 64-bit files; for systems with 32-bit integers it is more complex. On most 32-bit systems the offsets passed around inside the kernel (most importantly, at the VFS layer) the file offsets and sizes tend to be passed as 32-bit (signed) integers, meaning no files >2^31. On most systems, the argument to lseek is of type off_t, which (on SunOS and Linux, and plausibly on OSF/1 and others) is declared in a header file as "typedef long off_t;". For clients to really have access to large files, three pieces are required: local FS support, an appropriate network protocol, and server support for 64-bit FSes. For FTP access, I believe _literally_ inifinitely large files are possible, but I'm not sure(SHMO). For NFS access, NFS V2 supports only 2GB files. NFS V3, just becoming available now, supports full 64-bit files, I believe (anybody have a reference to the docs? RFC? SHMO). With the notable exception of Unitree (which does not use, depend on, or appear as, a local FS on the server), server support for 64-bit files is provided only when the server's own local FSes are 64-bit. Even for the systems that _do_ support large files, not all are programmer or user-transparent for supporting large files. UniCOS is, OSF/1 is, ConvexOS is not (there are two system calls, lseek and lseek64, with 32-bit and 64-bit file offsets, respectively, though the Fortran interface is transparent). This brings up the related issues. A complete large files implementation needs not only the system calls, but also the stdio library and the runtime libraries for the languages (Fortran, Cobol,...). Further, system utilities (sed, dd, etcetera) need to be capable of dealing with large files. (It has been pointed out that the GNU C compiler runs on most of these machines, so it is possible to use "long long" as a 64-bit int on them, but what matters for file systems is the system compiler.) Here's the start of a table on these. Really such a simple table can't do the problem justice, but it'll give you an idea. Keep in mind that many of these systems support many file system types; I've listed only the most interesting so far from this point of view. I'd like to flesh it out more completely, though. 1 GB = 2^30, 1 TB = 2^40, 1 PB = 2^50, 1 EB = 2^60 NYR = Not Yet Released OS/hardware 64-bit C max max NFS info datatype par- file V3 updated tition size sup size (bytes) UniCOS (Cray vector) int, long ? 8 EB? ? 8/94 ConvexOS long long 1 TB 1 TB N 9/94 Alpha AXP OSF/1 V3.0 AFS long 128 GB 16 TB 8/94 9/94 Paragon OSF/1 ? 8 EB* 8 EB N 2/95 UTS (Amdahl) ? ? 8 EB? ? 8/94 HP/UX 9 (HP 9xxx) ? 4 GB ? ? 8/94 Silicon Graphics IRIX 5.2 EFS long long 8 GB 2 GB N 9/94 IRIX 6.0 EFS long 8 GB 2 GB N 9/94 (NYR) IRIX 5.3 XFS ?long long? ? ?TB? Y 9/94 (NYR) AIX (IBM RS/6000) 4.1 JFS long long 64 GB 2 GB N 8/94 Solaris 2.x (Sun Sparc) long long 1 TB 2 GB (soon?) 9/94 BSD 4.4 long long ? 8 EB? ? 8/94 Linux long long 1 TB 2 GB ? 9/94 DG/UX 5.4 long long 2 TB 2 GB ? 9/94 Alliant Concentrix long long ?>2 GB ?>2 GB N 9/94 (dead) * The Paragon PFS (Parallel File System), as I understand it, parallelizes access to the files; each partition striped across is limited to 2GB, so really the max partition size is 2GB * # of disks that can be attached. A slightly more detailed description of certain implementations is available with the WWW version. In addition, the HPSS (see above) supports large files, as does Unitree (though the Unitree interface to them is limited). User Contributions:Top Document: comp.arch.storage FAQ 2/2 Previous Document: [7.7] Microsoft Windows NT {Brief} Next Document: [7.9] Non-Unix Large File Systems Part1 - Part2 - Single Page [ Usenet FAQs | Web FAQs | Documents | RFC Index ] Send corrections/additions to the FAQ Maintainer: rdv@alumni.caltech.edu (Rodney D. Van Meter)
Last Update March 27 2014 @ 02:11 PM
|
Comment about this article, ask questions, or add new information about this topic: