MC Press Online

V4R4 APIs Expand AS/400 IFS File Sizes to 256 GB

Contributed by Atul Kumar Sunday, 31 January 1999 Last Updated Sunday, 31 January 1999

256 GB

As applications become more data-intensive, the need to store and manipulate large amounts of information grows far beyond the current 2 GB limit imposed by existing interfaces. With existing 32-bit interfaces, sizes, offsets, and lengths are represented as signed 32-bit integers and are limited to the maximum values that can be represented in a signed integer, e.g., no file can be larger than 2 GB for a variable defined as a signed integer.

In OS/400 V4R4, the AS/400 Integrated File System (AS/400 IFS) provides 64-bit versions of its UNIX-type APIs. These 64-bit APIs allow applications to store and manipulate files that are much larger than 2 GB. With the new 64-bit interfaces, sizes, offsets, and lengths are represented as 64-bit integers, which eliminates the 2 GB limit. AS/400 IFS users can now store and manipulate large files up to 256 GB in the Root (/), QOpenSys, and user-defined file systems by using the new 64-bit versions of the OS/400 UNIX-type APIs.

How to Use Large File Support

AS/400 large file support is easy to use. It enables applications that use existing 32- bit AS/400 IFS interfaces to continue to run without recompilation or code changes if they do not intend to use AS/400 large file support or the new 64-bit AS/400 IFS interfaces. Applications that use large file support can either have the existing 32-bit interfaces mapped to their new 64-bit version counterparts, or you can code directly to the new 64-bit interfaces. The following compile-time flags are provided for applications to use the AS/400 IFS 64-bit or large file support:

_LARGE_FILES allows automatic mapping of 32-bit interfaces and data types to their 64-bit versions.

Applications compiled using the _LARGE_FILES flag will have their invocations of 32-bit interfaces mapped to the 64-bit versions and all the appropriate data types and data structures will also be mapped to 64-bit versions. For example, open() will be mapped to open64() and off_t will be defined as type long long (originally defined as type int).

_LARGE_FILE_API allows applications to access 64-bit interfaces and data types directly. Applications compiled using the _LARGE_FILE_API flag can code directly

to 64-bit interfaces and use 64-bit data types and data structures. For example, applications can use open64() and use off64_t (defined as type long long).

http://www.mcpressonline.com

Powered by Joomla!

Generated: 28 August, 2008, 04:49

MC Press Online

Application Protection Mechanism

Under this structure, applications are prevented from inadvertently using large files without using large file support (i.e., 64-bit APIs with the appropriate compile flag). Applications will need to open a large file with the open64() API. An attempt to open a large file with the open() API will fail unless the application is compiled with the _LARGE_FILES flag. Alternatively, open() API with O_LARGEFILE in oflag can also be used to open a large file.

If the application intends to access large offsets or sizes on file descriptor-based operations, e.g., lseek(), ftruncate(), then the file must be opened for large file access and the 64-bit version of these APIs must be used.

New Data Types and Structures

With 64-bit enablement, a number of data types and data structures are redefined, as shown in Figure 1. The standard definition for those data types will be mapped to their 64- bit definitions when the applications are compiled with the LARGE_FILES flag. The 64-bit definitions are available when the applications are compiled with the LARGE_FILE_API flag.

Large File API Support

Similarly, a number of new 64-bit AS/400 IFS API interfaces are provided to allow applications to create and manipulate large files in the AS/400 IFS. The existing 32-bit AS/400 IFS interfaces are mapped to 64-bit interfaces when user applications are compiled with _LARGE_FILES defined. And, like the data types and structures, the 64-bit interfaces are only available to applications when they are compiled with the _LARGE_FILE_API flag. Figure 2 presents the standard 32-bit API interfaces and their 64-bit versions.

Application Programmer Hints, Examples, and Warnings

The following are strategies that your programmers can use when considering how to use large file support on the AS/400 IFS once you migrate to OS/400 V4R4:

Ignore large file support in the AS/400 IFS. Your applications will run as they did before large file support was introduced. However, if an application attempts to open a file that is larger than 2 GB-1, the open() operation will fail. Any attempt to open a large file without using the open64() API will fail with an errno value of EOVERFLOW (3525). Similarly, an attempt to read or write beyond 2 GB-1 when the starting offset is greater than 2 GB-2 will fail unless the file is opened in a large file access mode. Read operations fail with an errno value of EOVERFLOW (3525) when the starting offset is greater or equal to 2 GB-1. The write() API will also fail with an errno value of EFBIG (3507) in this situation.

If you do not want to use large file support, but you want to be able to open large files (without going beyond the limit of 2 GB-1), then define _LARGE_FILE_API in the source code and open the file using open64(). This solution gives you access to 64-bit data types and interfaces, but it does not map standard definitions to their 64-bit version. Be aware that, since you have opened the file as a large file, read() and write() will not fail when you go beyond 2 GB-1 with a starting offset greater than 2 GB-2. However, other APIs may fail when you go beyond the 2 GB-1 limit. For example, an lseek operation to seek beyond 2 GB-1 will fail with errno value of EOVERFLOW. Similarly, an fstat() operation will fail if the size of the file is greater than 2 GB-1 with errno value of EOVERFLOW.
http://www.mcpressonline.com Powered by Joomla! Generated: 28 August, 2008, 04:49

MC Press Online

This technique enables you to mix and match the use of standard and 64-bit definitions. However, applications programmers need to be very careful when mixing and matching. For example, you could open a large file using open64() and use the lseek() API to seek within the 2 GB-1 boundary. An attempt to seek beyond 2 GB-1 would fail unless lseek64() is used. However, read() and write() would be enabled to go beyond 2 GB-1 and an application may inadvertently read or write data that is beyond its reach.

If you want to use large file support without changing your code, then define _LARGE_FILES in the source code and recompile. All the standard definitions will be mapped to their 64-bit version. However, you need to be careful because certain APIs can now return offsets and sizes that may not fit in integer data types. For example, after being mapped to stat64, using the stat() API on a large file will return a size that will not fit into an integer data type. So, if you are storing offsets and sizes in local variables, you may need to change those local variables.

If you want to use large file support with code changes, define _LARGE_FILE_API in the source code, change all the standard definitions to their 64-bit version, and recompile. From a code clarity standpoint, this solution is preferred.

Large File Support without the Pain

Large file support in the AS/400 IFS is similar to the large file support provided on AIX. The large file support is provided in such a manner that existing applications working with regular files will not be impacted. The large file support on the AS/400 is very easy to use, and two compile time flags are provided for applications to choose the way they want to use the large file. In addition, the following OS/400 V4R4 commands are also enabled to work with or display large stream files stored in the AS/400 IFS:

Work with Object Links (WRKLNK) Display Object Links (DSPLNK) COPY Object (CPY) Move Object (MOV) Copy to Stream File (CPYTOSTMF) Copy from Stream File (CPYFRMSTMF) Save Object (SAV) Restore Object (RST) Display Diskette (DSPDKT) Display Save File (DSPSAVF) Display Tape (DSPTAP) With OS/400 V4R4, IBM is anticipating the needs of a growing set of key applications that rely on the AS/400 IFS for their file storage. By using 64-bit interfaces, the AS/400 IFS allows applications to use and support very large files so that businesses can grow their applications beyond the 2 GB barrier.

Related Reading

AS/400 Unix-Type APIs V4R4 (SC41-5801-03)

http://www.mcpressonline.com

Powered by Joomla!

Generated: 28 August, 2008, 04:49

MC Press Online

Standard Definition 64-bit Definition

off_t off64_t struct flock struct flock64 off_t l_start; off64_t l_start; off_t l_len; off64_t l_len; F_SETLK, F_SETLKW, F_GETLK F_SETLK64, F_SETLKW64, F_GETLK64 struct stat struct stat64 off_t st_size; off64_t st_size; unsigned long st_allocsize; unsigned long long st_allocsize; struct statvfs struct stavfs64

_Bin8 f_blocks; unsigned long long f_blocks; _Bin8 f_bfree; unsigned long long f_bfree; _Bin8 f_bavail; unsigned long long f_bavail;

Figure 1: These are the redefined data types and data structures (along with their 32-bit counterparts) youll need for 64-bit enablement.

Existing Interfaces 64-bit Definition

open() open64() creat() creat64() fstat() fstat64() lstat() lstat64() stat() stat64() ftruncate() ftruncate64() lseek() lseek64() statvfs() statvfs64() fstatvfs() fstatvfs64()

note: Qp0lGetAttr() and DosSetFileLock() APIs are also enabled for large file access.

Figure 2: These are the 64-bit implementations of the standard 32-bit APIs.

http://www.mcpressonline.com

Powered by Joomla!

Generated: 28 August, 2008, 04:49