Categoria: File Management and Distribution

The log file zimsvlog.zim, located in the directory pointed to by the ZIM environment variable, records all events happening with Zim Server like, its start, its end, errors, tracing, and other information.

You can browse its contents using a text editor or using ZimAdmin.

After some time using ZimServer, this file can become very large. You should check the size of it from time to time, archive it (if you wish) and erase it. It can be erased at any time, even when Zim Server is running. Next time ZimServer needs to log some information, the file will be recreated.

Zim uses the operating system’s file management system to store entity sets, relationships, programs (source and compiled), database definitions, ZIM directories, and work files. By default, all files are stored in the same area of your file system. You can however, change the locations of the various types of files by, for example, moving the files to different disks, to improve the performance of your application. Zim provides methods for controlling the files that it uses.

The files that make up a Zim application are implemented in your operating environment’s file system. This section discusses how the various types of files are mapped onto that file system. This section looks only at the default organization. The default organization can be changed using Zim tools.

Database Directory

By default, Zim stores all database files, audit files, and document files in one operating system directory. This directory is the database directory; that is, the directory in which the ZIMINIT utility created your database.

When you start Zim Server, all databases that are going to be serviced are described in the zimdb.zim file.

When you start a Zim session, it assumes, unless told otherwise, that the default (i.e. current) operating system directory is the database directory and creates a working directory according the rules in the next item.

Working Directory

By default, Zim stores all working files in a sub-directory of the database directory, under a name that is formed using the number of the user connected to the database or under the name of the user that connected to the database.

The working directory can be created in a different place from the database directory by providing the configuration option “work directory” in the zimconfig.zim configuration file.

Another configuration option, “user name directory”, tells whether the working directory name must be a number or the name of the user.

WARNING: On UNIX systems, to properly allow users to log in and log off, the umask setting must be set correctly to allow working directory sharing by issuing the command:

umask 000

Relationship between Zim Directories and Operating System Directories

Many operating systems use the concept of the directory as a means of organizing the various components of a file system. An operating system directory contains the definition of file system components such as files and other directories.

Zim also uses the concept of directories as organizational tools. A Zim directory contains the definitions of objects in your database, such as EntitySets, forms, and other ZIM directories.

Zim directories are quite independent of operating system directories. In fact, a Zim directory is itself implemented as an operating system. For example, the Zim root directory is usually stored as a file called zim0001. For every Zim directory created, Zim also creates an operating system directory in which compiled Zim programs are stored. For example, the operating system directory that Zim creates for the Zim root directory is usually called zim0001.ws.

Special Document File Names

Within Zim, the file name associated with a Zim document is normally a valid operating system file name or device name. However, Zim also recognizes special file names, with specific file prefixes. For more information on these file prefixes and their meanings, see File Path Prefix Characters.

An application is composed of a number of different types of files including

• directory files
• entity sets and relationship files
• application program files
• compiled application program files

You can control where these files are located. Changing the location of files can be useful for

• increasing performance (locating files on the network nodes and servers where they are used most often)
• sharing files among applications

Distributing database files using the areas feature

By default, all database files are located in the database operating system directory.

The areas feature enables you to distribute the database files that correspond to entity sets, relationships, directories and compiled application programs in order to take advantage of the file system and possibly reducing the system overhead.

To use the areas feature, create an areas file named areas.zim.

To be effective, the areas file must reside in the database directory.

Using the Areas file

Entries in the areas file indicate the locations to which particular files were distributed. The areas file must contain one entry per record, and the entry must take the form

nnnn path

where

nnnn is the number taken from the file name (i.e. ZIMnnnn) of the file that corresponds to the desired directory, entity set or relationship.

path is the name of the directory where file ZIMnnnn is to be found. If ZIMnnnn is a file that stores a directory, then path is also the location where the operating system sub-directory for compiled application programs (ie ZIMnnnn.WS) is found.

Path can be preceded by three of the same special relocation prefixes used with document file names (see special document file names). The table below describes the meaning of the prefixes when used with path.

If a file lacks an entry in the areas file, the software assumes that the file is located in the database directory.

Distributing documents

Zim documents can be distributed throughout your file system by including an appropriate file name in the FileName field of the documents entity set in the object dictionary. The file name can be any file or device name that is valid in your operating environment.

Per-user documents

You can define documents that are stored on a per-user basis in the user’s work directory.

Per-user documents are declared in the Object Dictionary by placing the work path indicator (a right parenthesis) in front of the FileName entry in the Documents EntitySet. Per-user documents operate the same way as per-user database files. Identifying a document as per-user instructs the system to look for the file on the disk as indicated by the work path configuration option.

Distributing work files

The temporary work files used by the system are stored in the work directory. The work directory is set in the user configuration files by a work path entry.

Distributing Audit files

Audit files, which are normally created in the database directory, can be relocated by setting the audit path and backup path entries in the Zim Server configuration file.

Zim uses the operating environment’s file system to store data, programs, data dictionary definitions, configuration information, and work files. Use the ZIMFILES utility to obtain a complete list of database and document file names in your application.

The following files are created by or for Zim.

Each Zim database has its own object dictionary. All objects (i.e. entity sets, forms, variables, and so on) that are described in the object dictionary are created in a Zim directory. Normally, an application developer creates the Zim directories that are used within one particular database. A foreign directory, on the other hand, is a Zim directory that is defined by one database, but is used by other databases. By accessing a foreign directory, you gain access to all objects created in that directory.

Why Use a Foreign Directory?

There are many situations in which a foreign directory can be useful. One common example is the toolbox, a database containing application programs that are useful in, and therefore shared among, many different databases.

For example, while developing an application, you might have created a directory containing utility programs. These utilities could be useful in other situations, but you do not want to re-create them for each new database. By following the procedures described below, the utility directory can be defined as foreign in the other databases. The directory can then be used in databases other than the one in which it was created.

How Foreign Directories are Used

The procedure described below is an overview of how foreign directories are used. The remainder of this section describes this in more detail.

In general, the use of a directory from one database (the host database) within another database (the connecting database) involves five steps:

1. In the Object Dictionary of the host database, define the target directory, including a special directory ID number. Create the directory, using the CREATE command.
2. In the Object Dictionary of the connecting database, define the target directory again. This definition should be identical to that in the host database, except that the directory should be defined as foreign.
3. In the connecting database, create the directory, using the CREATE command.
4. At the operating system level, switch to the operating system directory that contains the connecting database. Use your text editor to edit the foreign directories file (called dirs.zim), adding an entry that gives the location of the foreign directory within the file system. You can also use the “Update Dirs.Zim” button on the Directories window in DC to edit the file directly.
5. Return to Zim and the connecting database. Use the ACCESS command to access the foreign directory.

Note: A connecting database can access many foreign directories simultaneously, provided each directory has a unique directory ID number (dirID). A single directory can be accessed as a foreign directory simultaneously by many databases.

Note2: If a single ZimServer manages different databases and at least one of them has foreign directories, all directories, local or foreign, must have unique dirID across all databases.

Defining a Foreign Dictionary – Detailed Description

A foreign directory consists of two components. The first component is the original directory created in the host database. It is this directory that is accessed by users of other databases. The second component is a directory that is created as foreign in the connecting database.

In the host database

If a particular directory is to be used from within other databases, the directory must be created in the host database, using the procedure described below. The directory is created essentially in the normal way; however, it must be assigned a special directory ID number.

1. Add a record to EntitySet Directories in the Object Dictionary. This record must set

• • DirName to the name of the directory
  • DirId to the directory ID number. This number should be between 1 and 240. The DirId should not conflict with the DirId of any other directory to which a user might connect at the same time.
  • Foreign to NO

2. Create the directory in the normal manner, using the CREATE command. If the foreign directories (dirs.zim) file belonging to the host database contains an entry that matches this directory’s DirId, then the operating system file that corresponds to this directory is relocated to that entry.

Note: The root directory (namely Zim) cannot be used as a foreign directory. You should ensure that the directory to be used in other databases is a sub-directory.

In the Connecting database

To enable users of another database to connect to an existing (foreign) directory, continue the above procedure with the following steps:

1. In the connecting database, add a record to entity set Directories in the Object Dictionary. This record must set

• • DirName to the name of the foreign directory
  • DirId to the directory ID of the foreign directory
  • Foreign to YES

2. Create the directory in the normal manner. Because the Foreign field in the Directories record has been set to Yes, no files are created for this new directory.

Note: The DirName and DirId of the foreign directory must be the same as the DirName and DirId in the host database.  

At the Operating System level

Before a user of the connecting database can employ the ACCESS command to connect to a foreign directory, the connecting database must contain a foreign directories (dirs.zim) file with an entry describing the location of each foreign directory. To create this file, use a text editor at the operating system level. The file should be stored in the operating system directory that contains the connecting database.

Each entry in the foreign directories file has the format:

dirid file# path

where

dirid is the dirid of the foreign directory

file# is the number of the file that corresponds to the directory in the host database (i.e. nnnn to zimnnnn). To determine the file#, run the ZIMFILES utility against the host database or use the command

output $filename (directory-name)

For example, the entry

11 126 usrtools

means that the directory whose dirid is 11 can be found in the file usrtoolszim0126.

Accessing a Foreign Directory

Once a foreign directory has been defined and created in the host and connecting databases, and the foreign directories file exists with the appropriate entry, the foreign directory is accessible from within the connecting database. To access the foreign directory, use the following procedure:

1. Invoke Zim in the connecting database.
2. Use the ACCESS command to access the foreign directory by name, as shown in the following example:

access Utilities read

All objects defined in the foreign directory are now available from within the connecting database.

Note: Foreign directories can be accessed only in read mode.

The Location of Files Associated with a Foreign Directory

The dirs.zim file specifies the location of the operating system (OS) file that corresponds to the foreign directory. But where does the system look for the files that correspond to the entity sets, relationships, programs and so on that are defined in the foreign directory?

By default, Zim looks for these files in the path specified in the dirs.zim file for the foreign directory file itself. It is possible, however, to distribute these files to other parts of the file system by using an areas file with special document file names in the host database. The procedure for using an areas file in a host database is exactly as described for independent databases in Distributing Database Files.

An areas file for a foreign directory can distribute some or all of the associated database files (i.e. EntitySets, relationships, compiled programs, and so on). Documents defined in a foreign directory can use the special file name prefixes – including the double quotation mark and the right parenthesis – to locate associated files in the database directory and work directory belonging to the connecting user.

Example

To illustrate, let’s look at a Utilities example. Consider the following background information:

• The directory Utilities in the host database is defined in the operating system directory D:UTILPROG. The directory ID for Utilities is 30.
• In directory Utilities, three documents are defined as shown below:

>list all docs

Docname        FileName       DirName

DictReport     dict.rep       Utilities

ReportOut      )report.out    Utilities

UserLog        “user.log      Utilities

• In directory Utilities, two entity sets are also defined as shown below:

UtilEnt1 (stored in operating system file ZIM0129)

UtilEnt2 (stored in operating system file ZIM0130)

• The host database has an associated file that is located, with directory Utilities, in operating system directory D:UTILPROG. The areas file contains the entry

129 C:UTZIM

• The connecting database has an associated foreign directories file that contains the entry

30 117 D:UTILPROG

• The connecting user’s work directory is C:APPLDIRUSER1.

Given this situation, the connecting and host databases both contain an areas file that relocates the files associated with entity sets. In the connecting database, a foreign directories file establishes D:UTILPROG as the default location for all files associated with objects in the foreign directory Utilities. The foreign object ReportOut (a document) is defined as being located in the connecting user’s work directory; similarly, UserLog is defined as being located in the connecting user’s database directory.

Accessing Foreign EntitySets

By connecting to a foreign directory, you gain access to the data in the EntitySets and relationships with fields that are defined in that directory.

To obtain read/write access to foreign database fields, you can declare those files as non-shared in the areas file associated with the host database. For example, to obtain read/write access to a foreign EntitySet called UtilEnt whose corresponding operating system file has the name ZIM0185, you would place the entry

185  D:UTILPROG       % This is a Zim on Windows example.

185 /usr3/utilprog/   % This is a Zim on UNIX example.

in the areas file associated with the host database.

In Zim 9, foreign databases can be read or written as any normal database.

Limitations on Foreign Directories

The following limitations apply to the user of foreign directories:

1. Each directory accessed by an application must have a directory ID unique among all directories being accessed at the same time. Directories, in this case, include sub-directories of foreign directories.
2. One database cannot connect to the root directory (Zim directory) of another database (i.e. the root directory cannot be a foreign directory).
3. A foreign directory cannot be updated using the COMPILE, UNCOMPILE, CREATE, ERASE, PERMISSION, or RENAME command. You cannot update the definition of objects defined in a foreign directory.
4. A connecting database cannot connect to foreign directories defined within the foreign directory of the host database.
5. Foreign database files that have been encrypted with an ENCRYPT command can be accessed from a connecting database only if the connecting and host databases were both initialized using the same encryption key. The encryption key is specified when the ZIMINIT utility is executed to initialize a new Object Dictionary.

File management can be adapted to the resources available in the operating environment. To manage files efficiently, it is important to know

• how to control the number of files that the system has open at any one time
• how to estimate file space
• how to control the growth of a file
• how to pre-allocate space in a file

Managing the number of open files

In many of the environments in which Zim runs, there is an operating system limit on the total number of files that can be opened at any one time (per task or for the entire operating system). In some cases, the definition of the file includes each use of a device, such as a terminal or a printer, for reading and writing. The operating system limit affects Zim’s use of its own directories, entity sets, relationships, documents, and compiled programs, and possibly, its use of the terminal, printers, and so on.

Zim manages its use of files in order to be able to function within the limitations imposed by the operating system. The management of line-oriented files is reasonably straightforward. For example, when you invoke a non-compiled application program, the current program file is closed before the new program file is opened. As a result, there is, at any one time, only one file being used for reading commands. Output is directed to two files (by the SET OUTPUT and SET TRACE OUTPUT commands). When a SET OUTPUT or a SET TRACE OUTPUT command is executed, the current output or trace output file is closed before the new file is opened. Each error causes the error file (containing templates for the Zim error messages) to be opened. The error file is closed after an error message is produced.

However, the main use of files involves block-oriented files such as directories, entity sets, relationships with fields, and compiled programs. Each of these objects has a corresponding operating system file. For these files, Zim maintains a pool of file control blocks. entity sets and relationships files are logically opened and closed around each command. At the end of a command, all these files are marked as no longer being in use; they are, however, left open as far as the operating system is concerned. During the execution of the next command, if a required file is already open, it is marked to show that the file is now in use. If the file is not open, then an unused file slot in the pool is sought. If a slot is not found, a file that is open, but not actually in use, is closed to free a file control block. The required file is then opened. In this way, any number of files can be used during a session while only a fixed number are actually open at one time.

The files configuration option determines the size of the file pool (i.e. the maximum number of block-oriented files that can be opened at any one time). Within operating environments that place a limit on this number, the file setting must be lower than the operating environment’s upper limit. The operating environment limit minus the files setting is the number of slots that are available for line-oriented files such as documents, terminals, and work files. For more information about the performance implications of the files configuration option, see Increasing Speed: Maximizing Memory Use.

Estimating File Sizes

When managing files, it is valuable to know the amount of space that an existing file currently occupies, and how much space a new file will occupy in the future. The method of estimating a file’s size depends on the use of the file. This section describes the techniques for estimating the size of the following types of files:

• EntitySets and relationships
• directories
• compiled programs

Entity Sets and Relationships

In Zim, an entity set and its related indices are stored in a single operating system file. Relationships with fields are stored in the same way. Every file in a database is organized into fixed-size pages; each page contains 1024 bytes.

Pages are the unit of transfer between disk and memory. Every page belongs either to the entity set (or relationship) or to some specific index on that entity set or relationship. Zim manages the data on each page. In particular, it tracks the free space available within partially filled pages and also tracks completely empty pages. The empty pages remain part of the file because the file systems that the program uses do not permit files to get smaller, only bigger. Thus, if you create a large entity set and delete all the members, you have lots of free space, but the file size is still large.

If a new page is needed, one of the empty pages is used, if available; otherwise, the size of the file is extended.

Entity-set records are packed into pages. A record can be split between pages, but excessive splitting is avoided. The size of a record in an entity set is determined by the formula

L + N + 5

where

L is the total length of the non-virtual fields in the record (virtual fields are not included in the calculation because their values are not stored in an EntitySet record)

N is the number of non-virtual fields in the record.

For char, alpha, and numeric fields, length is the length specified in the field definition. The size of int, longint, vastint, and date fields depends on the underlying machine. Usually, the sizes are 2, 4, 8, and 8 respectively. Some machines force the alignment of certain kinds of data. As a result, records can be somewhat longer than the above calculation indicates. Some RISC machines, for example, force all data types except char, alpha, and numeric to start on an even address or on a multiple of 4 or 8.

Each page contains a header of approximately 22 bytes, meaning that the number of records per page, ignoring splitting, is the greatest integer in

(1024 – 22) / (RL)

where

RL is the calculated length of a record in the entity set

These calculations are approximate and are further complicated by varchar or varalpha (variable length character) fields, which occupy an amount of space equal only to their actual length, plus two bytes to store the length itself.

Using your knowledge of the average size of each variable length field, you can reasonably estimate the number of pages to be occupied by the data in an entity set or relationship. For example, consider an entity set composed of fields shown in the following table:

In this case, the length of each record in the EntitySet is

(12 + 2 + 4 + 8 + 6 + 8) + 5 + 6 = 51

The most records that can be stored in one 1024-byte page is

(1024 – 22) / 51 = 19

For an entity set containing one thousand records, the data would require 53 pages (i.e. 53Kb bytes).

Index space is somewhat harder to estimate accurately. Accuracy is difficult because Zim uses a sophisticated BXtree algorithm that tries to keep the BX tree as balanced as possible and to keep pages as full as possible. This strategy optimizes performance. The actual result is heavily dependent on the data and its physical order.

For an indexed field, including virtual fields, each non-null field value is stored as a key in the index. The maximum number of keys that can be stored on a single page is approximately

(1024 – 12) / LIF + 10)

where

LIF is the length of the indexed field

Note: The length of a key for a variable length field is always its maximum length.

Assuming that all pages are completely full, you can calculate that the minimum number of pages used by an index is the smallest integer greater than

TR/ MK

where

TR is the total number of records in the entity set

MK is the calculated maximum number of keys per page

As previously noted, the index algorithms in Zim merge partially-filled pages in order to keep pages as full as possible. The actual utilization of blocks depends on the distribution of key values and on the pattern of adding and deleting. Typical utilization range from 50 percent to 80 percent.

If there is an index on field Fld1 from the preceding example, then the maximum number of keys per page is

(1024-12) / (12+10) = 46

For an entity set containing one thousand records, at 70 percent utilization, this index requires approximately

(1000/46) * (100/70) = 32 pages

The total size of a file includes the space used for entity set records, the space used for each index, any completely empty pages created by deletions, and one control page.

Directories

Directory files are also block-oriented. Zim directories contain information about every object defined in your application, including entity sets, relationships, fields, roles, variables, virtual fields, directories, named sets, constants, windows, menus, menu items, form fields, and displays. The amount of information maintained for each object varies. For example, an entity set is described by its name, the file number, and links to the information about its fields. Relationships require more space, primarily to store the encoded relationship condition. Information about an object is separated into basic information and descriptor information.

The number of pages occupied by a directory file can be estimated by the following formula:

3 + N/B + N/D

where

N is the number of objects in the directory

B is the number of objects whose basic information can be packed into a single page

D is the number of objects whose descriptor information can be packed into a single page

B is approximately 20 and D is approximately 10.

If you have chosen to store cross-reference information, that information is also kept in the directory file. Additional space is required for cross-reference information.

Compiled Programs

Files containing compiled application programs are also block-oriented. The amount of compiled code varies enormously from one source command to another. For example, the command

let V = 1

compiles a rather small amount of code that assigns the value 1 to the variable V. On the other hand, consider

add ent1 from form1

The compiled code for this instruction must assign values from the fields in form1 to the fields with the same name in ent1.

If these objects had twenty-five objects in common, the compiled code for this ADD command would be at least twenty-five times the size of that produced for the preceding LET command. This comparison is indicative of the expressive power of Zim. Unfortunately, the variability in compiled output makes it virtually impossible to estimate the size of compiled programs.

Controlling Grown Characteristics of Database Files

In most operating systems, files that grow frequently in small increments can become fragmented; each file is stored in many pieces throughout the disk. As a file becomes more heavily fragmented, access to the file becomes very inefficient, as increased disk head movement is required to locate all the pieces. Fragmentation can be reduced by forcing files to grow less frequently but in greater increments.

Overcoming Fragmentation

When all pages of a file have been filled with data, Zim Server normally extends the length of the file by 10% of the current size. This type of growth somewhat minimizes the size of the database file, but it can also reduce system performance if the file becomes extremely fragmented.

Many operating systems store files in numerous separate fragments. These fragments are often called extents. As a file grows, new extents can be created, increasing the fragmentation of the file. The average time that it takes to access the file can increase as the number of extents increases. Under some operating systems, this problem can be remedied by copying the file to another location; copying alone can reduce the number of fragments.

If the file is known to be growing, the data extend configuration options can be used to pre-allocate file space in the database, thereby controlling file growth and reducing fragmentation.