7/9/13

Database Engine Enhancements

Database Engine Enhancements

Andy Neil Micromega Systems, Inc

Exploring Data Access

Visual FoxPros database engine is loaded with new features. Rooted in the core of the product, these improvements provide the framework for building radically different FoxPro applications. At the same time, VFP clearly reflects its ancestry: an award-winning line of fast, flexible PC database management systems. The new engine will support everything that worked in FoxPro 2.x, while extending the range of development possibilities in powerful ways. Best of all, it lets you build your applications faster, with less code to write and results that are easier to maintain. The goal of this session is to introduce the many improved capabilities of the database engine. Well take a tour of the major landmarks in the new territory, stopping to observe some spectacular vistas and the occasional rock in the road. Well cover tools and commands that can serve as starting points for your own explorations. Well also discuss some implications of all this new technology for how we think about and build business applications.

The Guiding Vision

To understand the significance of the new enhancements, its helpful to consider some of the design goals of the Visual FoxPro development team: preserve flat-file (.DBF) data storage capabilities while adding relational (multi-table database) functionality make it easier to ensure the relational and logical integrity of data reduce the amount of time and code required to provide basic application services like transactions and multiuser access. make it easier to write client-server applications preserve FoxPros reputation for performance dont force anyone to use the new features

Topics to be Covered
Here, in brief, is how the development from these design goals has manifested as features in the new database engine. For a more thorough look, youll have to attend the session!

Database Container (DBC)

In Visual FoxPro, the central hub for data access is the Database Container (DBC). The DBC is a repository for information about the various stores for the applications data, along with information and code to provide checks to preserve the integrity of the data. In this usage, database means something very different than its traditional Xbase meaning. DBFs are now referred to as tables while databases are collections of tables. The DBC doesnt actually contain DBFs physically, instead it contains records with pointers to the DBFs. The DBC serves as the exclusive gateway for accessing them, preventing direct use of any included DBFs. (DBFs which dont belong to any DBC can be used just as in FoxPro 2.x.) DBCs may also contain views, which are virtual tables created via a SQL SELECT statement from local tables or server (ODBC) data sources. This part of the session will cover how to build, use, and maintain a DBC, and how to take advantage of its many features. Well also discuss how the structure of a DBC can be extended to provide custom data dictionary functionality.
www.dfpug.de/loseblattsammlung/migration/whitepapers/DBEngine.htm 1/5

7/9/13

Database Engine Enhancements

Indexes and relationships

One of the advantages of the more relational DBC model as compared to the earlier flat-file (DBF-only) model is that new varieties of information can be maintained within the data store. In particular, its very useful to be able to store descriptions of how tables are related. To support the establishment of these persistent relationships among the DBCs tables, two new types of index keys have been added. A primary index never allows the entry of duplicate values for the key expression. Unlike the unique indexes of earlier versions, which just indexed the first instance of a value, a primary index ensures that only one record may have a specific key value. Each table may have only one primary index, but it is possible to define additional indexes which must have genuinely unique key values as well. Such a non-primary unique index is called a candidate index. Primary and candidate indexes are created via the CREATE/ALTER TABLE syntax (which is SQL-92 compliant, for those who care). Once appropriate indexes have been created, it is possible to define a persistent relationship between (the indexes of) two tables. Persistent relationships can be oneto-one or one-to-many. Persistent relationships are used as the default join condition in the Query and View Designer, and can serve as the basis for enforcing referential integrity using triggers.

Integrity checks
Ensuring the physical and logical integrity of data is a vital job for any database system. In addition, proposed changes to data need to be validated for compliance with any applicable business rules. In the past, the bulk of this work has been performed in code embedded in the application (or applications) which use the data. Now, the DBC allows you to store data validation and referential integrity (i.e., deleting detail records when the master record is deleted) code in the data repository itself, where its automatically used by any application that tries to make changes to data. Visual FoxPros Referential Integrity Builder can even generate the stored code for you! Field rules serve as built-in VALID clauses, while row rules play a similar role in checking the record as a whole before any changes are committed. Triggers are procedures which are executed any time a record is inserted, deleted, or updated. Among other uses, triggers are tremendously useful in implementing referential integrity. By allowing developers to store rules, triggers, and stored procedures in the DBC, Visual FoxPro stores the data-specific code where it makes the most sense architecturallywith the data. This makes maintaining the code easier and ensures that its consistently used by all applications which access the data. By integrating the hooks for this code into the engine, we should realize performance benefits. For client-server applications, the ability to define rules in the local DBC relieves the server CPU from needing to perform the rule-enforcement work, speeding processing even more.

Buffering and transactions

To meet the goal of reducing the amount of effort and code required to perform common application tasks, Visual FoxPro implements an array of new features in the area of multiuser data access. These tools take over a lot of the mechanics of making sure that one users changes dont clobber anothers, and that changes get written to the data source only when its appropriate. Of course, if you want to do it the hard way, all of the techniques were used to from FoxPro 2.x will still work. When a table or view is opened (via the USE command), Visual FoxPro creates a cursor to carry information about how updates to the data should be handled. One of the properties of this cursor is called Buffering. Buffering establishes a regime of record locking and edit caching to provide a safe space where the user can make data changes without having to worry about collisions with other users. With the appropriate buffering settings, multiuser applications can be built without SCATTER/GATHER and manual record locking. This means typical business applications (for instance, a master-detail update form) can be built with much less hassle. Record buffering works on a record-at-a-time basis, flushing changes to the local data source when moving between records. Table buffering is similar, but works on multiple rows in the cursor at the same time (particularly useful with grid controls). For both types of buffering, changes can be explicitly committed or discarded using the TABLEUPDATE() and TABLEREVERT() commands. Either type of buffering can be set to use optimistic locking (take a lock only at the instant of writing the changes) or pessimistic locking (lock the record as soon as editing begins and keep it locked until changes are completed). When working with buffered cursors, applications have access to not just the buffered value of the field (ALIAS.fieldname), but also the original value via the OLDVAL() function, and the current value in the underlying data source via the CURVAL() function. Transactions make it possible to define batches of data changes which can then be committed as
www.dfpug.de/loseblattsammlung/migration/whitepapers/DBEngine.htm 2/5

7/9/13

Database Engine Enhancements

a block, or rolled back if some portion of the changes fails an integrity check or collides with someone elses changes. Transactions are particularly useful in cases where changes to multiple tables must all be made in synch to preserve the logical relationship among the records. The combination of buffering and transactions, with appropriate error handling, provides the basis for a fundamentally different approach to coding multiuser applications than was possible in FoxPro 2.x. In the session, well walk through some source code that illustrates some of the new options.

NULL support and new data types

In many DBMS implementations, its possible to specify a NULL as a placeholder where a value would normally appear. A NULL is not a blank or zero value, its actually a marker that says no known value can be assigned. Visual FoxPro now allows you to use NULLs in DBFs. NULL support is handled at the DBF level, so it can be used even for free tables (outside any DBC). Logic involving NULLs can be very powerful if handled properly, but is fraught with opportunities for confusion. In contrast to the familiar two-valued logic (yes or no), NULL support involves a threevalued logic (yes, no, and dont know). It can be very challenging to think about and build applications this way. In the session, well discuss some of the potential pitfalls of using NULLs, including the ways they can propagate through calculations. One of the main reasons NULL support was added is to provide compatibility with the most popular server DBMSes. Likewise, some new data types (Datetime, Double, and Currency) were added to avoid conversion hassles and loss of precision when using server data. The new data types, particularly Datetime, can also be handy in non-server applications. For some number-crunching apps, the enhanced precision of Double is imperative.

Server data access

Because this is a very popular and deep topic, there are a number of sessions scheduled to discuss various aspects of client-server computing. In this session, well take a very high-level look at the feature set for using server data. If you are interested in the nitty-gritty details of implementing client-server in Visual FoxPro, you may want to check out some of the other sessions as well. One of the most obvious enhancements in the server area is that everything you need is built into the engine, so there will no longer be a separate Connectivity Kit. Instructions for how to access a specific remote data source can be stored as a connection in the DBC. Connections have a variety of properties which allow developers to tune the way access takes place. To access data from remotely-stored tables, developers can build a view to specify tables, fields, filters and sorts (very similar to building a query). Views, which may contain references to connections, can also be stored in the DBC for reuse. Views can also reference local (DBF) data sources. Once a view has been defined, an application can issue a USE <viewname> command. This creates a cursor which looks to the application just like one created from a local table. If the view has been marked as updatable, its possible to use Xbase commands (like REPLACE) to see and change data thats actually stored in a table on the server. In the session, well walk through the process of using the visual tools to set up an updatable remote view. In addition to connections and views, Visual FoxPro provides commands and functions to allow developers to manually create and use a link to a server (ODBC) data source. This makes it possible to use SQL pass-through (SPT) techniques to exploit server features not directly supported by the ODBC drivers. In some cases, using SPT or parameterized queries can be indispensable for performance reasons, since attaching to a huge server table and reading all its rows can create undesirable delays and network traffic jams. Server data access performance has also been enhanced by integrating connectivity directly into the engine and adding smart features like progressive fetching and delayed memo download.

Integration with interface objects

Visual FoxPro provides some amazing tools for creating data access interfaces. To take advantage of the new database engine enhancements using visual controls and other objects, its helpful to understand some features effecting the relationships between forms and data sources. DataSessions are a way of providing a form with its very own set of work areas. Tables opened in one DataSession are invisible to forms which are pointing to some other DataSession. Moving a record pointer in one DataSession has no effect on the pointers in another DataSession, even if the exact same table is open in the other DataSession. One way to think of DataSessions is as a 3-D
www.dfpug.de/loseblattsammlung/migration/whitepapers/DBEngine.htm 3/5

7/9/13

Database Engine Enhancements

View window, where tables visible at one level (DataSession) are hidden when you move to the next level. With 32,767 work areas in each DataSession and an unlimited number of DataSessions possible, the problem of running out of work areas is history. Forms participate in DataSession #1 (the one the Command Window uses at startup) by default, but enabling them to have separate DataSessions is as easy as changing the form.DataSession property. DataSessions can be very useful for applications which use multiple instances of forms, protecting them from interfering with one anothers data. Forms can be set up to make automatic use of buffering. One of the properties of a form is BufferMode. By setting this property, any controls which are contained in the form can benefit from optimistic or pessimistic buffering. (Table buffering for grids, record buffering for other bound controls). In some cases, it may be desirable to force a specific flavor of buffering on a specific cursor. For example, if pessimistic locking is important for one but not all of the cursors referenced by a form, you can adjust the cursor.BufferModeOverride property in the DataEnvironment for the form.

Looking Ahead
The generational change from FoxPro 2.x to Visual FoxPro is a substantial one, and its likely to change the way we spend our time while developing business applications. The new event and object models place a heavy emphasis on design, both up front and iteratively throughout the development cycle. Database engine enhancements make it much easier to let FoxPro do it rather than writing procedural code to handle things like buffering and transactions. Heres a scenario that may be playing on a desktop near you quite soon. A user specifies a need for a new master-detail data entry form in an existing application. The developer takes these steps: opens the applications Project Container and DBC creates a new form (which automatically inherits characteristics from the corporate standard template form) and opens its DataEnvironment drags the two relevant tables from the Project Container to the forms DataEnvironment. Their relationship is automatically carried along adds several textboxes (for the fields from the master table) and a grid (for information from the detail table) to the form and sets their ControlSource/RowSource properties appropriately changes the forms BufferMode property to Optimistic and its DataSession property to .T. adds a general-purpose navigation toolbar (VCR buttons) from the applications visual class library gives the user a working data entry form with built-in support for multiple instancing and multiuser access Elapsed time: 10 minutes! This is the leverage that proper use of the new tools can provide. Whats conspicuous in this process is the absence of any code to handle setting up indexes, relationships, locking, SCATTER/ GATHER, and all the other drudgework weve been used to. The database engine enhancements make these complexities of data access transparent from the application builders point of view as well as the users. When assembling components is this easy, the burden of development shifts to making sure the components are well thought out and well debugged, and to writing code to handle the inevitable exceptions. Error handling, including dealing with violations of DBC-stored rules and triggers, will occupy proportionately more development time than before. Designing applications in the new world means building classes that are smart enough to let FoxPro handle what its best at, and robust enough to deal with all the exceptions gracefully, in a way that preserves the integrity of the data and meets the needs of the user. Visual FoxPro provides an exceptional set of enhancements to the database engine in order to support these goals.
www.dfpug.de/loseblattsammlung/migration/whitepapers/DBEngine.htm 4/5

7/9/13

Database Engine Enhancements

www.dfpug.de/loseblattsammlung/migration/whitepapers/DBEngine.htm

5/5