Data management support for statistical data editing and subset selection

Statistical analysis of large data sets often involves an initial data editing and preparation phase to check the validity of individual data items, check for consistency among related data, correct erroneous data, and supply (impute) values for missing data where possible. During this preparatory phase of analysis, it is often necessary to partition the data set into a number of subsets by logical selection and/or random sampling techniques for purposes of hypothesis testing. This paper examines the data management support required by these editing and subsetting operations in terms of data descriptions, data manipulation functions, and logical and physical data structures. The design of a data management system which seeks to meet these requirements is described in detail. The system, called SDB, is built around a self-describing transposed file structure and supporting data access software. SDB representations of some logical data structures which are commonly encountered in statistical databases are also described. Experiences with a partial implementation of the system and its application in an interactive data editor have been encouraging.