The study and review of the American Civil War has generally focused on the senior military officers and major battles and not on an analytical look at the War from the perspective of the individual soldier. Using information from each soldier's military and civilian experiences to build a database from "ground up" rather than "top down", Historical Data Systems has created the only database of its kind that can be used for statistical and analytical examinations of the War. It is now possible to examine and measure the impact these individual soldier experiences had upon regimental effectiveness.

The American Civil War Research Database is a relational database. This means that there are numerous files (i.e. roster records, pension index records, GAR records, etc. ) which are "related" to each other. With HDS' Database the "relationship" or connection between these multiple files is the soldier's name. These files contain information gathered from the different sources used as discussed below. (See the bibliography section for a data source list.) Not every soldier is in every file, nor is the type of information about a soldier in a particular file the same for every other soldier in that same file. Also, the pace at which we at HDS can enter the information about each soldier will vary significantly based on the type of source documents available to us. Therefore, the Database is not a complete biographical story on each soldier, but rather a source of significant war and post war events that a soldier experienced that can be used for the study of the Civil War.

Building a soldier's story is a continuous process that requires information from varied and diverse sources; sources that have often been generally unavailable or inaccessible to the public. By utilizing relational database technology HDS can continue to expand the Database as the source information becomes available to us. Our primary source of information comes from the State Rosters as published by the Adjutants' General of each state. In addition to this primary source, we have begun to enter information for Roll of Honor soldiers, soldiers awarded the Medal of Honor, soldiers or family members who filed for pensions (pension record index information), 1860 census town summary information, 1890 enumeration of Civil War veterans or widows, regimental histories, etc. Data entry for these files is slower than for our primary sources since the data entry process is more complex.

HDS has made a major investment of time, material, and money in building the Database. With over 100,000 hours already invested, the Database contains the records of over 4 1/4 million soldiers. We continue to add more soldiers and more information to the Database. We have released the Database now to the Web because we believe that it has reached a critical mass where significant research and study can be started. A greater appreciation of the expense of the War can be made by conducting statistical examination of data for the work completed and extrapolating many of the results.

THE HDS CHALLENGES

In building a "soldier-centric" database, Historical Data Systems is presented with four major challenges: gathering the data, completeness of data, quality of data, and merging multiple data sources into a single database.

There is no central repository for American Civil War soldier information. Thus, our first challenge is locating the data. There are three key steps to this process: locating the appropriate information, gaining accessibility to the data, and entering the data into the Database. The majority of the source documents used in building the Database are century old books or microfilm images of original documents. After locating and gaining access to the data the question of how to enter the information into the Database arises. The age and fragility of the source documents, as well as the font type and size makes scanning technology ineffective for data entering. Therefore, the data is manually keyed into the Database and then verified for accuracy. This is the most time consuming and costly part of building the Database.

The second challenge, data completeness, refers to the type and amount of information available for each soldier. Our primary source of information for each soldier is the states' official records as published by the Adjutants General for each state. These are typically referred to as the "State Rosters" and contain information on every soldier from the state as well as brief regimental histories (which have been included as part of the Database). The inconsistency of information published by each state in their "State Rosters" has led to variations of what is available for each soldier in the Database. By utilizing multiple data sources as well as information supplied by our subscribers, we are able to fill in many of the "holes" in a soldier's war and post war record. A detailed checklist of information available by state is included in the "Database Status" section of the site. As can be seen, Massachusetts records are among the most complete but unfortunately are atypical of how most states detailed the service records of their citizens.

The next challenge is database integrity. Before releasing the Database to the web site, each record goes through a series of data edits. Many of the source records contain printing, spelling, date, or factual errors or are incomplete. For example, with inter and intra regimental transfers, names may be recorded differently (the Database contains over 18,000 unique first names!) or aliases used creating the appearance of multiple soldiers (aliases were not uncommon.). Also, records were sometimes lost.

Utilizing sophisticated edit routines, many but not all of these errors are caught. For example, we can determine a military record is wrong if it lists a soldier as being wounded at the battle of Gettysburg in July, 1862 (rather then in 1863) or if he appears to have mustered out before mustering in. In many instances we can follow a soldier when transferred even if there is a name change. Likewise, we catch many of the name variations for a soldier (and list them) that appear throughout the various data sources used to build the Database.

The fourth Database challenge is the merging together of the various files of information. The "key relationship" between these files is the soldier's name. However, this can be difficult. For example, G. Washington Smith (as in his roster record) may have been awarded the Medal of Honor for bravery but yet in the Medal of Honor file there isn't a G. Washington Smith but there is a G. W. Smithe, in the Roll of Honor file there is a G.W. Smythe and in the pension record index file there is a George W. Smyth. HDS developed data checking routines to catch many of these and other data issues to reduce the occurrence of duplication and questionable data. However, even with these edit routines, HDS can not guaranty that the Database is error free. We welcome our subscribers' help in rectifying any errors or supplying us with information they may have about family members or others.