Project Tab

Select the Project tab in the Text Mining dialog box to access options to select the internal database file that will be used for indexing of the documents. For performance reasons, and as also briefly described in the Introductory Overview, Statistica Text Mining and Document Retrieval incorporates advanced relational database components to maintain and update the index of words or terms by document. With large document collections and complex text, these databases can become extremely large and, hence, an efficient relational database scheme was chosen to store this information. An additional advantage of this approach is that this database can be stored in a user-defined location on the hard disk and reused. You can use the options on the Defaults tab to save or retrieve the settings for these options, and to set the defaults for future analyses.

Note: Choosing an existing database; deploying databases: With the options on this tab, not only can you determine the location where the database containing the index of words/terms and documents is to be stored (note that this database can become quite large), but you can also select a specific existing database created during a previous analysis and use the information contained therein.

Databases created with Statistica Text Mining and Document Retrieval contain not only the list of indexed words and their frequencies in each document, but also information about which of the indexed words were selected for the analysis (words or terms can be indexed, but not selected and, hence, ignored for subsequent results), as well as results from a singular value decomposition of the frequencies or other derived indices for the selected words. For more details, see also Singular Value Decomposition in Statistica Text Mining and Document Retrieval. Using the options available on this tab, you can either update an existing database with the words or terms found in new documents, or you can index a new set of documents using only the selected terms in an existing database.

This type of indexing based on words selected during previous analyses can be considered a form of deployment of the database, in the sense of deployment commonly used in the context of trained models in predictive data mining. The program can also compute word coefficients and document scores based on results from singular value decomposition performed in a prior analysis and stored in the database. Hence, these options enable you to compute scores for new documents based on a previous analysis; this functionality may be critical if you want to use information extracted from text in predictive data mining projects based on numeric indices that were derived during training from unstructured text.