LDA Predictor

Information at a Glance

This operator also stores in HDFS the Featurized data set used as input for LDA Prediction (and created from the featurization parameters specified in the LDA Trainer). You can drag this data set onto the canvas for further analysis.

For more information about using LDA, see Unsupervised Text Mining and LDA Training and Model Evaluation Tips.

Input

The LDA Predictor requires the following 2 inputs.

• The output model from an LDA Trainer.
• A tabular data set with at least a unique document ID column and a text content column (for example, output of the Text Extractor operator).
  Note: The LDA Predictor operator includes the Text Featurization of Text Column, which converts raw text to n-gram features.

Bad or Missing Values

If a row contains a null value in at least one of the Doc ID column or Text column, the row is removed from the data set. The number of null values removed can be listed in the Summary section of the output (depending on the chosen option for Write Rows Removed Due to Null Data To File).

Configuration

Outputs

Visual Output

• A tabular data set output with the following columns (which can be further filtered, depending on the columns needed for the user's specific use case):
  • Doc ID Column and Pass Through columns selected
  • top_topics_summary column with the top topics and their corresponding weights in a dictionary format (the number of top topics displayed corresponds to the Max Topics to Describe Documents parameter set in the LDA Trainer).
  • Full topic distribution weights (weight_topic_X columns)
  • Top topic distribution as pairs of columns topic_ranked_X, weight_topic_ranked_X). The number of top topics displayed corresponds to the Max Topics to Describe Documents parameter set in the LDA Trainer)

  
  
  

• Featurized data set (stored in HDFS): featurized data set used as input for LDA prediction:

  
  
  

• Summary tab that describes the parameters selected, model prediction results, output location of the results, and the number of rows processed

  
  
  

Training data Log Likelihood (entire corpus): Lower bound on the log likelihood of the entire corpus.

Training data Average Log Likelihood (per document): Log Likelihood/Total Number of Documents

Training data Log Perplexity (per token): Upper bound on the log perplexity per token of the provided documents, given the inferred topics (lower is better). In information theory, perplexity is a measurement of how well a probability distribution (or probability model) predicts a sample, and is a common measure of model performance in topic modeling. More specifically, it measures how well the word counts of the documents are represented by the word distributions represented by the topics. A low perplexity indicates the probability distribution is good at predicting the sample.

Data Output

This is a semi-terminal operator. It can be connected to any subsequent operator at design time, but it does not transmit the full output schema until the user runs the operator. The partial output schema at design time is only the first columns of the output(Doc ID Column, Pass Through columns selected and top_topics_summary column). After running it, the output schema is automatically updated, and subsequent operators turn red in case the UI parameters selection is no longer valid.

Note: The final output schema of the Transpose operator is cleared you take one of the following actions.

• Change the configuration properties of the Transpose operator.
• Change the input connected to the Transpose operator.
• Clear the step run results of the Transpose operator.

In this case, the output schema transmitted to subsequent operators again becomes the partial schema defined at design time (hence, subsequent operators can turn invalid), and you must run the Transpose operator again to transmit the new output schema.

Example