Testing Scoring Rule Sets
For Visitor Intelligence segmentation, Score provides the functionality to test your rule sets to validate the results. You can test rule sets against a sample of your entire data set or compare the test results against one or more data segments. Score test results provide data in three different formats: the Visitor Summary, the Rule Contribution Summary, and the Score table.
Score automatically detects the size of your data set and selects a sample size that maximizes accuracy and processing speed. If you want to test against a larger sample, you can select a larger sample size. (If your optimized sample is the largest available sample, the larger sample option is not displayed.)
Using the Score Distribution
The Score Distribution graph shows the number of visits that received a score in each range during the test period. You can use this graph to determine whether your rule set is effectively sorting visits into different groups based on their value. For example, if all visits score within the same range, or if all the visits score at the top or bottom of the range, your rule set is probably not distinguishing high-value from low-value visits. Your rule set may not be capturing the events that identify the specific groups of visitors you are interested in, or it may include too many methods of earning points so that different groups of users can score similar point values in different ways. Alternately, if you do not provide large enough point ranges for different events, all your scores may be flattened within a narrow point range.
Using the Rule Contribution Summary
You can use the Rule Contribution Summary graph to determine which rules in your rule set are responsible for the largest percentage of points scored by all the users in the test during the test period. This graph can help you refine your rule tests by identifying whether scores are being skewed by one or more rules in the rule set. For example, if you assign too many points for one rule, point differences caused by other rules may not be visible.
Using the Score Table
The Score table shows the number of visits and the scores logged for each rule in the rule set during the time period. If you decide to specify the visitor segments included in the test, the Score table also shows the scores broken down by segment.
You can use the same preconfigured or user-configured segments to limit the scoring results to an individual segment. Drag and drop a segment from the Choose Segments menu onto the Test area.
Selected segments are displayed above the graphs. To remove a segment, mouse over it in the Selected Segments list and click the x in the corner.
If you want to compare the results for different segments within the same segment group, drag more segments from the same segment group onto the Test area. You cannot compare results for segments from different segment groups in the same test. Dragging a segment from a different segment group onto the Test area replaces the original segment(s) with the new segment.
Comparing a Rule Set to a Previous Configuration
To see comparative results when you are testing an unsaved rule set, select the Compare to saved configuration check box. Score shows side-by-side results for the last saved configuration and your new, unsaved configuration. This check box is not displayed when you are testing a saved rule set.
Configuring Score Tests
- In the Profiles and Rule Sets dialog, expand a profile.
- Mouse over a rule set and click Test on the Action menu. Alternately, edit a rule set and click Test in the Rules dialog.
- By default, Score tests the most recent 30 days for which visit data has been loaded. This ensures that you can see test data even when data is not loaded consistently. If you want to change the start date for test scoring, change the value in the Start Date menu.
- By default, Score tests data from 12:00 A.M. on. If you want to change the start time for test scoring, change the value in the Time menu.
- By default, Score tests a sample of visits optimized for accuracy and performance. If you want to test a larger sample, select a different sample size. Testing a smaller sample may result in inaccurate results.
- If you want to compare rule set results across different segments, click Choose Segments, select a segment from the menu, and drag the segment on top of a graph.
- If you want to see side-by-side results for a rule set you are currently editing and the last saved version of the rule set, select the Compare to saved configuration check box.