Discusses table limiting to increase performance.
Your system only has so much physical memory (called random access memory or RAM) in which to store the results of analysis. When data requirements exceed that memory, it has to use virtual memory, exchanging data as needed from RAM to the hard disk and back to RAM. This can create a low performance situation known as thrashing, in which a large amount of activity (swapping pages of data in and out of RAM) accomplishes very little.
Unfortunately, there is no perfect solution to the issue of overwhelming your memory with data. However, there are measures you can take to reduce how often your system has to swap data out to the disk. You can add more RAM, which up to a point will increase performance.
Most computers with 32-bit processors can address only 4 GB of memory (that is, virtual address space, regardless of how much physical RAM you might have), and they usually allocate half to user processes and half to the operating system. This limitation creates a 2 GB per-process limit. If you have 4 GB (or more) in a machine, two simultaneous user processes can each use 2GB of physical RAM. You can configure some versions of Windows versions, such as Windows 2003 Advanced Server, to provide 3 GB of memory for user processes and 1 GB for the operating system. Webtrends can use 3 GB of available memory.
Another approach is to limit the amount of data that you store in your summary database tables. The tradeoff with this approach is that by limiting the amount of entries in a summary table, you only collect records up to the point when you reach that limit. For example, if you limit the Top Pages table to 10,000 pages, Webtrends only aggregates data for the first 10,000 pages entered in the table. Any new pages encountered in the Web data activity file after that will not be entered in the table, but is recorded in the “Other” section. This means that if your site experiences a great deal of traffic and has 200,000 or 300,000 pages, then limiting it to the top 10,000 will significantly impact the accuracy of your reports. However, if you were to perhaps limit it to the top 50,000, or use Virtual Top Pages, you might expect to get a reasonably accurate representation of the top pages in your reports.
In addition to requiring less storage space in RAM, limiting tables also reduces the time spent inserting data into the database. This time savings is fairly minimal in comparison to the time savings achieved by avoiding swapping data out to the hard disk.
Whether you have to limit table sizes depends on three factors:
- System processing speed
- Amount of RAM
- Tables being created (daily, weekly, monthly, quarterly, and/or yearly)
System processing speed affects how long the instructions and data must stay in main memory, while the amount of RAM affects how much data can be kept in main memory at any given time. And finally, the periods for which you have chosen to generate reports determine which tables exist and have data aggregated in them. If you have selected to aggregate data in yearly tables, toward the end of a year, you would be maintaining almost an entire year’s worth of data. Because the summary tables have to be loaded in RAM to aggregate the data, the larger the amount of data, the more likely that you may have to swap out to hard disk.
For more detailed information about limiting tables, see “Optimizing Reports Using Table Limiting” in the Webtrends Administration User’s Guide.