Content Indexing

DigiScope "Content Index Creation" and "Search/eDiscovery of an Index" contains point in time contents of Mailboxes, Folders, and all messaging types like Emails, Contacts, Meetings, Appointments, Tasks, and any associated attachments. DigiScope is multi-threaded and subsequently is very performant when searching, exporting, or restoring large datasets.  That said if you make a query mistake or need to run various searches against a large database it can take a great deal of time. The DigiScope indexing process queries the source to create an index of the all the non-filtered content. Once an index is created, the operator can execute searches against the index of that database to produce blazing fast results.

NOTE: Indexing is only supported within the 64 bit version of DigiScope

DigiScope Content Indexing;

• Leverages the Elasticsearch engine, which is based upon the Lucene library.

• Generates indexes between 5 to 10% of the size of database content being indexed.

• ALL indexes are located within the same root location designated during the Elasticsearch setup wizard.

• Scans all mailboxes, folders, and items that are viewable within DigiScope. (i.e. without filters: everything will be indexed. However, if the operator applies filters during indexing: only those mailboxes, folders, and items will be available within the index.)

Before creating an Index, consider the following:

• Index creation is a "Resource Intensive Process" that requires the following for optimal performance:

• CPU:  Modern processor with multiple cores such as the 6th Generation+ Intel Core i5/i7/i9 or the AMD Zen series or later.

• MEMORY: 32GB+ recommended, 16 GB minimum. NOTE: Do not rely upon Swap/Paging files which are sometimes referred to as Virtual Memory as this will cause severe performance degradation.

• STORAGE: Solid State Drives are recommended for optimal performance. NOTE: Disks are the slowest critical resource for indexing that can quickly become the bottleneck. If you don’t have SSD's use the fastest low latency spinning disks available.

• Indexes may only be created & searched at the root of an Exchange Data Source (EDS) i.e. you cannot create nor search an index at a mailbox or folder level. It must ALWAYS run from the root/top-level of database or 365 tenant.

• That said: filters can be applied at the Mailbox, Folder, and Item level during the creation process so that only the desired data will be indexed.

• Multiple indexes can be created for each database or tenant; the only requirements are that you have enough disk space to store the indexes and each index should have a unique & meaningful name. (i.e. DB1, DB1_Joe-Sam-Harold_Mailboxes, DB1_12-20-20 etc...)

• Indexes may only be created & searched at the root of an Exchange Data Source (EDS) i.e. you cannot create nor search an index at a mailbox or folder level, it must ALWAYS run from the root/top-level of database or 365 tenant

• That said filters can be applied at the Mailbox, Folder and Item level  during the creation process so that only the desired data will be indexed.

• Multiple indexes can be created for each database or tenant, the only requirements are that you have enough disk space to store the indexes and each index should have a unique & meaningful  name, i.e. DB1, DB1_Joe-Sam-Harold_Mailboxes, DB1_12-20-20 etc...

• The Exchange Data Sources below may be indexed at their root/top-level:

• Offline Exchange Database (.EDB files.)

• On-Premises Exchange Server databases.

• Exchange Online/O365 tenant.

• Each Exchange Data Source must be indexed/searched separately. It is not currently possible to generate indexes across a group of Exchange Data Sources, nor is it currently possible to search across multiple indexes.

• Offline Exchange Databases (EDB's) are considered data-at-rest, or point-in-time snapshots/copies of an Exchange database which cannot accept new items, nor can it delete or modify existing messaging items.

• On-Premise/Live/Production Exchange Databases are Dynamic since they are in a constant state of change, i.e. creation, deletion, modification of messaging items.

• Once a point-in-time index is created; you can easily search the index without having to execute a real-time search over and over. (i.e. open and query every item within the source by search criteria.)

For example: Fabrikam-West is an 855GB Offline Exchange Database (.EDB) that contains 355 mailboxes with data from January 1st 1995 through today.

• The goal is to query 48 of the 355 mailboxes for keywords and phrases between August 15th, 2006 and January 15th, 2018.

• We could index the entire database without filters and then apply the required filters during the index search process, however in this example the request for the 48 mailboxes out of 355 mailboxes available is a one-off request for a relatively small number of mailboxes that can be accomplished much faster by applying filters pre-indexing. To accomplish this:

• Create & apply the desired Mailbox, Folder, and Item level filters as outlined within the knowledge base (KB) DigiScope articles on filters. https://support.lucid8.com/support/solutions/folders/6000029905

• Depending upon the circumstances you may want to apply the Mailbox, Folder, and Item level filters or just certain filters.

• Since this database has 355 mailboxes and we only want information from 48 of them, it makes sense to apply a Mailbox level filter to hide/filter out all mailboxes other than the 48 mailboxes requested. Doing so will dramatically decrease the time to index all 355 mailboxes within the database.

• If you will be executing subsequent searches within the mailboxes for various folder or date-time combinations: it would be best to not apply folder and item level filters during the indexing process; instead apply those parameters during the search of the index. This will provide more flexibility should it be required to execute multiple searches on the 48 mailboxes.

Continue to Creating DigiScope Content Indexes