Bulk Loading and Processing with Coherence

Bulk Writing to a Cache

A common scenario when using Coherence is to pre populate a cache before the application makes use of the cache. A simple way to do this would be:

This works, but each call to put may result in network traffic, especially for partitioned and replicated caches. Additionally, each call to put will return the object it just replaced in the cache (per the java.util.Map interface) which will add more unnecessary overhead. This can be made much more efficient by using putAll instead:

Efficient processing of filter results

Coherence provides the ability to query caches based on criteria via the filter API. Here is an example (given entries with integers as keys and strings as values):

This example works for small data sets, but it may encounter problems if the data set is too large, such as running out of heap space. Here is a pattern to process query results in batches to avoid this problem:

In this example, all keys for entries that match the filter are returned, but only BUFFER_SIZE (in this case, 100) entries are retrieved from the cache at a time.

Note that ${xhtml} can be used to process results in parts, similar to the example above. However LimitFilter is meant for scenarios where the results will be paged, such as in a user interface. It is not an efficient means to process all data in a query result.

A Complete Example

Here is an example program that demonstrates the concepts described above. Note this can be downloaded from the forums.

Here are the steps to running the example:

1. Save this file as com/tangosol/examples/PagedQuery.java
2. Point the classpath to the Coherence libraries and the current directory
3. Compile and run the example

$ vi com/tangosol/examples/PagedQuery.java

$ export TANGOSOL_HOME=[**Coherence install directory**]

$ export CLASSPATH=$TANGOSOL_HOME/lib/tangosol.jar:$TANGOSOL_HOME/lib/coherence.jar:.

$ javac com/tangosol/examples/PagedQuery.java

$ java com.tangosol.examples.PagedQuery

Tangosol Coherence Version 3.2.2/371

******************************************************************************
*
* Tangosol Coherence is licensed by Tangosol, Inc.
*
* Licensed for evaluation use from 2007-02-01 until 2007-04-01 (30 days
* remaining)
*   Tangosol Coherence: DataGrid Edition
*   Tangosol Coherence: Caching Edition
*   Tangosol Coherence: Data Client
*   Tangosol Coherence: Application Edition
*   Tangosol Coherence: Real-Time Client
*   Tangosol Coherence: Compute Client
*
* A production license is required for production use. For more information,
* see http://www.tangosol.com/license.jsp.
*
* Copyright (c) 2000-2006 Tangosol, Inc.
*
******************************************************************************

2007-02-28 22:48:38.208 Tangosol Coherence DGE 3.2.2/371  (thread=Cluster, member=n/a): Created a new cluster
 with Member(Id=1, Timestamp=2007-02-28 22:48:34.828, Address=192.168.0.6:8088, MachineId=26630, Edition=DataGrid
 Edition, Mode=Evaluation, CpuCount=2, SocketCount=2) UID=0xC0A80006000001110B9D118C68061F98
Adding 1024 entries to cache
Adding 1024 entries to cache

...repeated many times...

Adding 1024 entries to cache
Adding 1024 entries to cache
Adding 1024 entries to cache
processing chunk of 100 contacts:
  [25827]=Contact{Name=Cgkyleass Kmknztk, Phone=(285) 452-0000}
  [4847]=Contact{Name=Cyedlujlc Ruexrtgla, Phone=(255) 296-0000}
...repeated many times
  [33516]=Contact{Name=Cjfwlxa Wsfhrj, Phone=(683) 968-0000}
  [71832]=Contact{Name=Clfsyk Dwncpr, Phone=(551) 957-0000}
processing chunk of 100 contacts:
  [38789]=Contact{Name=Cezmcxaokf Kwztt, Phone=(725) 575-0000}
  [87654]=Contact{Name=Cuxcwtkl Tqxmw, Phone=(244) 521-0000}
...repeated many times
  [96164]=Contact{Name=Cfpmbvq Qaxty, Phone=(596) 381-0000}
  [29502]=Contact{Name=Cofcdfgzp Nczpdg, Phone=(563) 983-0000}
...
processing chunk of 80 contacts:
  [49179]=Contact{Name=Czbjokh Nrinuphmsv, Phone=(140) 353-0000}
  [84463]=Contact{Name=Cyidbd Rnria, Phone=(571) 681-0000}
...
  [2530]=Contact{Name=Ciazkpbos Awndvrvcd, Phone=(676) 700-0000}
  [9371]=Contact{Name=Cpqo Rmdw, Phone=(977) 729-0000}