Use AutoCommit as Commit Strategy

After a commit, Solr opens a new searcher to make the index changes available for search queries and regenerates caches Solr opens a new cache and seeds it with image IDdocID pairs of the old cache. The warming process can take some time (up to several minutes) and if another commit is triggered while Solr still opens a new searcher and fills a new cache a warning will be logged:

PERFORMANCE WARNING: Overlapping onDeckSearchers

The cache used by UniversalConnectorComponent takes a lot of RAM. If a commit is done the RAM usage is doubled until the new searcher is available and the old cache can be dismissed. The RAM usage will be even higher if several new searchers are opened in parallel (caused by frequent commits).

Think about your commit strategy with regards to the update frequency and warming up time to avoid warnings like the one shown above.

Solution: Instead of manually commit, configure a autoCommit with reasonable long time in between, to make sure, only one searcher is warmed up at a time. Each warming searcher will generate a new cache consuming twice as RAM as usual (the old cache is still available and used for search until the new one is ready). When the new cache is filled with the specified amount of elements AutoWarmCount the old cache will be dismissed.

The listing below shows an example configuration of the solrconfig.xml. The parameter maxTime sets the number of milliseconds since the oldest uncommitted update. If the maxTime limit is reached, Solr automatically performs a commit operation.

Example AutoCommit configuration. Solr automatically commits changes 60 seconds after an update:

<autoCommit>
  <maxTime>60000</maxTime>
  <openSearcher>true</openSearcher>
</autoCommit>

The optimal maxTime depends on your collection size, hardware settings and cache configurations.

Do the following to determine an optimal maxTime setting:

  1. Set your wished cache configuration (see DocIdCacheRegenerator).
  2. Change the index and do a commit to force Solr to open a new searcher and to warm the Universal Connector cache.
  3. Check the warming time of your cache (see listing below). If "warmupTime":120000, it means the new searcher and the new cache is available 120 seconds (2 minutes) after the commit.
  4. You should double the warmup time of the cache and set maxTime accordingly. In this example maxTime should be at least 4 minutes: <maxTime>240000</maxTime>.
  5. In production you can now add and remove documents without worrying about commits. Solr will take care about the commits.

Retrieve statistics and warmupTime of the configured cache with name universalConnectorCache. The specific name depends on your configuration (see DocIdCacheRegenerator):

http://localhost:8080/solr/admin/mbeans?wt=json&indent=true&cat=CACHE&key=universalConnectorCache&stats=true
{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "solr-mbeans":[
    "CACHE",{
      "universalConnectorCache":{
        "class":"org.apache.solr.search.LRUCache",
        "version":"1.0",
        "description":"LRU Cache(maxSize=100000, initialSize=50000, autowarmCount=100%, regenerator=de.pixolution.solr.search.DocIdCacheRegenerator@19d2e66e)",
        "src":null,
        "stats":{
          "lookups":0,
          "hits":0,
          "hitratio":0.0,
          "inserts":100000,
          "evictions":0,
          "size":100000,
          "warmupTime":1918,
          "cumulative_lookups":0,
          "cumulative_hits":0,
          "cumulative_hitratio":0.0,
          "cumulative_inserts":0,
          "cumulative_evictions":0}}}]}

Deactivate QueryResultCache

The QueryResultCache caches the results of a specific query. This cache consumes a lot of RAM and does not have a great impact on performance when using pixolution flow with the Universal Connector. The parsing and lookup of docIds consumes the most time of the query and this is always done before accessing the QueryResultCache.

Therefore, you can safely deactivate QueryResultCache in the solrconfig.xml by setting size=0:

<queryResultCache class="solr.LRUCache"
  size="0"
  initialSize="0"
  autowarmCount="0"/>