Faster search performance
If you experience long processing times when doing visual similarity searches you can speed up the process by distribute your collection over several shards. The search will then be done distributed leading to a faster overall performance.
There are plenty of other performance factors that you should consider (see Solr Wiki for details). Also make sure you are running Solr on up to date hardware.
Another approach could be to limit the time Solr is allowed to search.
timeAllowed determines how long a query can be at the most (see Solr documentation for details).
Solr will return results in a garantueed time frame but the drawback is, that not all documents might be scored, because the scoring process was aborted due to the time limit.
However, for a visual image search this approach might be feasable.
If there are many documents without an image descriptor in your collection, limit the search set to those that actually have one (see Find documents with or without image descriptor).