Refinement Process

The Keyword Suggester is designed as a semi-automatic system that assists you with your image tagging rather than fully automatically tag images. In this section we describe how you can implement a refinement process to optimize your keyword suggestions in a iterative refinement steps.

The brief workflow is as follows:

  1. After the user uploads an example image its URL is send to the Keyword Suggester.
  2. Keyword Suggester returns image suggestions as well as keyword suggestions.
  3. The user can then refine the suggestions by:
    • clicking one of the suggested images that have a similar context,
    • clicking one or more suggested keywords if they match the provided content,
    • manually adding keywords if all suggestions are not helpful.
  4. All of these actions trigger a new suggestion query and return adapted suggestions.

The refinement step should be repeated until the suggestions are meaningful. Then the image can be tagged.

In our experience a good suggestion is achieved after one or two refinement steps.

The refinement process logic has to be implemented on client side. Since Solr does not have any session management some middleware component or your UI component needs to keep track of the already chosen keywords and the last selected image.

You can also deviate from the proposed procedure and implement your own refinement process.

Use External image caching

To avoid repeated image downloading and analyzing when sending the same ksuggest=url: param several times it is recommended to configure a cache. Please read External image caching for how to do so. This will speed up your queries.

Refinement examples

In the following sections we present and explain three examples how a refinement process could look like for tagging a new image. Of course, these are just possible examples and the overall workflow can be adjusted according to your needs.

Refinement by selecting image

A typical refinement process starts with an image URL for which we want to get keyword suggestions. A contributor uploads an image to your server, assuming the image is then available via the URL http://example.org/uploaded.jpg.

In the first step the URL is used to retrieve images that look alike and keyword suggestions describing the uploaded image. The suggested images and keywords are shown to the user. In the illustration below in step 2 the user chooses one of the response images that shows the same or similar content as the uploaded image. A new suggestion query is triggered and the refined suggestions (images and keywords) are presented to the user.

Refinement process with selected response image:

Refinement by selecting keyword

The user can also refine the results by choosing a keyword or providing a keyword manually which describes the content of the uploaded image. The keywords should be somewhat specific to improve the following suggestions quickly. For example the user can be asked for a keyword when uploading the image in the first step.

In the illustration below an uploaded image (referenced by URL) and the keyword A are provided. After the response has been presented the user selects the suggested keyword B to change the context more in this direction. In step 2 both given keywords A and B as well as the uploaded image are provided to the Keyword Suggester to finally retrieve meaningful suggestions.

Refinement process starting with URL and a keyword. User chooses a suggested keyword as refinement:

Refinement with initial keyword input

Another typical refinement workflow is shown in the illustration below. The process starts with the given uploaded image URL and a manually added keyword A. After reviewing the response images and suggestions the user clicks on a contextual similar image as refinement. The image ID 123 as well as the manually provided keyword are sent to the Keyword Suggester and the refined suggestions are provided to the user in step 3.

Refinement process with URL and keyword. User selects response image as refinement: