https://www.elastic.co/guide/en/elasticsearch/reference/7.6/search-request-body.html#request-body-search-scroll

Specifies search criteria as request body parameters.

GET /twitter/_search
{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

HTTP GET and HTTP POST supported?

Both HTTP GET and HTTP POST can be used to execute search with body. Since not all clients support GET with body, POST is allowed as well.

  • In case we only want to know if there are any documents matching a specific query, we can set the size¬†to¬†0¬†to indicate that we are not interested in the search results.

About terminate_after

  • Also we can set¬†terminate_after¬†to¬†1¬†to indicate that the query execution can be terminated whenever the first matching document was found (per shard).
GET /_search?q=message:number&size=0&terminate_after=1

The response will not contain any hits as the size was set to 0.

hits.total

The hits.total will be either

  • equal to 0, indicating that there were no matching documents, or
  • greater than¬†0¬†meaning that there were at least as many documents matching the query when it was early terminated.

    Also if the query was terminated early, the terminated_early flag will be set to true in the response.

{
  "took": 3,
  "timed_out": false,
  "terminated_early": true,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total" : {
        "value": 1,
        "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

The took time in the response contains

  • the milliseconds that this request took for processing,
  • beginning quickly after the node received the query,
  • up until all search related work is done and
  • before the above JSON is returned to the client.
  • This means it includes the time spent waiting in thread pools, executing a distributed search across the whole cluster and gathering all the results.

Scroll

Q) search vs scroll

 

  • While a¬†search¬†request returns a single ‚Äúpage‚ÄĚ of results,
  • the¬†scroll¬†API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database.

Q) What is the scroll use-case?

  • Scrolling is not intended for real time user requests, but rather
  • Scrolling is intended for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration. or while using a reconciliation of data between clusters.

Q) Impact of scroll on newer data 

  • The results that are returned from a scroll request reflect the state of the index at the time that the initial search¬†request was made, like a snapshot in time.
  • Subsequent changes to documents (index, update or delete) will only affect later search requests. ( not the scroll requests)

 

Let‚Äôs see a scroll request in action ūüôā ‚Äď scroll/search-context/ scroll_id

In order to use scrolling,

the initial search request
  • the initial search request should specify the scroll¬†parameter in the query string,
  • which tells Elasticsearch how long it should keep the ‚Äúsearch context‚ÄĚ alive (see Keeping the search context alive),
  • eg¬†?scroll=1m.
POST /twitter/_search?scroll=1m
{
    "size": 100,
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    }
}

The size parameter allows you to configure the maximum number of hits to be returned with each batch of results.

Passing _scroll_id in scroll API for subsequent results

The result from the above request includes a _scroll_id, which should be passed to the scroll API in order to retrieve the next batch of results.

POST /_search/scroll 
{
    "scroll" : "1m", 
    "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==" 
}

 

  • Each call to the scroll¬†API returns the next batch of results until there are no more results left to return,
  • ie the¬†hits¬†array is empty.

 

Few points to note regarding GET/POST/URL/scroll parameter/scroll_id parameter

  • GET or POST can be used and
  • The URL should not include the index name‚ÄČ‚ÄĒ‚ÄČthis is specified in the original search request instead.
  • The scroll parameter tells Elasticsearch to keep the search context open for another 1m.
  • The scroll_id parameter is the one passed as part of ‚Äúscroll_id‚Äú

 

Notes regarding _scroll_id / request-with-aggs/requests-sort-order-as_doc:

  1. The initial search request and each subsequent scroll request each return a¬†_scroll_id. While the¬†_scroll_id¬†may change between requests, it doesn‚Äôt always change‚ÄČ‚ÄĒ‚ÄČin any case, only the most recently received¬†_scroll_id¬†should be used.
  2. If the request specifies aggregations, only the initial search response will contain the aggregations results.
  3. Scroll requests have optimizations that make them faster when the sort order is _doc. If you want to iterate over all documents regardless of the order, this is the most efficient option

 

More details if you are interested to read on :

Keeping the search context alive

A scroll returns all the documents which matched the search at the time of the initial search request. It ignores any subsequent changes to these documents. The scroll_id identifies a search context which keeps track of everything that Elasticsearch needs to return the correct documents. The search context is created by the initial request and kept alive by subsequent requests.

The¬†scroll¬†parameter (passed to the¬†search¬†request and to every¬†scroll¬†request) tells Elasticsearch how long it should keep the search context alive. Its value (e.g.¬†1m, see¬†Time units) does not need to be long enough to process all data‚ÄČ‚ÄĒ‚ÄČit just needs to be long enough to process the previous batch of results. Each¬†scroll¬†request (with the¬†scroll¬†parameter) sets a new expiry time. If a¬†scroll¬†request doesn‚Äôt pass in the¬†scroll¬†parameter, then the search context will be freed as part of¬†that¬†scroll¬†request.

Normally, the background merge process optimizes the index by merging together smaller segments to create new, bigger segments. Once the smaller segments are no longer needed they are deleted. This process continues during scrolling, but an open search context prevents the old segments from being deleted since they are still in use.

Keeping older segments alive means that more disk space and file handles are needed. Ensure that you have configured your nodes to have ample free file handles. See File Descriptors.

Additionally, if a segment contains deleted or updated documents then the search context must keep track of whether each document in the segment was live at the time of the initial search request. Ensure that your nodes have sufficient heap space if you have many open scrolls on an index that is subject to ongoing deletes or updates.

To prevent against issues caused by having too many scrolls open, the user is not allowed to open scrolls past a certain limit. By default, the maximum number of open scrolls is 500. This limit can be updated with the search.max_open_scroll_context cluster setting.

 

You can check how many search contexts are open with the nodes stats API:

GET /_nodes/stats/indices/search

 

Clear scroll API

Search context are automatically removed when the scroll timeout has been exceeded. However keeping scrolls open has a cost, as discussed in the previous section so scrolls should be explicitly cleared as soon as the scroll is not being used anymore using the clear-scroll API:

DELETE /_search/scroll
{
    "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}

Multiple scroll IDs can be passed as array:

DELETE /_search/scroll
{
    "scroll_id" : [
      "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==",
      "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAABFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAAAxZrUllkUVlCa1NqNmRMaUhiQlZkMWFBAAAAAAAAAAIWa1JZZFFZQmtTajZkTGlIYkJWZDFhQQAAAAAAAAAFFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAABBZrUllkUVlCa1NqNmRMaUhiQlZkMWFB"
    ]
}

All search contexts can be cleared with the _all parameter:

DELETE /_search/scroll/_all

The scroll_id can also be passed as a query string parameter or in the request body. Multiple scroll IDs can be passed as comma separated values:

DELETE /_search/scroll/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==,DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAABFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAAAxZrUllkUVlCa1NqNmRMaUhiQlZkMWFBAAAAAAAAAAIWa1JZZFFZQmtTajZkTGlIYkJWZDFhQQAAAAAAAAAFFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAABBZrUllkUVlCa1NqNmRMaUhiQlZkMWFB