Tag Archives: ES

The ES query found 1024 elements and reported errors

Errors are reported as follows:

Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: too_many_clauses: maxClauseCount is set to 1024

Cause: in the in query of ES, the IDS in the in are greater than 1024, resulting in ES error. The default number of elements supported by ES is 1024

Solution:

Edit elasticsearch.yml and add the following configuration:

 

index.query.bool.max_clause_count: 10240

An error is reported in the new version. The configuration item name has been modified. The following fields need to be added:

indices.query.bool.max_clause_count: 300000

 

Note YML syntax: there is a space after the colon

ES Error:”illegal_argument_exception” [How to Solve]

The specific errors reported by ES are as follows:

{
“error”: {
“root_cause”: [
{
“type”: “illegal_argument_exception”,
“reason”: “Fielddata is disabled on text fields by default. Set fielddata=true on [createHour] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.”
}
],
“type”: “search_phase_execution_exception”,
“reason”: “all shards failed”,
“phase”: “query”,
“grouped”: true,
“failed_shards”: [
{
“shard”: 0,
“index”: “gmall1205_order”,
“node”: “LCQa858ERH6qw_7asM2R3Q”,
“reason”: {
“type”: “illegal_argument_exception”,
“reason”: “Fielddata is disabled on text fields by default. Set fielddata=true on [createHour] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.”
}
}
],
“caused_by”: {
“type”: “illegal_argument_exception”,
“reason”: “Fielddata is disabled on text fields by default. Set fielddata=true on [createHour] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.”,
“caused_by”: {
“type”: “illegal_argument_exception”,
“reason”: “Fielddata is disabled on text fields by default. Set fielddata=true on [createHour] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.”
}
}
},
“status”: 400
}

2: The statement that caused this error query:

GET gmall1205_order/_search
{
“query” : {
“bool” : {
“filter” : {
“term” : {
“createDate” : “2019-09-17”
}
}
}
},
“aggregations” : {
“groupby_createHour” : {
“terms” : {
“field” : “createHour”,
“size” : 24
},
“aggregations” : {
“sum_totalamount” : {
“sum” : {
“field” : “totalAmount”
}
}
}
}
}
}

3: java code:

@Override
public Map getOrderAmontHourMap(String date) {

    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
    boolQueryBuilder.filter(new TermQueryBuilder("createDate",date));
    searchSourceBuilder.query(boolQueryBuilder);
    TermsBuilder termsBuilder = AggregationBuilders.terms("groupby_createHour")
            .field("createHour.keyword").size(24);
    SumBuilder sumBuilder = AggregationBuilders.sum("sum_totalamount").field("totalAmount");

    termsBuilder.subAggregation(sumBuilder);
    searchSourceBuilder.aggregation(termsBuilder);


    Search search = new Search.Builder(searchSourceBuilder.toString()).addIndex(GmallConstant.ES_INDEX_ORDER).addType("_doc").build();

    System.out.println(searchSourceBuilder.toString());

    Map<String,Double> hourMap=new HashMap<>();
    try {
        SearchResult searchResult = jestClient.execute(search);
        System.out.println("====>"+searchResult.toString() + searchResult.getTotal());
        List<TermsAggregation.Entry> buckets = searchResult.getAggregations().getTermsAggregation("groupby_createHour").getBuckets();
        for (TermsAggregation.Entry bucket : buckets) {
            Double hourAmount = bucket.getSumAggregation("sum_totalamount").getSum();
            String hourkey = bucket.getKey();
            hourMap.put(hourkey,hourAmount);
        }
    } catch (IOException e) {
        e.printStackTrace();
    }

    return hourMap;
}

Error analysis:

“Fielddata is disabled on text fields by default. Set fielddata=true on [createHour] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead

The Fielddata text field is disabled by default. Set fielddata = true (createHour) in order to load fielddata uninverting inverting index in memory. Note that this can use a lot of memory. Or use a keyword field

 

4: Solution

The first:

GET gmall1205_order/_search
{
“query” : {
“bool” : {
“filter” : {
“term” : {
“createDate” : “2019-09-17”
}
}
}
},
“aggregations” : {
“groupby_createHour” : {
“terms” : {
“field” : “createHou.keyword”,
“size” : 24
},
“aggregations” : {
“sum_totalamount” : {
“sum” : {
“field” : “totalAmount”
}
}
}
}
}
}

 

 

 

The second: the solution, custom indexing rules, do not use the default value to create the index

PUT gmall1205_order
{
“mappings” : {
“_doc” : {
“properties” : {
“provinceId” : {
“type” : “keyword”
},
“consignee” : {
“type” : “keyword”,
“index”:false
},
“consigneeTel” : {
“type” : “keyword”,
“index”:false
},
“createDate” : {
“type” : “keyword”
},
“createHour” : {
“type” : “keyword”
},
“createHourMinute” : {
“type” : “keyword”
},
“createTime” : {
“type” : “keyword”
},
“deliveryAddress” : {
“type” : “keyword”
},
“expireTime” : {
“type” : “keyword”
},
“id” : {
“type” : “keyword”
},
“imgUrl” : {
“type” : “keyword”,
“index”:false
},
“operateTime” : {
“type” : “keyword”
},
“orderComment” : {
“type” : “keyword”,
“index”:false
},
“orderStatus” : {
“type” : “keyword”
},
“outTradeNo” : {
“type” : “keyword”,
“index”:false
},
“parentOrderId” : {
“type” : “keyword”
},
“paymentWay” : {
“type” : “keyword”
},
“totalAmount” : {
“type” : “double”
},
“trackingNo” : {
“type” : “keyword”
},
“tradeBody” : {
“type” : “keyword”,
“index”:false
},
“userId” : {
“type” : “keyword”
}
}
}
}
}