{% extends "docs/docs_base.html" %} {% block doc_title %}Super Search{% endblock %} {% block doc_content %}

What is Super Search?

Super Search is an interface to crash reports data. There is a human-friendly interface and a public API. This guide covers the API, but the interface can be used to easily query and visualize the data.

There are 3 key points to Super Search. First, it is a secured window on all the raw crash reports data we have. Second, it allows you to use almost every field as a filter, giving you great power to explore the data we have. And third, it has advanced aggregation features that can be used to extensively analyze a subset of data.

This documentation aims at extensively covering the API.

API endpoints

SuperSearch

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}

This is the public endpoint that you should mostly be using. It doesn"t require any permissions, and gives you access to almost all of the data about crash reports. The only restrictions are fields that cover protected data like personally identifiable information.

SuperSearchUnredacted

{{ full_url(request, "api:model_wrapper", model_name="SuperSearchUnredacted") }}

This is a restricted endpoint, only accessible through the use of an API Token and your having the right set of permissions. It gives you an exhaustive access to our entire dataset, so please use this very carefully.

Basic Usage

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(product=product_name, _results_number=10) }}

User-Agent

Please set a User-Agent when using the API. This helps us know who to talk to when we're making changes.

Response

The service returns a JSON document with a defined structure. The root will always contain the following 3 keys:

hits contains a portion of the crash reports that matched the query. The number of results, the fields of those results, the ordering can all be controled using control parameters. It is an ordered list of JSON objects.

total contains the total number of crash reports that matched the query. It is a positive integer.

facets contains the results of the various aggregations that are set in the query. It is a JSON object containing ordered lists of JSON objects. Each sub-object represents a "term" and its count, and can have sub-aggregations. Theoratically, there can be any number of nested sub-aggregations.

Filters

The first thing that Super Search allows you to do is to filter on our dataset to reduce it. For example, if you only want to see crashes that happened on a given product and version, or with a specific build id. 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(product=product_name, version=version) }}

There are lots of available filters, listed in the API Reference. That list is constantly changing as we add more data to our crash reports.

Logic

When using several filters together, you do not have full control over how filter are combined together. Note that all filters can receive several values. Here are the rules:

For example, product={{ product_name }} & version=1.0 & version=2.0 & date=>2000-01-01 & date=<2001-01-01 will translate to: 

( product = {{ product_name }} ) AND
( version = 1.0 OR version = 2.0 ) AND
( date > 2000-01-01 AND date < 2001-01-01 )

Data types

Each filter has a data type, that corresponds to the type of data the associated field can receive. Those types have a direct impact on the operators you can use for each field, as explained in the Operators section. Here are the existing data types:

Data type Operators Description
enum Simple type for strings, relies on inputs being turned into "terms".
string , ~, =, $, ^, @, __null__ Advanced type for strings, gives a lot more options.
integer , >, >=, <, <=, __null__ Type for all whole numbers (short, long, integer).
float , >, >=, <, <=, __null__ Type for all float numbers.
date >, >=, <, <= Type for all dates, underlying data can be date or datetime.
boolean __true__, __null__ Type for fields that can be true or false.

Operators

Each filter supports a number of operators, that will allow you to refine your searches. Concretely, those operators translate to a string that will be put at the beginning of the value of the parameter. For example, ~ is the "contains" operator. If you want to search for all crashes that have a signature containing moz, you would do this: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(signature="~moz") }}

Some operators (like __null__ or __true__) do not care about a value, you can use them alone. All operators can be negated using the meta operator ! as a prefix. Here's how to search for all crashes with an address that is not empty: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(address="!__null__") }}

Most operators are only usable with some data types. For example, comparison operators (like < or >) are not usable with string data, but only with integers, floats, and dates. Operators for each filter are listed on the API Reference page.

Here is a list of all operators, and what data types they apply to:

Operator Code Data types Description
not ! all Meta operator to put before any other operator to turn it into the opposite.
matches all field types

Default operator.

The field value matches the searched value. This will use the analyzer of the field, so it's not case-sensitive, handles differences in spaces and punctuation, and matches closely.

contains ~ string The field value contains the searched value as a substring. This is case-sensitive.
is exactly = string The field value is exactly the searched value.
starts with ^ string The field value starts with the searched value.
ends with $ string The field value ends with the searched value.
matches regex @ string The field value matches the given regular expression. The accepted syntax is described in the Elasticsearch documentation.
does not exist __null__ string, boolean, integer, float

The field value is missing or has a null value.

Note: This will not match empty string values.

greater than > date, integer, float The field value is greater than the searched date, float, or integer value.
greater than or equal >= date, integer, float The field value is greater than or equal the searched date, float, or integer value.
lower than < date, integer, float The field value is lower than the searched date, float, or integer value.
lower than or equal <= date, integer, float The field value is lower than or equal to the searched date, float, or integer value.
is true __true__ boolean The field value is true.

Meta Parameters

Along with filters, Super Search exposes a set of "meta" parameters. Their names start with an underscore, like _results_number. These parameters control various aspects of the results in the hits key. They are pretty straight forward and are well described in the API Reference.

_results_number and _results_offset allow you to paginate over the results of a request. It is generally safer, when you need a lot of results, to run the same request in a loop while incrementing _results_offset of the value of _results_number, instead of just requesting an arbitraty large number of results at once. 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(_results_number=10, _results_offset=100) }}

Using _sort, you can, well, sort the results of a request. This parameter accepts an ordered list of fields. By default, the sorting will be ascendant. If you want to sort a field by descending order, you can prefix the field name with a minus sign (-). 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(_sort=["platform", "-build_id"]) }}

Each returned document only contains a subset of the available fields. By default, those fields are uuid, date, signature, product and version. You can control that list with the _columns parameter, that accepts a list of fields. 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(_columns=["platform", "build_id", "uptime"]) }}

Aggregations

"Facets" and "aggregations" are the same thing. Those names come from Elasticsearch, which renamed the feature to "aggregations" in its version 1.0. We are thus using the name "aggregations", or its abbreviation "aggs", in all new features.

Aggregations are all about numbers--or rather, they give you all the numbers about a dataset. There are several types of aggregations, depending of the type of data you're interested in, but they have in common that they are used for counting things. For example, you would use aggregations to:

Aggregations can be queried in various ways, but always come out in the same structure, in the facets key of the returned document. The structure is as follows: 

{
    "facet_name": [
        {
            "term": "foo",
            "count": 42,
            ["facets": {
                "other_facet_name": []
            }]
        }
    ]
}

Aggregations can theoratically be infinitely nested, but are currently limited by the available arguments to 3 levels.

There are 3 ways of getting aggregations. The first and simplest one is to use _facets. The second is to use _aggs.field_name and its derivatives, like _aggs.field1.field2. And the third one is to use special aggregations, like _histogram.field_name or _cardinality.field_name.

Basic aggregations

Using the _facets parameter allows you to run simple aggregations. You can pass it a field name (see the List of fields), and it will count the terms of that field in all the documents of the dataset. It will then return them sorted by descending count. 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(_facets="signature") }}

This will return the list of signatures that appear the most in all crash reports for the last week, sorted by descending count.

The number of results in an aggregation is 50 by default. If you want to change it, you can use the _facets_size parameter. Note that there is no way to paginate over aggregation results, and that the bigger the _facets_size the longer the request will take.

Nested aggregations

Let's say you want to get the count of platforms per product. To do that, you would use a nested aggregation: aggregate on products, and then for each product aggregate on platforms. Here's what that translates to in Super Search: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(**{"_aggs.product": "platform"}) }}

The list of all available nested aggregations is in the API Reference.

Histograms

The previous aggregations work great for string fields, but what if you want to aggregate on a date for example? Or if you want to manipulate ranges of numbers? That is what histograms are made for. Histograms work with dates, integers, and floats, and allow you to treat that data as ranges instead of "terms". For example, if you want to get a count of crashes per product per day, you would run a histogram on the date field, using an interval of 1d (one day). Here's what it looks like: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(**{"_histogram.date": "product", "_histogram_interval.date": "1d"}) }}

This works as follows: for each value of the field, it will find in which range it belongs, and put it in a bucket for that range. In our case, each date value will be put in the corresponding day bucket. Then it counts the number of document per bucket, and will run any other aggregation in each bucket. If a bucket is empty, it will not be returned, so missing values in the results simply mean that they had zero documents.

The list of accepted units for date intervals is available in the Elasticsearch documentation.

For integers and floats, you do not need a unit, just a number. Let's take the uptime field for example. If you want to count the number of crashes happening at startup (let's say startup is up to 60 seconds after launching the app), here is what you can do: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(**{"_histogram.uptime": "product", "_histogram_interval.uptime": 60}) }}

Cardinality

Sometimes you just need to count the distinct values of a field. That can be done using the cardinality feature. For example, let's count the number of distinct builds of {{ product.name }}: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(product=product.name, _facets="_cardinality.build_id") }}

Note that the results of a cardinality aggregation are different from others. There is a single key in the results, called value, and it cannot contain sub-aggregations. Also, there is no parameter for cardinality like there is for histograms, you can only use it as part of another aggregation parameter.

Combining aggregations

Did you notice, in that last example? We did not use _cardinality as a parameter but as the value of the _facets parameter. It is indeed possible to combine aggregations in various ways. Notably, _histogram.field and _cardinality.field can be used as values of _facets or _aggs.field. Let us, for example, count the number of distinct values of install_time per product per version: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(**{"_aggs.product.version": "_cardinality.install_time"}) }}

Now let's count the number of distinct build_id per day for {{ product.name }}: 

{{ full_url(request, "api:model_wrapper", model_name="SuperSearch") }}?{{ make_query_string(**{"product": product_name, "_histogram.date": "_cardinality.build_id", "_histogram_interval.date": "1d"}) }}

What's next?

If you want to see more concrete examples of problems you can solve with Super Search, head to the Examples page. If you want an exhaustive list of parameters, check out the API Reference page.

{% endblock %}