Search¶

With the release of Abaco 1.6.0, a search capability has been introduced using the Mongo aggregation system, full-text searching, and indexing. Searching can be done on actor, worker, execution, and log collections. This feature allows users to search based on any attribute associated with resources that they have permission to view. For example, using search, a user could retrieve all viewable executions with status “ERROR” in one API call. The search currently makes use of logical operators and datetime to allow for easy searching of any object based on any specific field.

Attention

Search in Abaco was implemented in version 1.6.0.

Search has been implemented via query parameters on a new /search/<collection> endpoint. To use it, specify the collection to be searched (actors, workers, executions, or logs) in the URL. With no query arguments Abaco will return all entries in the collection that you have permission to view. To specify query arguments, add a ? to the end of the url and specify the parameters in form <attribute_name>.<operator>=<param_value> separated by &. If not specified, the search defaults the operator to the equality operator (i.e., the eq operator). The general form for requests to the search endpoint looks like:

GET /actors/v2/search/<collection>?<attr_1>.<op_1>=<value_1>&<attr_2>.<op_2>=<value_2>&...

where <attr_1>, <attr_2>, etc. are valid attributes on an instance of <collection> (for example, image for the actors collection), <op_1>, <op_2>, etc. are valid Abaco search operators (see table below), and <value_1>, <value_2>, etc. are values for the attribute type. The response from a search consists of the list of objects of type <collection> that meet the search criteria and that the caller has view access to.

The same query parameters can also be used on the following existing endpoints:

Endpoint	Description
/actors	Search all actors; equivalent to /search/actors
/actors/{aid}/executions	Search all executions for a fixed actor.
/actors/{aid}/executions/{eid}/logs	Search logs for a specific execution.
/actors/{aid}/workers	Search all workers for a fixed actor.

When applied to one of the existing endpoints above, the query parameters can be thought of as filters, refining the set of objects that would have been returned by the listing.

A table of valid operator parameters, their function, and examples are below.

Operator	Function	Examples
eq	Checks if given value is equal to db value matching given key. This is the default operator.	?id.eq=AKY5o4Z847lB3
neq	Checks if given value is not equal to db value matching given key.	?id.neq=AKY5o4Z847lB3
gt	Checks if given value is greater than db value matching given key.	?start_time.gt=2020-04-29+06:00
gte	Checks if given value is greater than or equal to db value matching given key.	?runtime.gte=423
lt	Checks if given value is less than db value matching given key.	?message_received_time.lt=2020
lte	Checks if given value is less than or equal to db value matching given key.	?final_state.FinishedAt.lte=2020-04-29
in	Checks if db value matching given key match any values in the given list of values.	?status.in=[“BUSY”,”REQUESTED”,”READY”]
nin	Checks if db value matching given key does not match any values in the given list of values.	?status.nin=[“COMPLETED”, “READY”]
like	Checks if given value in (through regex) db value matching given key.	?image.like=abaco_docker_username
nlike	Checks if given value not in (through regex) db value matching given key.	?image.nlike=abaco_test
between	Checks if db value matching given key is greater than or equal to first given value, and less than or equal to second given value.	?start_time.between= 2020-04-29T20:15:52:246Z, 2021-06-24-05:00
limit	Sets a limit on total amount of results returned. Defaults to 10 results.	?limit=20
skip	Skips a specified amount of results when returning.	?skip=4

Additionally, the Abaco search supports the following special parameters

Parameter	Function	Examples
search	Completes a fuzzy full-text search based on inputs. Returns results by best accuracy/score.	?search=stringToSearchFor
exactsearch	Completes a full-text search and looks for exact matches with inputs.	?exactsearch=stringToMatchExactly
limit	Limit the number of results to return.	?limit=5
skip	The number of results to skip.	?skip=10

You may use as many search parameters as you want in one query sans limit and skip, where each may only be used once.

Metadata¶

Abaco formats the responses to searches slightly differently from a typical response in the fact that the response to a search returns two objects within the result object: a search object, containing the actual results, and a _metadata object. The _metadata object returns pertinent information about the amount of records returned, the amount of records the return is limited to, the amount of records skipped (specified in query), and the total amount of records that match the query searched for. This is useful to implement paging or to only receive a set amount of records.

Important

A search result object contains a _metadata object and a search object, the latter is a JSON list containing the actual search results.

Example of a Search Response¶

{'message': 'Executions search completed successfully.',
 'result': {'_metadata': {'countReturned': 1,
                         'recordLimit': 10,
                         'recordsSkipped': 0,
                         'totalCount': 1},
             'search': [{'_links': {'logs': 'https://dev.tenants.aloedev.tacc.cloud/actors/v2/joBjeDkWyBwLx/logs',
                                 'owner': 'https://dev.tenants.aloedev.tacc.cloud/profiles/v2/testuser',
                                 'self': 'https://dev.tenants.aloedev.tacc.cloud/actors/v2/joBjeDkWyBwLx/executions/1JKkQwX75vE56'},
                         'actorId': 'joBjeDkWyBwLx',
                         'cpu': 444097006,
                         'executor': 'testuser',
                         'exitCode': 0,
                         'finalState': {'Dead': False,
                                     'Error': '',
                                     'ExitCode': 0,
                                     'FinishedAt': '2020-04-29T21:47:21.385Z',
                                     'OOMKilled': False,
                                     'Paused': False,
                                     'Pid': 0,
                                     'Restarting': False,
                                     'Running': False,
                                     'StartedAt': '2020-04-29T21:47:19.382Z',
                                     'Status': 'exited'},
                         'id': '1JKkQwX75vE56',
                         'io': 716,
                         'messageReceivedTime': '2020-04-29T21:47:18.7Z00',
                         'runtime': 2,
                         'startTime': '2020-04-29T21:47:18.954Z',
                         'status': 'COMPLETE',
                         'workerId': '7kvAAKYKB6Qk6'}]},
 'status': 'success',
 'version': ':dev'}

Inputs¶

All inputs are given to the search function as query parameters and thus are transmitted as strings. Abaco attempts to convert these inputs to the native type associated with the attribute. Strings are left untouched. Booleans are expected to be “False” or “false” and “True” or “true” to be converted. Numbers are all converted to floats. Lists are parsed with json.loads and will accept either ["test"] or ['test'] with post-processing on Abaco’s end to convert to lists.

The last consumed input type is datetime objects. Abaco accepts a broad range of ISO 8601 like strings. An example of the most detailed string accepted is 2020-04-29T20:15:52:246252-06:00. 2020-04-29T20:15:52:246Z, 2020-04-29T20:15:52-06:00, 2020-04-29T20:15-06:00, 2020-04-29T20-06:00, 2020-04-29-06:00, 2020-04Z, and 2020 are also acceptable.

Attention

Abaco stores all times in UTC, so addition of your timezone or conversion to UTC is important. If no timezone information is given (-06:00 or Z (to signal UTC)) the datetime is assumed to be in UTC.

Important

For the purposes of comparison, unspecified elements of a datetime string are set to the minimal possible value. For example, the string “2020-12-30” is greater than “2020-12” which in turn is greater than “2020”. As datetime objects, these are converted to 2020-12-30T00:00:00Z, 2020-12-01T00:00:00Z and 2020-01-01T00:00:00Z, respectively. This holds true until you reach millisecond accurate time.

Note that use of nonces is limited to searching within the actor or alias the nonce was created for. Abaco does not allow the use of nonce on the global search enpoint.

Important

Nonces can be used along side search query parameters by setting the x-nonce parameter as usual; queries should still work as expected and do not need any additional modification. Searches using a nonce count as a nonce “use” as with any other API call using a nonce.

Creating ISO 8601 formatted strings¶

The following examples may be helpful for working with datetime objects in Python.

Python - String with Timezone¶

The following gets the current time as an ISO 8601 formatted string with timezone:

import datetime
import pytz

austin_time_zone = pytz.timezone("America/Chicago")
isoString = datetime.datetime.now(tz=austin_time_zone).isoformat()
print(isoString)

This prints 2020-04-29T16:21:34.602078-05:00.

Python - UTC String¶

The following gets the current UTC time as an ISO 8601 formatted string:

import datetime

isoString = datetime.datetime.utcnow().isoformat()
print(isoString)

This prints 2020-04-29T21:21:34.602078. Feel free to add the Z or leave it absent.

Search Examples¶

In this section we provide some example searches using the new search endpoint as well as query parameters applied to some existing endpoints.

1. Use the new search endpoint to search for all actors defined with image “abacosamples/test”, created since 4/29/2020 and in either “READY” or “BUSY” status:

cURL¶

$ curl -H "Authorization: Bearer $TOKEN" \
https://api.tacc.utexas.edu/actors/v2/search/actors?image=abacosamples/test&create_time.gt=2020-04-29&status.in=["READY", "BUSY"]

Result¶

{'message': 'Search completed successfully.',
'result': {'_metadata': {'countReturned': 1,
                        'recordLimit': 10,
                        'recordsSkipped': 0,
                        'totalCount': 1},
            'search': [{'_links': {'executions': 'https://dev.tenants.aloedev.tacc.cloud/actors/v2/joBjeDkWyBwLx/executions',
                                'owner': 'https://dev.tenants.aloedev.tacc.cloud/profiles/v2/testuser',
                                'self': 'https://dev.tenants.aloedev.tacc.cloud/actors/v2/joBjeDkWyBwLx'},
                        'createTime': '2020-04-29T21:46:53.393Z',
                        'defaultEnvironment': {'default_env_key1': 'default_env_value1',
                                            'default_env_key2': 'default_env_value2'},
                        'description': '',
                        'gid': None,
                        'hints': [],
                        'id': 'joBjeDkWyBwLx',
                        'image': 'abacosamples/test',
                        'lastUpdateTime': '2020-04-29T21:46:53.393Z',
                        'link': '',
                        'maxCpus': None,
                        'maxWorkers': None,
                        'memLimit': None,
                        'mounts': [{'container_path': '/_abaco_data1',
                                    'host_path': '/data1',
                                    'mode': 'ro'}],
                        'name': 'abaco_test_suite_default_env',
                        'owner': 'testuser',
                        'privileged': False,
                        'queue': 'default',
                        'state': {},
                        'stateless': True,
                        'status': 'READY',
                        'statusMessage': ' ',
                        'tasdir': None,
                        'token': 'false',
                        'type': 'none',
                        'uid': None,
                        'useContainerUid': False,
                        'webhook': ''}]},
'status': 'success',
'version': ':dev'}

2. Use the global search endpoint to look for all executions that are not in COMPLETE status across all actors the user has access to.

cURL¶

$ curl -H "Authorization: Bearer $TOKEN" \
https://api.tacc.utexas.edu/actors/v2/search/executions?status.neq="COMPLETE"

Result¶

{
  "message": "Search completed successfully.",
  "result": {
    "_metadata": {
      "countReturned": 4,
      "recordLimit": 100,
      "recordsSkipped": 0,
      "totalCount": 4
    },
    "search": [
      {
        "_links": {
          "logs": "https://api.tacc.utexas.edu/actors/v2/7zxX7DlZ0eeZY/logs",
          "owner": "https://api.tacc.utexas.edu/profiles/v2/testuser",
          "self": "https://api.tacc.utexas.edu/actors/v2/7zxX7DlZ0eeZY/executions/8mzXG1vxDaZZ1"
        },
        "actorId": "7zxX7DlZ0eeZY",
        "cpu": 0,
        "executor": "testuser",
        "exitCode": null,
        "finalState": null,
        "id": "8mzXG1vxDaZZ1",
        "io": 0,
        "messageReceivedTime": "2020-05-05T19:50:23.813Z",
        "runtime": 0,
        "startTime": null,
        "status": "SUBMITTED",
        "workerId": null
      },
      {
        "_links": {
          "logs": "https://api.tacc.utexas.edu/actors/v2/7zxX7DlZ0eeZY/logs",
          "owner": "https://api.tacc.utexas.edu/profiles/v2/testuser",
          "self": "https://api.tacc.utexas.edu/actors/v2/7zxX7DlZ0eeZY/executions/7mYv7rxYNNYw1"
        },
        "actorId": "7zxX7DlZ0eeZY",
        "cpu": 0,
        "executor": "testuser",
        "exitCode": null,
        "finalState": null,
        "id": "7mYv7rxYNNYw1",
        "io": 0,
        "messageReceivedTime": "2020-05-05T19:50:23.296Z",
        "runtime": 0,
        "startTime": null,
        "status": "RUNNING",
        "workerId": "1zLYaONZxWQAX"
      },
      {
        "_links": {
          "logs": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/logs",
          "owner": "https://api.tacc.utexas.edu/profiles/v2/testuser",
          "self": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/executions/jg0oLKJg8VvgM"
        },
        "actorId": "jm6kjmDmW885N",
        "cpu": 0,
        "executor": "testuser",
        "exitCode": null,
        "finalState": null,
        "id": "jg0oLKJg8VvgM",
        "io": 0,
        "messageReceivedTime": "2020-05-05T19:50:20.113Z",
        "runtime": 0,
        "startTime": null,
        "status": "RUNNING",
        "workerId": "7XZN4aqvzoJ33"
      },
      {
        "_links": {
          "logs": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/logs",
          "owner": "https://api.tacc.utexas.edu/profiles/v2/testuser",
          "self": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/executions/jgM7JBmqDDjM5"
        },
        "actorId": "jm6kjmDmW885N",
        "cpu": 0,
        "executor": "testuser",
        "exitCode": null,
        "finalState": null,
        "id": "jgM7JBmqDDjM5",
        "io": 0,
        "messageReceivedTime": "2020-05-05T19:50:20.925Z",
        "runtime": 0,
        "startTime": null,
        "status": "SUBMITTED",
        "workerId": null
      }
    ]
  },
  "status": "success",
  "version": "1.6.0"
}

Find all executions for actor jm6kjmDmW885N that completed after “2020-05-05T19:50:23.748”.

cURL¶

$ curl -H "Authorization: Bearer $TOKEN" \
https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/executions?status=COMPLETE&start_time.gt=2020-05-05T19:50:23.748

Result¶

{
  "message": "Executions search completed successfully.",
  "result": {
    "_metadata": {
      "countReturned": 2,
      "recordLimit": 100,
      "recordsSkipped": 0,
      "totalCount": 2
    },
    "search": [
      {
        "_links": {
          "logs": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/logs",
          "owner": "https://api.tacc.utexas.edu/profiles/v2/testuser",
          "self": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/executions/jg0oLKJg8VvgM"
        },
        "actorId": "jm6kjmDmW885N",
        "cpu": 159212854,
        "executor": "testuser",
        "exitCode": 0,
        "finalState": {
          "Dead": false,
          "Error": "",
          "ExitCode": 0,
          "FinishedAt": "2020-05-05T19:50:29.038Z",
          "OOMKilled": false,
          "Paused": false,
          "Pid": 0,
          "Restarting": false,
          "Running": false,
          "StartedAt": "2020-05-05T19:50:27.003Z",
          "Status": "exited"
        },
        "id": "jg0oLKJg8VvgM",
        "io": 266,
        "messageReceivedTime": "2020-05-05T19:50:20.113Z",
        "runtime": 2,
        "startTime": "2020-05-05T19:50:26.697Z",
        "status": "COMPLETE",
        "workerId": "7XZN4aqvzoJ33"
      },
      {
        "_links": {
          "logs": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/logs",
          "owner": "https://api.tacc.utexas.edu/profiles/v2/testuser",
          "self": "https://api.tacc.utexas.edu/actors/v2/jm6kjmDmW885N/executions/jgM7JBmqDDjM5"
        },
        "actorId": "jm6kjmDmW885N",
        "cpu": 172730092,
        "executor": "testuser",
        "exitCode": 0,
        "finalState": {
          "Dead": false,
          "Error": "",
          "ExitCode": 0,
          "FinishedAt": "2020-05-05T19:50:32.123Z",
          "OOMKilled": false,
          "Paused": false,
          "Pid": 0,
          "Restarting": false,
          "Running": false,
          "StartedAt": "2020-05-05T19:50:30.085Z",
          "Status": "exited"
        },
        "id": "jgM7JBmqDDjM5",
        "io": 396,
        "messageReceivedTime": "2020-05-05T19:50:20.925Z",
        "runtime": 2,
        "startTime": "2020-05-05T19:50:29.723Z",
        "status": "COMPLETE",
        "workerId": "7XZN4aqvzoJ33"
      }
    ]
  },
  "status": "success",
  "version": "1.6.0"
}