Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

TIDE bulk export service endpoints


The TIDE bulk data export API allows for the accessing of active threats using the TIDE API and running the TIDE bulk data export API.

  • To access active threats, use tide/api/data/threats/state/ and specify a provider organization using the "profile" query string parameter.
  • The TIDE bulk data export API requires an endpoint to fetch the bulk threat data and allows specifying the “rlimit” query string parameter for limiting returned records. Note: The rlimit is set to a maximum of 100 responses.
  • The authorization for this process is via the gateway, and the expected response is 200 OK with a file location provided.


Request: 

GET /tide/bulk-export/threats?type=<type>&rlimit=<limit>

Headers: 
AuthContext

Note: This authorization is via the gateway.

Expected response: 
200 OK

{
“data_stream_location”:“https://csp.infoblox.com/tide/bulk-export/file-access/>”
}

The file will be internally uploaded to an S3 bucket specifically dedicated to the client, using the file-id as the object name.

Sample request: 

GET 'https://csp.infoblox.com/tide/api/data/bulk-export/threats?rlimit=10&type=host'

Sample response: 

"data_stream_location": "https://csp.infoblox.com/tide/api/data/bulk-export/data-stream?file_id=ba06742e-8006-4171-89b0-29a641dc04f0"

Note: Run the call again, this time using only the file id (ba06742e-8006-4171-89b0-29a641dc04f0)

""file_uploaded": true,
"pre_auth_url": "https://tide-prod-bulk-export.s3.amazonaws.com/1049207/ba06742e-8006-4171-89b0-29a641dc04f0/date.json?X-Amz-Algorithm=AW%4-
"pre_auth_url_expiry_nanosecs": "9999999999999",
"pre_auth_url_expiry": "2024-07-10 01:27:39.84706186 +0000 UTC m=+16242.785383444"

Note: When the generated url is clicked, the file can be directly updated.

Expected Response Errors

  1. 401 Unauthorized: This error occurs when the tide-ng-atk-gateway is unable to validate the user's token, indicating an authentication issue.
  2. 400 Bad Request: Encountered if the request lacks a valid type or presents incorrect parameters, suggesting a client-side input error.
  3. 500 Internal Server Error: Triggered by issues such as data filtering errors or problems with S3 IO operations, pointing to server-side complications.

Exception Handling

  1. When attempting to retrieve materialized files, an IOException occurs if the specified key is not found on S3. This error is communicated to the user as a 500 internal server error.
  2. The process of uploading generated files containing threat data to S3 involves multipart upload sessions. These sessions either conclude successfully or fail entirely; there are no partial successes.
  3. Should the client's S3 bucket be nonexistent, one will be created accordingly.

Endpoint to access the threat data from the uploaded file

Sample request: 

GET  'https://csp.infoblox.com/tide/bulk-export/file-access/<file-id≥

' --header 'Authorization: Token <token>'

Response from tide 

{
 Status: 400
} 
{
 Status: 200
 “s3://…../tide/bulk-export/<clientId>/<fileId>”
} 

{
 Status: 200
 Redirection_url: “s3://…../tide/bulk-export/<clientId>/<fileId>”
}

Note: Implementing Endpoint Handlers in Tide-ng-atk-gateway and Managing Redirection for File Access.

Expected Response Errors

  1. 401 Unauthorized: Occurs when the tide-ng-atk-gateway is unable to verify user authentication.

  2. 400 Bad Request: Triggered if the request lacks a valid fileID.

  3. 404 Not Found: Indicates that the file does not exist.

    1. Either the file has not been uploaded yet, retry after 20 seconds. The message will specify this.
    2. Or, re initiate the bulk upload.

  4. 403 Forbidden: This error is returned if the file does not belong to the client (note: this may also qualify as a 400 Bad Request).

  5. 500 Internal Server Error: Arises in cases where there's an error generating the pre-signed URL.

  • No labels