Configuring ETL Filters
- Gary Leicht
- Shirly Tsang
Data Connector ETL filters are used to exclude specific information. Using filter expressions from flow configuration page, you can send or drop specific information. After configuring an ETL filter to exclude specific information, you can apply the filter to your traffic flow configuration. The data that has not been excluded or removed will be transferred to the configured destinations. To set up ETL filters, use regexes (regular expressions) for Grid member names and for IP/Network, FQDN, DNS Record Type, OPHID, and ON-PREM HOST.
The following wildcards are supported:
Wildcard | Description | Example |
|---|---|---|
* | Applicable to one or more domain name labels. It can be specified only on the left side of the domain name. | *.foo.com |
# | Applicable to one or more labels for a domain name. Can be specified only on the left side of a domain name. | #./foo.com |
? | Used to specify exactly one label for a domain name. Can be specified on the left or right side of the domain name. | ?.foo.com ?, ?. corp.?. test.? |
Note
- For Threat Class/Property, the supported ETL data filters are processed in the following order: client_ip, member, query FQDN, DNS record type, and Threat Class/Property.
- Data Connector automatically filters out NIOS log messages received from Infoblox Platform. In the past, they were sent to Infoblox Platform by Data Connector.
The stated information is for reference only. It represents the results of lab testing in a controlled environment focused on individual protocol services. Enabling additional protocols, services, cache hit ratio for recursive DNS, and customer environment variables will affect performance. To design and size a solution for a production environment, please contact your Infoblox Solution Architect.
The following details of ETL filters are supported for log types of source data:
ETL filters supported for source data log types | ||||||||
|---|---|---|---|---|---|---|---|---|
Source | Log Type | FQDN | Client IP/Network | Member | DNS Record Type | Ophid | Hostname | Threatclass/Property |
Infoblox Source | Threat Defense Query/Response Log | Yes | Yes | — | Yes | Yes | Yes | — |
Threat Defense Threat Feeds Hits Log | Yes | Yes | — | Yes | Yes | Yes | Yes | |
DDI Query/Response Log | Yes | Yes | — | Yes | Yes | Yes | — | |
DDI DHCP Lease Log | — | Yes | — | — | — | Yes | — | |
NIOS Source | Query/Response Log | Yes | Yes | Yes | Yes | — | — | — |
IPAM Metadata/DHCP Lease Information | — | Yes | — | — | — | — | ||
RPZ Logs | Yes | Yes | Yes | Yes | — | — | — | |
You must configure the NIOS appliance to send syslog messages to an external Data Connector over TCP. By default, the NIOS appliance sends these messages over UDP.
Advisory
The NIOS UI provides a mechanism for filtering the domain names it sends to Cloud Data Connector. Because NIOS sends cache logs, when configuring NIOS for use with Cloud Data Connector, make sure to configure Cloud Data Connector to exclude internal corporate and authoritative domains: *.<corp domains> and *.<Authoritative Zones>. If you exclude corporate and authoritative domains, internal traffic logs will not be added. The complete list of domains to be excluded is listed here as a downloadable test file.
Note that the domains recommended for exclusion can be applied as an ETL filter on Cloud Data Connector or as a list of excluded domains on NIOS.
Image: The configuration panel from NIOS, specifically for setting up DNS properties in relation to a Data Connector.
For more details, see Setting Up the NIOS Grid.
To view all ETL configurations, do the following:
- Log in to the Infoblox Portal.
- Click Configure > Administration > Data Connector.
In the ETL tab, the Infoblox Portal displays the following, for all ETL configurations:
- NAME: the name of the ETL configuration
- DATA TYPE: the filter criterion for the ETL process
- DESCRIPTION: the information about the ETL configuration
- STATE: the indication of whether the configuration is enabled or disabled
Tags: Click Add and specify the following to associate a key with the ETL filter configuration:
KEY: Enter a meaningful name for the key, such as a location or department.
VALUE: Enter a value for the key. For details, see Managing Tags.
ETL Configuration
In the details panel located to the right of the page, you can view the ETL configuration. Click the information icon to open/close the panel.
Filters
Click the filter icon to open the filter options panel. ETL configuration filters can be applied based on Name, Data Type, Description, and State.
Search
Use the search functionality (search box) to conduct a local search based on ETL criterion.
For more information on ETL configurations, see the following: