Document toolboxDocument toolbox

Configuring ETL Filters

Owned by Shirly Tsang
Last updated: Sept 06, 2024 by Gary Leicht
3 min read

Data Connector ETL filters are used to exclude specific information. Using filter expressions from flow configuration page, you can send or drop specific information. After configuring an ETL filter to exclude specific information, you can apply the filter to your traffic flow configuration. The data that has not been excluded or removed will be transferred to the configured destinations. To set up ETL filters, use regexes (regular expressions) for Grid member names and for IP/Network, FQDN, DNS Record Type, OPHID, and ON-PREM HOST.

The following wildcards are supported:

Wildcard

Description

Example

*

Applicable to one or more domain name labels. It can be specified only on the left side of the domain name. 

*.foo.com

#

Applicable to one or more labels for a domain name. Can be specified only on the left side of a domain name. 

#./foo.com

?

Used to specify exactly one label for a domain name. Can be specified on the left or right side of the domain name. 

?.foo.com

?, ?.

corp.?.

test.? 

Note

  • For Threat Class/Property, the supported ETL data filters are processed in the following order: client_ip, member, query FQDN,  DNS record type, and Threat Class/Property.
  • Data Connector automatically filters out NIOS log messages received from Infoblox Platform. In the past, they were sent to Infoblox Platform by Data Connector.

The stated information is  for reference only. It represents the results of lab testing in a controlled environment focused on individual protocol services. Enabling additional protocols, services, cache hit ratio for recursive DNS, and customer environment variables will affect performance. To design and size a solution for a production environment, please contact your Infoblox Solution Architect.

The following details of ETL filters are supported for log types of source data:

ETL filters supported for source data log types

Source

Log Type

FQDN

Client IP/Network 

Member

DNS Record Type

Ophid

Hostname

Threatclass/Property

Infoblox Source




Threat Defense Query/Response Log

Yes

Yes

—

Yes

Yes

Yes

—

Threat Defense Threat Feeds Hits Log

Yes

Yes

—

Yes

Yes

Yes

Yes

DDI Query/Response Log

Yes

Yes

—

Yes

Yes

Yes

—

DDI DHCP Lease Log

—

Yes

———

Yes

—

NIOS Source



Query/Response Log

Yes

Yes

Yes

Yes

———

IPAM Metadata/DHCP Lease Information

—

Yes

—
———

RPZ Logs

Yes

Yes

Yes

Yes

———

You must configure the NIOS appliance to send syslog messages to an external Data Connector over TCP. By default, the NIOS appliance sends these messages over UDP.

Advisory

The NIOS UI provides a mechanism for filtering the domain names it sends to Cloud Data Connector. Because NIOS sends cache logs, when configuring NIOS for use with Cloud Data Connector, make sure to configure Cloud Data Connector to exclude internal corporate and authoritative domains: *.<corp domains> and *.<Authoritative Zones>. If you exclude corporate and authoritative domains, internal traffic logs will not be added. The complete list of domains to be excluded is listed here as a downloadable test file.

Note that the domains recommended for exclusion can be applied as an ETL filter on Cloud Data Connector or as a list of excluded domains on NIOS.

The screenshot shows the Member DNS Properties window, where the Toggle Basic Mode switch is on and the Logging tab is selected. In the Logging tab, the Advanced tab is open and shows the Advanced settings.
Image: The configuration panel from NIOS, specifically for setting up DNS properties in relation to a Data Connector.


For more details, see Setting Up the NIOS Grid. 

To view all ETL configurations, do the following:

  1. Log in to the Infoblox Portal.
  2. Click Configure > Integrations > Data Connector.

  3. In the ETL Configuration tab, the Infoblox Portal displays the following, for all ETL configurations:

      • NAME: the name of the ETL configuration
      • DATA TYPE: the filter criterion for the ETL process
      • DESCRIPTION: the information about the ETL configuration
      • STATE: the indication of whether the configuration is enabled or disabled

      • Tags: Click Add and specify the following to associate a key with the ETL filter configuration:

        • KEY: Enter a meaningful name for the key, such as a location or department.  

        • VALUE: Enter a value for the key. For details, see Managing Tags.

ETL Configuration

In the details panel located to the right of the page, you can view the ETL configuration. Click the information icon to open/close the panel.

Filters

Click the filter icon to open the filter options panel. ETL configuration filters can be applied based on Name, Data Type, Description, and State. 

Use the search functionality (search box) to conduct a local search based on ETL criterion. 


For more information on ETL configurations, see the following: