Document toolboxDocument toolbox

Using a Syslog Server

Syslog is a widely used mechanism for logging system events. NIOS appliances generate syslog messages that you can view through the Syslog viewer and download to a directory on your management station. In addition, you can configure a NIOS appliance to send the messages to one or more external syslog servers for later analysis. Syslog messages provide information about appliance operations and processes. NIOS appliances include syslog messages generated by the bloxTools service. You can choose logging categories to send specific syslog messages. The prefixes in the syslog messages are based on the logging categories you configure in the syslog. Note that syslog messages are prefixed only when you select logging categories. For information about how to configure logging categories, see Specifying Syslog Servers. You can also include audit log messages and specific BIND messages among the messages the appliance sends to the syslog server.
In addition to saving system messages to a remote syslog server, a NIOS appliance also stores the system messages locally. When the syslog file reaches its maximum size, which is 300 MB for Infoblox appliances and VMware virtual appliances, and 20 MB for Riverbed virtual appliances, the appliance automatically writes the file into a new file by adding a .0 extension to the first file and incrementing subsequent file extensions by 1.
Files are compressed during the rotation process, adding a.gz extension following the numerical increment (file.#.gz). The sequential incrementation goes from zero through nine. When the eleventh file is started, the tenth log file (file.9.gz) is deleted, and subsequent files are renumbered accordingly. For example, the current log file moves to file.0.gz, the previous file.0.gz moves to file.1.gz, and so on through file.9.gz. A maximum of 10 log files (0-9) are kept.
You can set syslog parameters at the Grid and member levels. At the member level, you can override Grid-level syslog settings and enable syslog proxy.
You can configure the appliance to back up rotated syslog files to external servers through FTP or SCP. When you do so, the appliance forwards the rotated syslog files to the external servers that you configure. You can configure up to 10 external syslog backup servers each at the Grid and member levels. You can also override the Grid-level server configuration at the member level. For information about configuring syslog backup servers, see Configuring Syslog Backup Servers.
This section includes the following topics:

Specifying Syslog Servers

To configure a NIOS appliance to send messages to a syslog server:

  1. From the Grid tab, select the Grid Manager tab -> Members tab, and then click Grid Properties -> Edit from the Toolbar.
  2. In the Grid Properties editor, select the Monitoring tab, and then complete the following: 
    Syslog
    In addition to storing the syslog on a Grid member, you can configure the Grid to send the log to an external syslog server.
    • Syslog size (MB): Specify the maximum size for a syslog file. Enter a value between 10 and 300. The default is 300.
      When the syslog file reaches the size you enter here, the appliance automatically writes the file into a new file by adding a .0 extension to the first file and incrementing subsequent file extensions by 1.
    • Log to External Syslog Servers: Select this to enable the appliance to send messages to a specified syslog server. Grid Manager displays the current syslog servers in the table. To define a new syslog server, click the Add icon and complete the following:
      • Address: Enter the IP address of the syslog server. Entries may be an IPv4 or IPv6 address.
      • Transport: From the drop-down list, select whether the appliance uses Secure TCP, TCP or UDP to connect to the external syslog server.
      • Server Certificate: Click Select to upload a self-signed or a CA-signed server certificate. In the Upload dialog, click Select and navigate to the certificate file, and then click Upload. Note that this is valid only for Secure TCP transport.
      • Interface: From the drop-down list, select the interface through which the appliance sends syslog messages to the syslog server.
        • Any: The appliance chooses any port that is available for sending syslog messages.
        • LAN: The appliance uses the LAN1 port to send syslog messages.
        • MGMT: The appliance uses the MGMT port if it has been configured. Otherwise, it uses the LAN1 port.
      • Source: From the drop-down list, select which syslog messages the appliance sends to the external syslog server:
        • Any: The appliance sends both internal and external syslog messages.
        • Internal: The appliance sends syslog messages that it generates.
        • External: The appliance sends syslog messages that it receives from other devices, such as syslog servers and routers.
      • Node ID: Specify the host or node identification string that identifies the appliance from which syslog messages are originated. This string appears in the header message of the syslog packet. Select one of the following:
        • LAN: Use the LAN1 IP address of the appliance. For an HA pair, this is the LAN1 address of the active or passive node. This is the default.
        • Host Name: Use the host name of the appliance in FQDN format.
        • IP and Host Name: Use both the FQDN and the IP address of the appliance. The IP address can be the LAN1 or MGMT IP address depending on whether the MGMT port has been configured. Note that if the MGMT port is not configured, the LAN1 IP address is used. 
        • MGMT: Use the MGMT IP address, if the port has been configured. If the MGMT port is not configured, the LAN1 IP address is used. This can be an IPv4 or IPv6 address.
      • Port: Enter the destination port number. The default is 514 for TCP and UDP. For Secure TCP, the default port is 6514.
      • Severity: Choose a severity filter from the drop-down list. When you choose a severity level, the appliance sends log messages with the selected level and the levels above it. The severity levels range from the lowest, debug, to the highest, emerg. For example, if you choose debug, the appliance sends all syslog messages to the server. If you choose err, the appliance sends messages with severity levels err, crit, alert, and emerg.
        • emerg: Panic or emergency conditions. The system may be unusable.
        • alert: Alerts, such as NTP service failures, that require immediate actions.
        • crit: Critical conditions, such as hardware failures.
        • err: Error messages, such as client update failures and duplicate leases.
        • warning: Warning messages, such as missing keepalive options in a server configuration.
        • notice: Informational messages regarding routine system events, such as "starting BIND".
        • info: Informational messages, such as DHCPACK messages and discovery status.
        • debug: Messages that contain information for debugging purposes, such as changes in the latency timer settings and AD authentication failures for specific users.
      • Logging Category: Select one of the following logging categories:
        • Send all: Select this to log all syslog messages, irrespective of categories to which it belongs. When you select this option, the appliance logs syslog messages for all the events, including all DNS and Infoblox related events. However, the syslog messages are not prefixed when you select this option.
        • Send selected categories: Select this to configure logging categories from the list of available logging categories. Use the arrows to move logging categories from the Available table to the Selected table and vice versa. The appliance sends syslog messages for the categories that are in the Selected table. When you select this option, you must add at least one logging category. The syslog messages are prefixed with a category name to which it belongs. Also, the RPZ events logged in the syslog messages uses specific prefixes for the selected categories. Note that the syslog messages are prefixed when you set logging categories for at least one external syslog server, even if you set other external syslog servers as Send All. For information about syslog prefixes, see Syslog Message Prefixes.

Note: The syslog categories you specify here is different from that of logging categories specified in the Logging tab in the Grid DNS Properties or Member DNS Properties editor. The external server preserves contents of the selected categories even when selection is changed from Send all to Send selected categories and vice versa.


        • Click Add to add the extern al syslog server information.
    • Copy Audit Log Messages to Syslog: Select this for the appliance to include audit log messages it sends to the syslog server. This function can be helpful for monitoring administrative activities on multiple appliances from a central location.
      • Syslog Facility: This is enabled when you select Copy audit log messages to syslog. Select the facility that determines the processes and daemons from which the log messages are generated.

3. Save the configuration and click Restart if it appears at the top of the screen.

Syslog Message Prefixes

You can configure the syslog external backup servers to send (archive) syslog files to different destinations by their logging categories. This allows you to split syslog files based on the service and efficiently perform troubleshooting. For example, you can archive all DNS related logs on Server 1, and all DHCP related logs on Server 2. For information about how to configure an external syslog backup server, see Specifying Syslog Servers.
When you select the Send selected categories option, the syslog messages are prefixed with a category name to which it belongs.

For syslog message prefixes to be enabled, you must check the Log to External Syslog Servers check box in Grid Properties > Monitoring. Also, the external syslog server (which can be a virtual or a physical server) must have at least one of the syslog categories selected instead of the Send all option selected in the Logging Category field. 


Note: When you set Send all in the Logging Category, the appliance logs syslog messages for all the events and they are not prefixed. The syslog messages are prefixed even if one external syslog server is set with the Send selected categories option.


Following are the prefixes used for different logging categories:

  • DNS Logging Categories: All DNS related messages use the following prefixes: client, config, database, dnssec, general, lame_servers, network, notify, queries, query_rewrite, resolver, responses, rpz, security, update, update_security, xfer_in, and xfer_out.

Sample syslog message for queries:

2014-10-27T08:15:49+00:00 daemon ib-10-35-117-12.infoblox.com named[1923]: info
queries: client 10.35.117.12#55190 (1.0.0.127.in-addr.arpa): query:
1.0.0.127.in-addr.arpa IN PTR +E (10.35.117.12)

Sample syslog message for xfer-out:

2014-10-10T06:44:09+00:00 daemon infoblox.localdomain named[17630]: info xfer-out:
client 10.120.20.157#58275 (zone.com): transfer of 'zone.com/IN': AXFR started

  • ADP: All Infoblox related messages use prefix adp.

Note: There is no prefix for RPZ syslog messages that does not belong to the DNS or ADP category.


  • DHCP: All DHCP related messages use the following prefixes: dhcpd, omshell, dhcrelay, and dhclient.

Sample syslog message for dhcp:

Sep 4 09:23:44 10.34.6.28 dhcpd[20310]: DHCPACK on 70.1.20.250 to fc:5c:fc:5f:10:85 via

eth1 relay 10.120.20.66 lease-duration 600

  • DTC: All DTC related messages use the following prefixes: idns_healthd and idnsd

Sample syslog message for idns_healthd:

Sep 3 12:12:35 10.34.6.30 idns_healthd[1220]: resource health status [Monitor 'icmp'

(ICMP, port 0) checked server 's1' (IP 10.34.6.23), status: IPv4=ONLINE]

  • Cloud: All cloud related messages use prefix cloud_api.

Sample syslog message for cloud_api:

Sep 4 10:53:30 10.34.6.32 cloud_api[5354]: [admin]: Login_Allowed - -

to=Serial\040Console apparently_via=Remote ip=10.120.20.66 auth=Local

group=.admin-group

  • NTP: All NTP related messages use prefix ntpd.

Sample syslog message for NTP:

Sep 28 06:57:21 10.35.116.7 ntpd[12186]: precision = 0.053 usec

Sep 28 06:57:21 10.35.116.7 ntpd[12186]: Listening on interface #0 wildcard, 0.0.0.0#123

Disabled

  • File Distribution: All File Distribution related messages use the following prefixes: ftpd and tftp

Sample syslog message for TFTP:

Sep 3 13:03:09 10.34.6.30 monitor[23623]: Type: TFTP, State: Red, Event: A TFTPD daemon

failure has occurred

  • Authentication: All Authentication related messages use the following prefixes: auth, authpriv, AD, and radiusd.

Sample syslog message for RADIUS authentication:

Sep 28 10:09:55 10.35.116.4 httpd: 2015-09-28 10:09:55.912Z [user1]: Login_Allowed - -

to=AdminConnector ip=10.120.253.227 auth=RADIUS group=admin-group apparently_via=GUI

  • Microsoft Integration: All Microsoft Integration related messages use the following prefixes: dns_server, connect_status, dns_zone, dhcp_server, dhcp_leases, clear_lease, ad_site, and ad_users.

Sample syslog message for microsoft integration:

dns_server:

Sep 7 09:46:17 10.34.22.20 mssyncd[22315]: dns_server address 10.102.30.157 : Conflict

in property Forwarders: NIOS value (property=<NULL IP array>) and Microsoft value

(property={10.0.2.35, 10.0.2.60}). Resolved by using the Microsoft value

dhcp_server:

Sep 7 10:08:48 10.34.22.20 mssyncd[22316]: dhcp_server address 10.102.30.157 : Couldn't

open RPC interface <MS-WKST>: an instance of a named pipe cannot be found in the listening

state

Sep 7 10:08:48 10.34.22.20 mssyncd[22317]: dns_server address 10.102.30.157 : Opened

RPC interface <MS-WKST> as user 'ad-15\frtest'

IP Address Used in the Syslog Configuration File

The following table describes which IP address the appliance uses as the node ID in the syslog configuration file, provided that the MGMT port has been configured. If the MGMT port is not configured, the LAN1 IP address is always used regardless of the configuration.
Table 37.1 IP address Used in Syslog Config File when MGMT Port is Configured

Interface

Node ID

IP used in syslog configuration file

Any

MGMT

MGMT IP address

Any

IP and Host Name

MGMT IP address

MGMT

MGMT

MGMT IP address

MGMT

IP and Host Name

MGMT IP address

LAN

MGMT

LAN1 IP address

LAN

IP and Host Name

LAN1 IP address

Configuring Syslog Backup Servers

You can configure external syslog backup servers to forward rotated syslog files. You can configure up to 10 external syslog backup servers.
To configure external backup servers:

  1. Grid: From the Grid tab -> Grid Manager tab, expand the Toolbar and click Grid Properties -> Edit.
    Member: From the Grid tab -> Grid Manager tab, click the Members tab, select the member check box, and click the Edit icon.
  2. Grid: In the Grid Properties editor, select the Syslog Backup tab.
    Member: In the Grid Member Properties editor, select the Syslog Backup tab and then click Override to override the Grid-level settings.
    Complete the following to modify backup server settings:
    • Address: Enter the IP address of the external backup server. You are not allowed to configure more than one server using the same IP address at the same level (Grid or member). However, you can use the same server IP address at different levels (Grid or member). Note that you cannot modify the IP address for the overridden server.
    • Protocol: Select SCP or FTP from the drop-down list.
    • Port: Enter the destination port number. The default port is 20 for FTP and 22 for SCP.
    • Path: Enter the directory path for the syslog file.
    • Username: Enter the username of your FTP or SCP account.
    • Password: Enter the password of your FTP or SCP account. If you do not change the password of the overridden server, then make sure that you use the same password specified at the Grid level.
    • Enabled: Select this check box to enable the FTP or SCP server. The appliance forwards the rotated syslog files to the external servers that you configure only after you select this check box. Clear the check box to disable the server.

3. Click Save and Close.

Configuring Syslog for Grid Me mbers

You can override Grid-level syslog settings and enable syslog proxy for individual members. When you enable syslog proxy, the member receives syslog messages from specified devices, such as syslog servers and routers, and then forwards these messages to an external syslog server. You can also enable appliances to use TCP for sending syslog messages. Using TCP is more reliable than using UDP; this reliability is important for security, accounting, and auditing messages sent through the syslog. Note that you cannot enable syslog proxy for Grid members, if they are configured on a Grid Master.
To configure syslog parameters for a member:

  1. From the Grid tab, select the Grid Manager tab -> Members tab -> member check box, and then click the Edit icon.
  2. In the Grid Member Properties editor, select the Monitoring tab -> Basic tab, click Override in the Syslog section, and then complete the fields as described in Specifying Syslog Servers.
    In addition to storing the system log on a Grid member, you can configure a member to send the log to a syslog server.
  3. Select the Advanced tab and complete the following:
    • Enable syslog proxy: Select this to enable the appliance to receive syslog messages from other devices, such as syslog servers and routers, and then forward these messages to an external syslog server.
      • Enable listening on TCP: Select this if the appliance uses TCP to receive messages from other devices. Enter the number of the port through which the appliance receives syslog messages from other devices. 
      • Enable listening on UDP: Select this if the appliance uses UDP to receive messages from other devices. Enter the number of the port through which the appliance receives syslog messages from other devices.
    • Proxy Access Control: Select one of the following to configure access control when receiving syslog messages from specific syslog servers or routers:
      • None: Select this if you do not want to configure syslog proxy. When you select this option, none of the devices can send syslog messages to the appliance. This is selected by default.
      • Named ACL: Select this and click Select Named ACL to select a named ACL that contains only IPv4 addresses and networks. This does not support TSIG key based ACEs. When you select this, the appliance permits clients that have Allow permission in the named ACL to allow syslog messages from specific syslog servers or routers. You can click Clear to remove the selected named ACL.
      • SetofACLs: Select this to configure individual access control entries (ACEs). Click the Add icon and select one of the following from the drop-down list. Grid Manager adds a row to the table.
        • IPv4 Address or IPv6 Address: Select this to add an IPv4 or IPv6 address entry. Click the Value field and enter the address. The default permission is Allow, which means that the appliance allows access to and from this device. You can change this to Deny to block access.
        • IPv4 Network or IPv6 Network: Select this to add an IPv4 or Ipv6 network entry. Click the Value field and enter the network. The default permission is Allow, which means that the appliance allows syslog messages sent by this network. You can change this to Deny to block access.
        • Any Address/Network: Select this to allow or deny access to all IPv4 and IPv6 addresses and networks. The default permission is Allow, which means that the appliance allows syslog messages sent by all addresses and networks. You can change this to Deny to block access.

After you have added access control entries, you can do the following:

        • Select the ACEs that you want to group and put into a named ACL. Click the Create new named ACL icon and enter a name in the Convert to Named ACL dialog box.
        • Reorder the list of ACEs using the up and down arrows next to the table.
        • Select an IPv4 network and click the Edit icon to modify the entry.
        • Select an ACE and click the Delete icon to delete the entry. You can select multiple ACEs for deletion.

4. Save the configuration and click Restart if it appears at the top of the screen.

Setting DNS Logging Categorie s

You can specify logging categories you want the syslog to capture. Furthermore, you can filter these messages by severity at the Grid and member levels. For information about severity types, see Specifying Syslog Servers.
To specify logging categories:

  1. From the Data Management tab, select the DNS tab, and then click Grid DNS Properties from the Toolbar.
    or
    From the Data Management tab, select the DNS tab -> Members tab -> Grid_member check box, and then click the Edit icon.
  2. In the Grid DNS Properties or Member DNS Properties editor, click Toggle Expert Mode if the editor is in the basic mode, select the Logging tab, and then complete the following:
    • Logging Facility: Select a facility from the drop-down list. This is the location on the syslog server to which you want to sort the DNS logging messages.
    • Logging Category: Select one or more of these log categories:
      • general: Records the BIND messages that are not specifically classified.
      • client: Enables the logging of messages related to query processing, but not the queries themselves. Examples of messages include exceeding recursive client quota, and other errors related to recursive clients, blacklist and NXDOMAIN interception, query name rewrite, and others.
      • config: Records the configuration file parsing messages.
      • database: Records BIND's internal database processes.
      • dnssec: Records the DNSSEC-signed responses.
      • lame servers: Records bad delegation instances.
      • network: Records the network operation messages.
      • notify: Records the asynchronous zone change notification messages.
      • queries: Records the DNS queries. Note that enabling the logging of queries and responses will significantly affect system performance. Ensure that your system has sufficient CPU capacity before you enable DNS query logging.
      • rate-limit: Logs RRL (Response Rate Limiting) events. You must enable RRL in order for the appliance to log RRL events to this logging category.
      • resolver: Logs messages related to outgoing queries from the 'named' process, when it is acting as a resolver on behalf of clients.
      • responses: Records DNS responses. Note that enabling the logging of queries and responses will significantly affect system performance. Ensure that your system has sufficient CPU capacity before you enable DNS response logging.
      • rpz: Records log messages when responses are modified through RPZs or for which explicit passthrus were invoked in the RPZs. This check box is not selected by default.
      • security: Logs miscellaneous messages that are related to security, such as denial or approval (mostly denial) of certain operations.
      • transfer-in: Records zone transfer messages from the remote name servers to the appliance.
      • transfer-out: Records zone transfer messages from the NIOS appliance to remote name servers.
      • update: Records the dynamic update instances.
      • update-security: Records the security updates.
      • DTC load balancing: Records information about which client is directed to which server.
      • DTC health monitors: Records any changes to the health state of a monitored server.

3. Save the configuration and click Restart if it appears at the top of the screen.

Viewing the Syslog

  1. From the Administration tab, select the Logs tab -> Syslog tab.
  2. From the drop-down list at the upper right corner, select the Grid member on which you want to view the syslog.
  3. Optionally, use the filters to narrow down the system messages you want to view. Click Show Filters to enable the filters. Configure the filter criteria, and then click Apply.
    Based on your filter criteria (if any), Grid Manager displays the following in the Syslog viewer:
    •  : The Action icon column is displayed only when you have installed the RPZ license. Click this to view threat details in the RPZ Threat Details dialog box. For information, see Viewing the RPZ Threat Details.
    • Timestamp: The date, time, and time zone of the log message. The time zone is the time zone configured on the member.
    • Facility: The location on the syslog server that determines the processes and daemons from which the log messages are generated.
    • Level: The severity of the message. This can be ALERT, CRITICAL, DEBUG, EMERGENCY, ERROR, INFO, NOTICE, or WARNING.
    • Server: The name of the server that logs this message, plus the process ID.
    • Message: Detailed information about the task performed. For Cloud Network Automation, this contains comma separated values of the admin, source, action, object, object type and message values. Note that source is defined only if the cloud API request was proxied by the Cloud Platform Appliance. The format for this field is proxied from:host,IP where host and IP are the host name and IP address of the proxy.

Note: If the selected member is an HA pair, Grid Manager displays the syslog in two tabs — Active and Passive.
Click the corresponding tab to view the syslog for each node.


Viewing the RPZ Threat Details

Make sure that DNS resolution is enabled and running properly on the member to view threat details. To view threat details for the RPZ zones being queried, complete the following:

  1. From the Administration tab, select the Logs tab -> Syslog tab.
  2. Click the Action icon  and select View Threat Context to open the RPZ Threat Details dialog. The View Threat Context option is disabled if there is no RPZ rule.
    • RPZ Rule: Displays the name of the RPZ rule.
    • First Identified: The date and timestamp of the first occasion that the threat was detected.
    • Short Description: The brief description of the threat.
    • Description: The description of the RPZ rule.

Note: The RPZ Threat Details dialog box may display Unknown if threat is unknown or Unavailable if threat is known and threat details are not available.


3. Click the Close icon to close the RPZ Threat Details dialog.

You can also do the following in the Syslog viewer:

  • Toggle between the single line view and the multi-line view for display.
  • Navigate to the next or last page of the file using the paging buttons.
  • Refresh the syslog output with newly logged messages.
  • Click the Follow icon to have the appliance automatically refresh the log every five seconds.
  • Clear the contents of the syslog.
  • Use filters and the Go To function to narrow down the list. With the autocomplete feature, you can just enter the first few characters of an object name in the Go to field and select the object from the possible matches.
  • Create a quick filter to save frequently used filter criteria. 
  • To filter Microsoft synchronization related events, click Show Filter, select Server from the first drop-down list, and select MS_Server from the drop-down list in the value field. This filter displays entries that begin with the prefix ms. To view values that belong to a specific Microsoft server, you must specify either the name or IP address of a given Microsoft server in the Message field. When you filter the syslog for a specific Grid member, it displays the log entries of Microsoft servers that are assigned to the respective Grid member when the entries are logged.
  • Print the report or export it in CSV format.
  • Bookmark the syslog page.

Searching in the Syslog

Instead of paging through the syslog to locate messages, you can have the appliance search for syslog messages with certain text strings. To search for specific messages:

  • Enter a search value in the search field below the filters, and then click the Search icon.
    The appliance searches through the syslog and highlights the search value in the viewer. You can use the arrow keys next to the Search icon to locate the previous or next message that contains the search value.

Downloading the Syslog File

You can download the syslog file to a specified directory, if you want to analyze it later.

  1. From the Administration tab, select the Logs tab -> Syslog tab, and then click the Download icon.
  2. Navigate to a directory where you want to save the file, optionally change the file name (the default names are node_1_sysLog.tar.gz and node_2_sysLog.tar.gz), and then click OK. If you want to download multiple syslog files to the same location, rename each downloaded file before downloading the next.

Note: If your browser has a pop-up blocker enabled, you must turn off the pop-up blocker or configure your browser to allow pop-ups for downloading files.