Generate logs for troubleshooting the infrastructure agent

Problem

When troubleshooting your infrastructure agent, generate verbose logs for a few minutes to find and investigate errors. This can be useful for your own troubleshooting or when working with New Relic Support.

Important

Verbose logging generates a lot of data very quickly. When finished generating logs, be sure to set verbose: 0 to reduce disk space consumption.

If you have New Relic's infrastructure agent 1.4.0 or higher, you can automate this process by using the newrelic-infra-ctl command. For more information, see the troubleshooting binary documentation.

Solution

Generating verbose log files requires editing your configuration file. For a sample config file that includes all applicable settings, see the example template. To generate detailed logs:

Step	Procedures
	Edit your `newrelic-infra.yml` file: Enable verbose logging: `verbose: 1`. (If you use a containerized infrastructure agent on CoreOS, see system-specific notes.) Set `log_file` to a convenient log file location. Restart the agent so the agent notices the new settings.
	Let your host run at normal load for about three minutes to generate sufficient logging data.
	Return your settings to default: Disable verbose logging by setting `verbose: 0` in `newrelic-infra.yml`. Optional: Disable logging to a custom file by removing the `log_file` line from `newrelic-infra.yml`. Restart the agent so the agent notices the new settings.
	Examine the log file for errors. If you need to send your log file to New Relic Support: Include the line in your log file that contains the agent version: `New Relic infrastructure agent version X.YY.ZZZ` Attach the log file to your support ticket, along with your `newrelic-infra.yml`.

Smart verbose mode

Sometimes errors don't occur until after quite some time has passed. This makes debugging difficult, because typically verbose logs are only enabled for a short period time; otherwise there will be many debug logs. For example, if an error occurs one hour after the infrastructure agent has started, getting debug logs around the time of the error can be tricky or impractical.

As of infrastructure agent v1.9.0 or higher, you can use smart verbose mode for logs. Smart verbose mode only logs the most recent debug messages after an error has been logged. This allows you to leave smart verbose mode running until an error occurs, without logging lots of irrelevant debug messages, and only logging the most recent debug messages. (The number of messages is determined by your configuration.)

For more information on smart verbose mode, see the Infrastructure agent logging behavior docs, and use the Infrastructure configuration settings documentation for details on how to enable smart verbose mode.

Forward the agent logs to New Relic Logs

The Infrastructure agent can be configured to send its own logs to New Relic Logs. This can be useful for troubleshooting issues with log forwarding, the Infrastructure agent, or when contacting support.

For details on how to enable log forwarding for the Infrastructure agent, see Troubleshoot log forwarding.

Notes for specific systems

These are some additional notes and requirements for specific systems, used to supplement the general logging instructions:

If you are using a containerized infrastructure agent on CoreOS:

Choose one of these options to change the log level to verbose:
- Recommended: Set the environment variable NRIA_VERBOSE to 1. Running this on the command line would look like:
  -e NRIA_VERBOSE=1
  OR
- Edit the config file to set verbose: 1. (Editing the config file in a container is not recommended, because it requires rebuilding the image twice: once to add verbose logging and once to remove it.)

Use journalctl to collect the logs:

journalctl -u newrelic-infra > newrelic-infra.log

Set the verbose logging level back to 0 after collecting logs for a few minutes.