Stream logs using Kinesis Data Firehose

If your log data is already being monitored by Amazon CloudWatch Logs, you can use our Kinesis Data Firehose integration to forward and enrich your log data in New Relic. Kinesis Data Firehose is a service that can stream data in real time to a variety of destinations, including our platform.

Forwarding your CloudWatch Logs or other logs compatible with a Kinesis stream to New Relic will give you enhanced log management capabilities to collect, process, explore, query, and alert on your log data.

Create the delivery stream for New Relic

To forward your logs from Kinesis Data Firehose to New Relic:

1. Make sure you have a New Relic license key.

2. Follow the procedures to create an Amazon Kinesis Data Firehose delivery stream, and select New Relic as the destination: In the AWS Management Console, go to Amazon Kinesis.

3. Select Kinesis Data Firehose and click Create delivery stream.

4. Enter a name for the stream and select your data source.

5. When selecting the destination, click Third-party partner and then New Relic.

   

6. Under HTTP endpoint URL, select New Relic logs - US from the dropdown.

   Note: To send your logs to the EU, complete the remaining steps in this section, then proceed to the configuration procedures for EU accounts.

7. Paste your license key in the API key field.

8. Ensure that Content encoding is set to GZIP.

9. Set a buffer size of 1 MiB under Buffer hints/Buffer size (note that the Buffer hints section is folded by default). See the buffer settings section for a more thorough explanation.

10. Configure and review the remaining metadata settings.

Any optional key/value pairs you add in the AWS Management Console will result in attribute/value pairs you can use in New Relic.

Kinesis Data Firehose includes these key-value pairs in each HTTP call. These Kinesis Firehose Parameters can help you identify and organize your destinations.

Configure your stream to send logs to EU accounts

If you have a New Relic account in our EU data center:

1. Follow standard procedures to create your delivery stream.

2. Once your stream has been created, navigate to the Amazon Kinesis Console, and click Delivery streams.

3. Click the name of your newly created delivery stream, and select the Configuration tab.

4. Next to the Destination settings section, click Edit.

5. Update the value for HTTP endpoint URL to our Firehose endpoint for EU accounts:

   https://aws-api.eu.newrelic.com/firehose/v1

   Copy

6. Click Save changes to ensure your stream is updated.

View log data

If everything is configured correctly and your data is being collected, you should see data logs in both of these places:

• New Relic Logs UI
• New Relic tools for running NRQL queries. For example, you can execute a query like this:

SELECT * FROM Log

If no data appears after you enable our log management capabilities, follow our standard log troubleshooting procedures.

Special note on the buffer configuration

When selecting our Kinesis Firehose integration for Logs in AWS, the wizard hides a small section called Buffer hints, at the bottom of the screen. This section contains two very important configuration options: Buffer size and Buffer interval.

The default values are 5 MiB for Buffer size, and 60 seconds for Buffer interval. This means that Firehose accumulates logs until either they contain 5 MiB of data, or until 60 seconds have passed since the last time they were flushed to New Relic. These default settings are not appropriate for New Relic, we strongly advise to use 1 MiB as the Buffer size and activating GZIP body compression.

Why do we recommend this? New Relic accepts at most payloads of 1 MB (1.000.000 bytes) for an individual HTTP POST request to its Logs API. If during a given 60 second accumulation period the size of the accumulated logs exceeds 1 MB, we will reject those logs with a 413 HTTP error. Note that the delivery stream allows configuring, at a minimum, a buffer size of 1 MiB (1.048.576 bytes). Despite this value is slightly greater than the 1MB allowed by the New Relic Logs API, by using GZIP compression the resulting payload becomes smaller than the 1MB limit.

What's next?

Explore logging data across your platform with the New Relic One UI.

• Get deeper visibility into both your application and your platform performance data by forwarding your logs with our logs in context capabilities.
• Set up alerts.
• Query your data and create dashboards.

Disable log forwarding

To disable log forwarding capabilities, follow standard procedures in Kinesis Data Firehose documentation. You do not need to do anything else in New Relic.