Forward logs from Google Cloud Platform

We support two different methods to forward your logs from a Google Cloud Platform Pub/Sub topic to New Relic.

Select an option

The following may help you decide which option is best for your business needs.

GCP log forwarding options	Considerations
Headerless API	Best suited for low log volumes, since it performs one API call for each log record it sends. Does not incur any extra cost to your GCP subscription. If your log volume increases, this solution might hit the quota limits of your New Relic account.
Dataflow job	Best suited for larger log volumes, since it groups log records in batches before sending them to New Relic. Reduces the number of API calls and allows you to reduce your quota usage. May incur extra costs at your GCP subscription, due to the execution of a Dataflow job in your premises.

Use headerless API

To send your GCP logs to New Relic using our headerless API:

Start by creating an ingest URL for your GCP Pub/Sub topic.

Navigate to Logs in New Relic One, and click Add more data sources.
Click Google Cloud Platform, then select your New Relic account where you want to forward logs, and click Continue.
Optional: Configure metadata (attribute-value pairs) to be included in every log event sent to the ingest URL you will generate in the next step.
Click Generate URL.
Copy your newly generated ingest URL and keep it in a safe place.
You will use your new ingest URL to configure a Pub/Sub topic that sends logs to New Relic.

Once you have created your Pub/Sub topic, create a subscription for it.

Return to the GCP Pub/Sub Console.
Click the Pub/Sub topic you previously created.
Scroll down and select the Subscriptions tab, then click Create Subscription and select Create a simple subscription.
Enter a Subscription ID. Then, under Delivery Type, select Push.
In the Endpoint URL field, paste the ingest URL you previously generated.
Configure remaining settings as needed, and click Create.

To finish setup, create a routing sink for your GCP Pub/Sub topic that will forward your logs to New Relic.

Navigate to the GCP Logs Router Console.
Click Create Sink.
Enter a Sink name and Sink description, then click Next.
Under Select sink service, select Cloud Pub/Sub topic, and select the Pub/Sub topic you previously created.
Configure remaining filters as needed, then click Create sink to complete setup.

Use Dataflow job

To send your GCP logs to New Relic using a Dataflow job, you will use our Dataflow template. Before you begin, make sure you have the following tools on your local computer:

A Unix terminal for Linux or macOS
Git
Java JDK 8
Apache Maven 3.2 or higher. We've seen earlier versions fail during the compilation process.
The Google Cloud SDK, which includes gcloud and gsutil command line tools

Run the following command and follow the prompts to log into GCP and select your cloud project:

gcloud init

Using the wizard, you will select a cloud project to use, and you will be able to optionally select a default compute region and zone for the resources you create using gcloud or gsutil. We will not presume a default project, location or region for the following commands.

Clone the DataflowTemplates GitHub repository using the following command:
```
git clone https://github.com/newrelic-forks/DataflowTemplates.git
```
Enter the directory you just created:
```
cd DataflowTemplates
```
Then continue with the next section to execute additional commands.

To set the required configuration to compile and run the Dataflow forwarder, run the following commands in your DataflowTemplates directory. The only required values are:

PROJECT_ID
BUCKET_NAME
NR_LICENSE_KEY
INPUT_SUBSCRIPTION_NAME
You can leave the other default values as is.
bash
```
$# The Google Cloud Platform project id where your logs are and where the Dataflow log forwarder will run
$PROJECT_ID=<your_project_id>
$# Temporary bucket that will store intermediary files as a result of compiling the Dataflow template. Its name must be unique.
$BUCKET_NAME=<a_nonexisting_gcs_bucket_name>
$# New Relic license key
$NR_LICENSE_KEY=<your_newrelic_license_key>
$# Input PubSub subscription to read logs from
$INPUT_SUBSCRIPTION_NAME=<your_pubsub_input_subscription_name>
$
$# Region where the created resources will be placed
$REGION=us-west1
$# Service account used to execute the Dataflow template
$SERVICE_ACCOUNT_NAME=nr-dataflow-forwarder-sa
$# File name where the service account credentials will be stored
$SERVICE_ACCOUNT_KEY_FILENAME=service-account-key.json
$# The name your Dataflow log forwarder job will have
$JOB_NAME=nr-log-forwarder
```

Execute the following commands:

Create the service account:

bash

$gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME}

Grant permissions to the service account:

bash

$gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" --role="roles/owner"

Generate the service account key file:

bash

$gcloud iam service-accounts keys create ${SERVICE_ACCOUNT_KEY_FILENAME} --iam-account=${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

Reference your service account key file using the GOOGLE_APPLICATION_CREDENTIALS environment variable, as it will be used by subsequent commands:
bash
```
$export GOOGLE_APPLICATION_CREDENTIALS=${SERVICE_ACCOUNT_KEY_FILENAME}
```

Execute the following command:

bash

$mvn compile exec:java \
>    -Dexec.mainClass=com.google.cloud.teleport.templates.PubsubToNewRelic \
>    -Dexec.cleanupDaemonThreads=false \
>    -Dexec.args=" \
>        --project=${PROJECT_ID} \
>        --region=${REGION} \
>        --enableStreamingEngine \
>        --stagingLocation=gs://${BUCKET_NAME}/staging/ \
>        --gcpTempLocation=gs://${BUCKET_NAME}/temp/ \
>        --templateLocation=gs://${BUCKET_NAME}/template/PubsubToNewRelic \
>        --runner=DataflowRunner \
>    "

Execute the following command to start shipping logs using a Dataflow job that reads from your Pub/Sub topic:

bash

$gcloud dataflow jobs run ${JOB_NAME} \
>    --gcs-location=gs://${BUCKET_NAME}/template/PubsubToNewRelic \
>    --region=${REGION} \
>    --parameters "inputSubscription=projects/${PROJECT_ID}/subscriptions/${INPUT_SUBSCRIPTION_NAME},licenseKey=${NR_LICENSE_KEY}"

This command only requires these two values:

The input PubSub subscription used to read log messages
The New Relic license key used to send your logs

European accounts should also add the logsApiUrl parameter with https://log-api.eu.newrelic.com/log/v1 as value.

bash

$gcloud dataflow jobs run ${JOB_NAME} \
>    --gcs-location=gs://${BUCKET_NAME}/template/PubsubToNewRelic \
>    --region=${REGION} \
>    --parameters "inputSubscription=projects/${PROJECT_ID}/subscriptions/${INPUT_SUBSCRIPTION_NAME},licenseKey=${NR_LICENSE_KEY},logsApiUrl=https://log-api.eu.newrelic.com/log/v1"

Tip

For other values, the command uses default configuration settings that you can further customize as needed.

Here is a reference of available options you can use to further tune up the execution of your Dataflow log forwarder job.

Configuration parameter	Description
`licenseKey` Required.	New Relic license key.
`inputSubscription` Required.	The Cloud Pub/Sub subscription used to consume logs. Use this format: `projects/<project-id>/subscriptions/<subscription-name>`
`logsApiUrl`	New Relic's URL for the Log API. This routes from the VPC where the Dataflow pipeline runs. Default: `https://log-api.newrelic.com/log/v1` Europe region: `https://log-api.eu.newrelic.com/log/v1`
`batchCount`	Maximum number of log records to aggregate into a batch before sending them to New Relic in a single HTTP POST request. Default: `100`
`flushDelay`	Number of seconds to wait for additional logs (up to `batchCount`) since the reception of the last log record in non-full batch, before flushing them to New Relic. Default: `2`
`parallelism`	Maximum number of parallel requests. Default: `1`
`disableCertificateValidation`	Disable SSL certificate validation. Default: `false`
`useCompression`	Compress (in GZIP) the payloads sent to the New Relic Logs API. Default: `true`
`tokenKMSEncryptionKey`	KMS Encryption Key for the token. Use this format: `projects/{gcp_project}/locations/{key_region}/keyRings/{key_ring}/cryptoKeys/{kms_key_name}` Default: `null`

What's next?

Explore logging data across your platform with the New Relic One UI.

Get deeper visibility into both your application and your platform performance data by forwarding your logs with our logs in context capabilities.
Set up alerts.
Query your data and create dashboards.

Disable log forwarding

To disable log forwarding capabilities, follow standard procedures in Google Cloud Platform documentation. You do not need to do anything else in New Relic.

Forward logs from Google Cloud Platform

Select an option

Use headerless API

1. Generate a GCP Pub/Sub ingest URL.

2. Create a GCP Pub/Sub Topic.

3. Prepare GCP Pub/Sub Topic to forward logs to New Relic.

4. Create routing sink to forward logs to New Relic.

Use Dataflow job

2. Clone the DataflowTemplates GitHub repository.

3. Compile and run the Dataflow forwarder.

4. Create a GCP bucket for the Dataflow template.

5. Create a service account.

6. Compile and publish the PubsubToNewRelic template.

7. Run the template as a Dataflow job.

European Region Accounts

Tip

8. Optional: Tune up your Dataflow log forwarder job.

What's next?

Disable log forwarding

Forward logs from Google Cloud Platform

Select an option

Use headerless API

2. Create a GCP Pub/Sub Topic.

3. Prepare GCP Pub/Sub Topic to forward logs to New Relic.

4. Create routing sink to forward logs to New Relic.

Use Dataflow job

1. Log into your Google Cloud Platform account.

2. Clone the DataflowTemplates GitHub repository.

3. Compile and run the Dataflow forwarder.

4. Create a GCP bucket for the Dataflow template.

5. Create a service account.

6. Compile and publish the PubsubToNewRelic template.

7. Run the template as a Dataflow job.

8. Optional: Tune up your Dataflow log forwarder job.

What's next?

Disable log forwarding