When using synthetic monitoring's private locations with New Relic's alerts, you can be notified if a location is under-provisioned, mis-configured, or generally misbehaving.
This guide will help you answer the following basic questions regarding private location health by using New Relic dashboards and NRQL alerts:
- Are my private minions online?
- Does my private location need more minions?
- Can I check the status of a specific minion directly?
Prerequisites
Before following the instructions in this guide, ensure you have:
- A synthetic private location
- At least one private minion installed at that location
- Checks scheduled to run at that location
- An alert policy for the private location, with a configured notification channel to notify your team when a violation occurs
The following Private Minion dashboard example JSON can be imported to your account using:
{ "name": "Synthetics Private Minions", "description": "Details on events from SyntheticsPrivateLocationStatus, SyntheticsPrivateMinion, SyntheticCheck, SyntheticRequest, NrAuditEvent, plus a page for Performance Analysis.", "permissions": "PUBLIC_READ_WRITE", "pages": [ { "name": "SyntheticsPrivateLocationStatus", "description": "Details on the private location queue.", "widgets": [ { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 1, "height": 5, "width": 3 }, "title": "pending checks by location", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateLocationStatus SELECT latest(checksPending) FACET name" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 4, "row": 1, "height": 4, "width": 9 }, "title": "private location queue", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateLocationStatus SELECT max(checksPending) FACET name TIMESERIES MAX SINCE 2 weeks ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 4, "row": 5, "height": 4, "width": 9 }, "title": "rate of queue growth", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateLocationStatus SELECT clamp_min(derivative(checksPending,1 minute),0) AS 'queue growth rate' FACET name TIMESERIES MAX SINCE 2 weeks ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.billboard" }, "layout": { "column": 1, "row": 6, "height": 3, "width": 3 }, "title": "Checks Pending", "rawConfiguration": { "dataFormatters": [], "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateLocationStatus SELECT average(checksPending) SINCE 3 minutes ago UNTIL 2 minutes ago" } ], "thresholds": [ { "alertSeverity": "WARNING", "value": 1 }, { "alertSeverity": "CRITICAL", "value": 5 } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 1, "row": 9, "height": 4, "width": 12 }, "title": "audit events for private locations", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM NrAuditEvent SELECT * WHERE targetType = 'PRIVATE_LOCATION' SINCE 2 days ago LIMIT MAX" } ] }, "linkedEntityGuids": null } ] }, { "name": "SyntheticsPrivateMinion", "description": null, "widgets": [ { "visualization": { "id": "viz.table" }, "layout": { "column": 1, "row": 1, "height": 3, "width": 3 }, "title": "private locations", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) - 1 AS 'restarts',latest(timestamp) WHERE minionLocation NOT LIKE 'AWS_%' FACET minionLocation" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 4, "row": 1, "height": 3, "width": 9 }, "title": "minion details", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT latest(minionId),latest(timestamp) AS 'last seen',latest(minionStartTimestamp) AS 'started',uniqueCount(minionId) - 1 AS 'restarts',latest(minionIpv4),latest(minionJobsQueued),latest(minionJobsFinished),100*latest(minionJobsFailed)/latest(minionJobsQueued) AS 'job failure rate',latest(minionJobsRunning),latest(minionJobsTimedout),latest(minionJobsSkipped),latest(minionJobsInternalEngineError),latest(minionWorkers),latest(minionProcessors),latest(minionPhysicalMemoryUsedBytes/(1024*1024*1024)) AS 'used memory (GiB)',latest(minionPhysicalMemoryTotalBytes/(1024*1024*1024)) AS 'total memory (GiB)',latest(minionPhysicalMemoryTotalBytes/(1024*1024*1024))/latest(minionWorkers) AS 'memory (GiB) per heavy worker' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionBuildNumber,minionLocation,minionOsVersion,minionContainerSystemVersion,minionHostname LIMIT 100 SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.area" }, "layout": { "column": 1, "row": 4, "height": 3, "width": 2 }, "title": "minion instances over time", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'minions' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 3, "row": 4, "height": 3, "width": 4 }, "title": "CPU utilization (%)", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT average(minionProcessorsUsagePercentage) AS 'avg',max(minionProcessorsUsagePercentage) AS 'max' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 4, "height": 3, "width": 6 }, "title": "memory utilization (%)", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT average(minionPhysicalMemoryUsedPercentage) AS 'average',max(minionPhysicalMemoryUsedPercentage) AS 'max' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 1, "row": 7, "height": 3, "width": 4 }, "title": "swap (GiB)", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT average(minionSwapMemoryUsedBytes) AS 'avg',max(minionSwapMemoryUsedBytes) AS 'max' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.area" }, "layout": { "column": 5, "row": 7, "height": 3, "width": 4 }, "title": "used vs free memory (GiB)", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT max(minionPhysicalMemoryUsedBytes) / (1024*1024*1024) AS 'used',(max(minionPhysicalMemoryTotalBytes)-max(minionPhysicalMemoryUsedBytes)) / (1024*1024*1024) AS 'free' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 9, "row": 7, "height": 3, "width": 4 }, "title": "count of Internal Engine Errors (IEE) vs failed jobs", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsInternalEngineError) AS 'IEE',max(minionJobsFailed) AS 'failed jobs' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": false } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 1, "row": 10, "height": 6, "width": 4 }, "title": "jobs running", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsRunning) AS 'concurrent',clamp_min(derivative(minionJobsFinished,1 minute),0) AS 'finished per minute' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 10, "height": 3, "width": 4 }, "title": "jobs finished", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsFinished) - min(minionJobsFinished) AS 'total jobs' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.stacked-bar" }, "layout": { "column": 9, "row": 10, "height": 3, "width": 4 }, "title": "job failure rates", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT (max(minionJobsFailed/minionJobsReceived))*100 AS 'failed rate',(max(minionJobsInternalEngineError/minionJobsReceived))*100 AS 'IEE rate' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 13, "height": 3, "width": 4 }, "title": "skipped jobs", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsSkipped) - min(minionJobsSkipped) AS 'skipped jobs' WHERE minionLocation NOT LIKE 'AWS_%' FACET minionHostname TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": false } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.stacked-bar" }, "layout": { "column": 9, "row": 13, "height": 3, "width": 4 }, "title": "IEE as a percentage of failed jobs", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT average(minionJobsInternalEngineError/minionJobsFailed)*100 AS 'IEE',(average((minionJobsFailed-minionJobsInternalEngineError)/minionJobsFailed))*100 AS 'failed jobs' WHERE minionLocation NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ] }, "linkedEntityGuids": null } ] }, { "name": "SyntheticCheck", "description": null, "widgets": [ { "visualization": { "id": "viz.table" }, "layout": { "column": 1, "row": 1, "height": 3, "width": 4 }, "title": "minion IDs by location", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT latest(minion) WHERE location NOT LIKE 'AWS_%' FACET location,minionId LIMIT MAX" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 1, "height": 3, "width": 4 }, "title": "minion count", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(minionId) WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 9, "row": 1, "height": 3, "width": 4 }, "title": "IEE by minion", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT count(error) AS 'count',latest(timestamp) WHERE location NOT LIKE 'AWS_%' AND error LIKE '%code:%' FACET error,minion LIMIT 100 SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 4, "height": 3, "width": 2 }, "title": "monitors by account", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 3, "row": 4, "height": 3, "width": 2 }, "title": "monitors by account", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 5, "row": 4, "height": 3, "width": 2 }, "title": "avg monitor frequency by type (m)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET type SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 4, "height": 3, "width": 2 }, "title": "avg monitor frequency by type (m)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 9, "row": 4, "height": 3, "width": 2 }, "title": "Internal Engine Errors", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT count(error) AS 'IEE count' WHERE error LIKE '%code:%' AND location NOT LIKE 'AWS_%' FACET error SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 11, "row": 4, "height": 3, "width": 2 }, "title": "Internal Engine Errors", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT count(error) AS 'IEE count' WHERE error LIKE '%code:%' AND location NOT LIKE 'AWS_%' FACET error TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 7, "height": 3, "width": 2 }, "title": "monitors by status", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET result SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 3, "row": 7, "height": 3, "width": 2 }, "title": "monitors by status", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 5, "row": 7, "height": 3, "width": 2 }, "title": "job duration by status (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET result SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 7, "height": 3, "width": 2 }, "title": "job duration by status (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 9, "row": 7, "height": 12, "width": 4 }, "title": "monitor details", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(id) AS 'results',uniqueCount(location)/rate(uniqueCount(id), 1 minute) AS 'avg job frequency (m)',average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg duration (s)' WHERE location NOT LIKE 'AWS_%' FACET monitorName,monitorId,type LIMIT 20" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 10, "height": 3, "width": 2 }, "title": "monitors by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET type SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 3, "row": 10, "height": 3, "width": 2 }, "title": "monitors by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 5, "row": 10, "height": 3, "width": 2 }, "title": "job duration by type (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET type SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 10, "height": 3, "width": 2 }, "title": "job duration by type (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 13, "height": 3, "width": 2 }, "title": "successful monitors by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 3, "row": 13, "height": 3, "width": 2 }, "title": "successful monitors by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 5, "row": 13, "height": 3, "width": 2 }, "title": "successful job duration by type (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 13, "height": 3, "width": 2 }, "title": "successful job duration by type (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 16, "height": 3, "width": 2 }, "title": "failed monitors by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 3, "row": 16, "height": 3, "width": 2 }, "title": "failed monitors by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 5, "row": 16, "height": 3, "width": 2 }, "title": "failed job duration by type (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 16, "height": 3, "width": 2 }, "title": "failed job duration by type (s)", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 1, "row": 19, "height": 3, "width": 4 }, "title": "ping monitors by location", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'monitors with results',uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg job frequency (m)',average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg duration (s)' WHERE location NOT LIKE 'AWS_%' AND type = 'SIMPLE' FACET location SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 19, "height": 3, "width": 2 }, "title": "ping duration with monitor count", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)',uniqueCount(monitorId) AS 'monitor count' WHERE type = 'SIMPLE' AND location NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 7, "row": 19, "height": 3, "width": 6 }, "title": "top 5 ping errors", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT latest(monitorId) AS 'latest monitor id',latest(type),average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)',count(*),max(timestamp) AS 'last occurred' WHERE type = 'SIMPLE' AND result = 'FAILED' AND location NOT LIKE 'AWS_%' FACET error LIMIT 5 SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 1, "row": 22, "height": 3, "width": 4 }, "title": "non-ping monitors by location", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'monitors with results',uniqueCount(monitorId)/rate(uniqueCount(id), 1 minute) AS 'avg job frequency (m)',average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg duration (s)' WHERE location NOT LIKE 'AWS_%' AND type != 'SIMPLE' FACET location SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 22, "height": 3, "width": 2 }, "title": "non-ping duration with monitor count", "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)',uniqueCount(monitorId) AS 'monitor count' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 7, "row": 22, "height": 3, "width": 6 }, "title": "top 5 non-ping errors", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT latest(monitorId) AS 'latest monitor id',latest(type),average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)',count(*),max(timestamp) AS 'last occurred' WHERE type != 'SIMPLE' AND result = 'FAILED' AND location NOT LIKE 'AWS_%' FACET error LIMIT 5 SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 1, "row": 25, "height": 4, "width": 6 }, "title": "jobs per minute by minion", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' FACET minion TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 25, "height": 4, "width": 6 }, "title": "errors per minute by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' AND error != '' AND error IS NOT NULL FACET type,result TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 1, "row": 29, "height": 4, "width": 6 }, "title": "jobs per minute by type", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 7, "row": 29, "height": 4, "width": 6 }, "title": "error counts by class", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT count(*) WHERE location NOT LIKE 'AWS_%' AND result != 'SUCCESS' AND error NOT LIKE '%case:%' FACET capture(error, r'(?P<firstWord>\\S+)\\s.+') AS 'error class' TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null } ] }, { "name": "SyntheticRequest", "description": "Details on responses.", "widgets": [ { "visualization": { "id": "viz.table" }, "layout": { "column": 1, "row": 1, "height": 6, "width": 5 }, "title": "jobs per minion", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticRequest SELECT uniqueCount(id) AS 'jobs' WHERE location NOT LIKE 'AWS%' FACET location,minion,minionId" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.area" }, "layout": { "column": 6, "row": 1, "height": 6, "width": 4 }, "title": "top 5 error response codes", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticRequest SELECT count(responseCode) AS 'responses' WHERE responseCode NOT IN (200,201,202,203,204,300,301,302,303,304) AND responseCode IS NOT NULL AND location NOT LIKE 'AWS%' FACET responseCode,responseStatus TIMESERIES AUTO LIMIT 5" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 10, "row": 1, "height": 6, "width": 3 }, "title": "job rate", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "legend": { "enabled": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticRequest SELECT rate(uniqueCount(id), 1 minute) AS 'job rate' WHERE location NOT LIKE 'AWS%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 1, "row": 7, "height": 4, "width": 12 }, "title": "audit events for monitors", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM NrAuditEvent SELECT * WHERE targetType = 'MONITOR' SINCE 2 days ago LIMIT MAX" } ] }, "linkedEntityGuids": null } ] }, { "name": "Performance Analysis", "description": "Determine if there are enough heavy worker threads.", "widgets": [ { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 1, "height": 3, "width": 4 }, "title": "minions (hosts) by location", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'minions' FACET minionLocation SINCE 5 minutes ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.table" }, "layout": { "column": 5, "row": 1, "height": 3, "width": 6 }, "title": "count of heavy worker threads and cpu cores", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT latest(minionWorkers),latest(minionProcessors) FACET minionHostname,minionLocation SINCE 5 minutes ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.markdown" }, "layout": { "column": 11, "row": 1, "height": 3, "width": 2 }, "title": "", "rawConfiguration": { "text": "See [this Google sheet](https://docs.google.com/spreadsheets/d/1k2Aw11r6-S8pIpXQUINQZ0T5qlxS8iSwSo9t9GKDvzc/edit?usp=sharing) to assist in calculating how many workers will be needed based on the values on this page. This will help you to assess the required size of each host (scaling up) and how many hosts you need (scaling out).\n\nFor more info: \nhttps://discuss.newrelic.com/t/relic-solution-scaling-and-rightsizing-for-the-cpm/123999" }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 4, "height": 3, "width": 4 }, "title": "non-ping monitors", "rawConfiguration": { "dataFormatters": [], "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'non-ping monitors' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS%' FACET location SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 4, "height": 3, "width": 6 }, "title": "non-ping avg job duration and timeout", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS%' AND result = 'FAILED' FACET location,cases(WHERE error NOT LIKE '%timeout%' AS 'job duration (s)', WHERE error LIKE '%timeout%' AS 'job timeout (s)') SINCE 2 days ago TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.bar" }, "layout": { "column": 1, "row": 7, "height": 3, "width": 4 }, "title": "non-ping jobs per minute", "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id), 1 minute) AS 'job rate' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS%' FACET location SINCE 2 days ago" } ] }, "linkedEntityGuids": null }, { "visualization": { "id": "viz.line" }, "layout": { "column": 5, "row": 7, "height": 3, "width": 6 }, "title": "job timeout rate by location", "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 1, "query": "FROM SyntheticsPrivateMinion SELECT 100*latest(minionJobsTimedout)/latest(minionJobsReceived) AS 'job timeout rate' FACET minionLocation TIMESERIES AUTO SINCE 2 days ago" } ], "yAxisLeft": { "zero": true } }, "linkedEntityGuids": null } ] } ]}
Steps to import:
- Copy the Dashboard JSON and paste into a text editor.
- Replace
"accountId": 1,
with your New Relic account ID for each occurrence in the JSON code. - Copy the Dashboard JSON from your text editor and import using one of the methods described above.
- Edit any charts that you'd like to use facet filtering with.
Tip
If your private locations exist in a parent account and Synthetics monitors in a sub account, insert the parent account ID for NRQL queries that use SyntheticPrivateLocationStatus
and SyntheticsPrivateMinion
, and the sub account ID for queries that use SyntheticCheck
and SyntheticRequest
.
Are my private minions online?
To answer this question, you can rely on attributes from the SyntheticsPrivateMinion
event. Private minions send this event to New Relic every 30 seconds. A simple way to check if your minions are online is to compare the unique count of minion IDs with the number of minions you expect to be online.
To understand how many minions are reporting, run this example NRQL query:
SELECT uniqueCount(minionId)
FROM SyntheticsPrivateMinion
WHERE minionLocation = '1-acme_okc_dc-309'
Using this query, you can create an alert condition to notify your team when fewer minions are reporting than expected. This condition is configured with a static threshold of 2 units
, which means you will receive an alert if any of your minions are offline.
You can verify that the alert policy works as expected by manually stopping one of your minions. Then, when the alert violation occurs, you will be notified by any notification channels that have been set up. Once the minion is restarted and it comes back online, the alert will recover.
There are more robust ways to check whether minions are functioning correctly, but this query and condition simply and successfully handle the case where a machine fails, is accidentally decommissioned, or the minion process crashes. It also ensures that the minion can communicate with New Relic.
Does my private location need more minions?
To answer this question, you can use the checksPending
attribute of the SyntheticsPrivateLocationStatus
event. The checksPending
attribute reflects the number of monitor checks that are scheduled (or "queued") but have yet to be accepted by a minion in the designated location. For a location with scheduled checks and no minions, this graph would grow linearly up and to the right.
This metric is more complicated to monitor than uniqueCount(minionId)
because a high value does not necessarily mean the location is in a bad state. As long as the metric is not growing linearly up and to the right (and checks are being run on schedule), the location is in a good state.
This use case is perfect for baseline NRQL alert conditions, which allow you to monitor the deviation of a metric rather than its static value. For example:
SELECT average(checksPending)
FROM SyntheticsPrivateLocationStatus
WHERE name = '1-acme_tokyo_dc-512'
To test this alert condition, schedule one-minute, browser-based monitors to run from your location. Browser-based jobs consume more resources than ping jobs, which is why they are a better fit for load simulation. New Relic will quickly notify you of a growing number of pending checks.
After doubling the number of minions to handle the load, the alert recovers. For example, using the Synthetics private location dashboard example, notice the growth and decline of pending checks over the course of the incident and recovery. By using the NRQL condition, New Relic will notify you if and when the location needs more minion capacity.
Can I check the status of a specific minion directly?
You can also check how a minion is operating by contacting it directly. You can use a set of HTTP endpoints exposed by the minion to determine what the application is doing. In order to access these endpoints, bind ports 8080
and 8180
to ports on the host. For example, for Docker, use docker run -p 80:8080 -p 81:8180 ...
):
:8080/status/check
: Details about internal health-checks the minion performs; HTTP 200 means "healthy.":8080/status
: Details about a minion's status; the same data is then reported as aSyntheticsPrivateMinion
event.:8180/
: JVM application admin endpoints; an advanced view of a minion's internal state.
This approach is not as automated or flexible as the checksPending
example. However, if you have total network connectivity failure, this manual approach can help troubleshoot the situation.