You can build your own charts and query all your Kubernetes integration data using the query builder and the NerdGraph API. Our integration collects Kubernetes data by instrumenting the container orchestration layer. For a simpler and more visual experience, use the cluster explorer.
one.newrelic.com > Dashboards: Using the query builder you can query your Kubernetes data and create clear visualizations.
Query Kubernetes data
The simplest way to query your Kubernetes data is using the query builder, which accepts NRQL queries. Alternatively, you can use the NerdGraph API to retrieve Kubernetes data.
Event types
Kubernetes data is attached to the following event types:
Event name | Type of Kubernetes data | Available since |
---|---|---|
| v1.0.0 | |
| v1.0.0 | |
| v1.0.0 | |
| v1.0.0 | |
| v1.13.0 | |
| v1.13.0 | |
| v1.0.0 | |
| v1.0.0 | |
| v1.0.0 | |
| v1.0.0 | |
| v1.11.0 | |
| v1.11.0 | |
| v1.11.0 | |
| v1.11.0 | |
| v1.13.0 | |
| v1.13.0 | |
| v2.3.0 |
Manage alerts
You can be notified about alert violations for your Kubernetes data:
Kubernetes attributes and metrics
The Kubernetes integration collects the following metrics and other attributes.
Node data
Query the K8sNodeSample
event for node data:
Node attribute | Description |
---|---|
| Node allocatable CPU cores |
| Node allocatable memory bytes |
| Node allocatable pods |
| Node allocatable ephemeral-storage bytes |
| Node CPU capacity |
| Node memory capacity (in bytes) |
| Pod capacity of the node |
| Node ephemeral-storage capacity |
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Status of the current observed node condition. The reported conditions can vary depending on your Kubernetes flavor and installed operators. |
| Node CPU usage measured in core milliseconds |
| Node CPU usage measured in cores |
| Total amount of CPU cores requested |
| Percentage of CPU cores actually used with respect to the CPU cores allocatable |
| Bytes available in the node filesystem |
| Total capacity of the node filesystem in bytes |
| Total number of inodes in the node filesystem |
| Free inodes in the node filesystem |
| Used inodes in the node filesystem |
| Used bytes in the node filesystem |
| Percentage of used bytes in the node filesystem with respect to the capacity |
| Bytes of memory available in the node |
| Number of major page faults per second in the node |
| Number of page faults in the node |
| Bytes of rss memory |
| Bytes of memory used |
| Bytes of memory in the working set |
| Total amount of requested memory |
| Percentage of bytes of memory in the working set with respect to the node allocatable memory |
| Number of errors per second while receiving/transmitting over the network |
| Host name that the pod is running on |
| Bytes available to the container runtime filesystem |
| Total capacity assigned to the container runtime filesystem in bytes |
| Total number of inodes in the container runtime filesystem |
| Free inodes in the container runtime filesystem |
| Used inodes in the container runtime filesystem |
| Used bytes in the container runtime filesystem |
| Status of node schedulability of new pods. Its value can be 0 (false) or 1 (true) |
| Labels associated with your node, so you can filter and query for specific nodes |
Namespace data
Query the K8sNamespaceSample
event for namespace data:
Namespace attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of the namespace when it was created |
| Name of the namespace to be used as an identifier |
| Labels associated with your namespace, so you can filter and query for specific namespaces |
| Current status of the namespace. The value can be |
Deployment data
Query the K8sDeploymentSample
event for deployment data:
Deployment attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of when the deployment was created |
| Name of the deployment to be used as an identifier |
| Name of the namespace that the deployment belongs to |
| Labels associated with your deployment, so you can filter and query for specific deployments |
| Number of replicas that are currently available |
| Number of replicas that you defined in the deployment |
| Total number of replicas that are currently running |
| Number of replicas that are currently unavailable |
| Number of replicas that have been updated to achieve the desired state of the deployment |
| Total number of replicas that are missing (number of desired replicas, podsDesired, minus the total number of replicas, podsTotal) |
ReplicaSet data
Query the K8sReplicasetSample
event for ReplicaSet
data:
Replica attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of when the |
| Name of the deployment to be used as an identifier |
| Name of the namespace that the |
| Integer representing generation observed by the |
| Number of replicas that you defined in the deployment |
| Number of pods that have labels that match the |
| Number of replicas that are ready for this |
| Total number of replicas that are currently running |
| Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady) |
| Name of the |
DaemonSet data
Query the K8sDaemonsetSample
event for DaemonSet
data:
DaemonSet attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of when the |
| Name of the namespace that the |
| Labels associated with your |
daemonsetName | Name associated with the |
| The number of nodes that should be running the daemon pod |
| The number of nodes running at least one daemon pod and are supposed to |
| The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and available |
| The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready |
| The number of nodes that should be running the daemon pod and have none of the daemon pod running and available |
| The number of nodes running a daemon pod but are not supposed to |
| The total number of nodes that are running updated daemon pod |
| Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady) |
| Sequence number representing a specific generation of the desired state |
StatefulSet data
Query the K8sStatefulsetSample
event for StatefulSet
data:
StatefulSet attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of when the |
| Name of the namespace that the |
| Labels associated with your |
| Name associated with the |
| Number of desired pods for a |
| The number of ready replicas per |
| The number of current replicas per |
| The number of replicas per |
| The number of updated replicas per |
| Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady) |
| The generation observed by the |
| Sequence number representing a specific generation of the desired state for the |
| Indicates the version of the |
| Indicates the version of the |
Pod data
Query the K8sPodSample
event for pod data:
Pod attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of when the pod was created in epoch seconds |
| Name of the Kubernetes object that created the pod. For example, |
| Kind of Kubernetes object that created the pod. For example, |
| Name of the deployment to be used as an identifier |
| Boolean representing whether or not the pod is ready to serve requests |
| Boolean representing whether or not the pod has been scheduled to run on a node |
| Labels associated with your pod, so you can filter and query for specific pods |
| Details related to the last pod status change |
| Name of the namespace that the pod belongs to |
| Number of errors per second while receiving/transmitting over the network |
| Number of errors per second |
| Number of bytes per second received over the network |
| Number of bytes per second transmitted over the network |
| Host IP address that the pod is running on |
| Host name that the pod is running on |
| IP address of the pod. If it doesn't have an IP, it'll be empty |
| Name of the pod to be used as an identifier |
| Reason why the pod is in the current status |
| Timestamp of when the pod started running in epoch seconds |
| Current status of the pod. Value can be |
Cluster data
Query the K8sClusterSample
event to see cluster data:
Cluster attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Kubernetes version that the cluster is running |
Container data
Query the K8sContainerSample
event for container data:
Container attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Unique ID associated with the container. If you are running Docker, this is the Docker container id |
| Name of the image that the container is running |
| Unique ID associated with the image that the container is running |
| Name associated with the container |
| Integer representing limit CPU cores defined for the container in the pod specification |
| Requested CPU cores defined for the container in the pod specification |
| CPU cores actually used by the container |
| Percentage of CPU cores actually used by the container with respect to the CPU limit specified. This percentage is based on this calculation: ( |
| Percentage of CPU cores actually used by the container with respect to the CPU request specified |
| Name of the deployment to be used as an identifier |
| Boolean. Whether or not the container's readiness check succeeded |
| Labels associated with your container, so you can filter and query for specific containers |
| Integer representing limit bytes of memory defined for the container in the pod specification |
| Integer. Requested bytes of memory defined for the container in the pod specification |
| Integer. Bytes of memory actually used by the container |
| Percentage of memory actually used by the container with respect to the memory limit specified |
| Percentage of memory actually used by the container with respect to the memory request specified |
| Integer. Bytes of memory in the working set |
| Percentage of working set memory actually used by the container with respect to the memory limit specified |
| Percentage of working set memory actually used by the container with respect to the memory request specified |
| Name of the namespace that the container belongs to |
| Host IP address the container is running on |
| Host name that the container is running on |
| Name of the pod that the container is in, to be used as an identifier |
| Provides a reason why the container is in the current status |
| Number of times the container has been restarted |
| Current status of the container. Value can be |
| Delta change of elapsed enforcement period intervals |
| Delta change of throttled period intervals |
| Delta change of duration the container has been throttled, in seconds |
| Total number of elapsed enforcement period intervals |
| Total number of throttled period intervals |
| Total time duration the container has been throttled, in seconds |
| Total size of memory mapped files used by this container, in bytes |
Volume data
Query the K8sVolumeSample
event for volume data:
Volume attribute | Description |
---|---|
| Name that you assigned to the volume at creation |
| Cluster where the volume is configured |
| Namespace where the volume is configured |
| The pod that the volume is attached to. The Kubernetes monitoring integration lists Volumes that are attached to a pod |
| If this is a persistent volume, this value is set to |
| Namespace where the Persistent Volume Claim is configured |
| Name that you assigned to the Persistent Volume Claim at creation |
| Capacity of the volume, in bytes |
| Usage of the volume, in bytes |
| Capacity available of the volume, in bytes |
| Usage of the volume in percentage |
| Total |
|
|
|
|
Volume data is available for volume plugins that implement the MetricsProvider interface:
- AWSElasticBlockStore
- AzureDisk
- AzureFile
- Cinder
- Flexvolume
- Flocker
- GCEPersistentDisk
- GlusterFS
- iSCSI
- NFS
- StorageOS
- VsphereVolume
API server data
Query the K8sApiServerSample
event to see API Server data. For more information, see Configure control plane monitoring:
API server attribute | Description |
---|---|
| Resident memory size, in bytes |
| Difference of the user and system CPU time spent, in seconds |
| Number of OS threads created |
| Number of goroutines that currently exist |
| Difference of the number of apiserver requests, broken out for each verb and HTTP response code |
| Rate of apiserver requests, broken out for each verb and HTTP response code |
| Difference of the number of HTTP requests, partitioned by method and code |
| Rate of the number of HTTP requests, partitioned by method and code |
| Number of stored objects at the time of last check, split by kind |
Controller manager data
Query the K8sControllerManagerSample
event to see Controller manager data. For more information, see Configure control plane monitoring:
Controller manager attribute | Description |
---|---|
| Resident memory size, in bytes |
| Difference of the user and system CPU time spent in seconds |
| Number of OS threads created |
| Number of goroutines that currently exist |
| Difference of the total number of adds handled by workqueue |
| Current depth of workqueue |
| Difference of the total number of retries handled by workqueue |
| Gauge of if the reporting system is master of the relevant lease, |
Scheduler data
Query the K8sSchedulerSample
event to see Scheduler data. For more information, see Configure control plane monitoring:
Scheduler attribute | Description |
---|---|
| Resident memory size, in bytes |
| Difference of the user and system CPU time spent in seconds |
| Number of OS threads created |
| Number of goroutines that currently exist |
| Gauge of if the reporting system is master of the relevant lease, |
| The HTTP request latencies in microseconds, per quantile |
| The sum of the HTTP request latencies, in microseconds |
| The number of observed HTTP requests events |
| Difference of the number of HTTP requests, partitioned by status code, method, and host |
| Rate of the number of HTTP requests, partitioned by status code, method, and host |
| Difference of the number of attempts to schedule pods, by the result. |
| Rate of the number of attempts to schedule pods, by the result. |
| Scheduling latency in seconds split by sub-parts of the scheduling operation |
| The sum of scheduling latency in seconds split by sub-parts of the scheduling operation |
| The number of observed events of schedulings split by sub-parts of the scheduling operation. |
| Difference of the total preemption attempts in the cluster till now |
| Number of selected preemption victims |
etcd data
Query the K8sEtcdSample
event to see etcd data. For more information, see Configure control plane monitoring:
etcd attribute | Description |
---|---|
| Resident memory size, in bytes |
| Difference of the user and system CPU time spent in seconds |
| Number of OS threads created |
| Number of goroutines that currently exist |
| Whether or not a leader exists. |
| Difference of the number of leader changes seen |
| Total size of the underlying database physically allocated, in bytes |
| Difference of the total number of consensus proposals committed |
| Rate of the total number of consensus proposals committed |
| Difference of the total number of consensus proposals applied |
| Rate of the total number of consensus proposals applied |
| The current number of pending proposals to commit |
| Difference of the total number of failed proposals seen |
| Rate of the total number of failed proposals seen |
| Number of open file descriptors |
| Maximum number of open file descriptors |
| Percentage open file descriptors with respect to the maximum number that can be opened |
| Rate of the total number of bytes received from gRPC clients |
| Rate of the total number of bytes sent to gRPC clients |
Endpoint data
Query the K8sEndpointSample
event for endpoint data:
Endpoint attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of when the endpoint was created |
| Name of the namespace that the endpoint belongs to |
| Name associated with the endpoint |
| Labels associated with your endpoint, so you can filter and query for specific endpoints |
| Number of addresses available in endpoint |
| Number of addresses not ready in endpoint |
Service data
Query the K8sServiceSample
event for service data:
Service attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Timestamp of when the service was created |
| Name of the namespace that the service belongs to |
| Labels associated with your service, so you can filter and query for specific service |
| Name associated with the service |
| The IP of the external load balancer, if |
| The external name value, if |
| The internal cluster IP, if |
| Type of the service |
| The label selector that this service targets |
Horizontal Pod Autoscaler data
Query the K8sHpaSample
event for Horizontal Pod Autoscaler data:
HPA attribute | Description |
---|---|
| Name that you assigned to the cluster when you installed the Kubernetes integration |
| Labels associated with your HPA, so you can filter and query for specific autoscaler |
| Current number of replicas of pods managed by this autoscaler |
| Desired number of replicas of pods managed by this autoscaler |
| Lower limit for the number of pods that can be set by the autoscaler, 1 by default |
| Upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than |
| The metric specifications used by this autoscaler when calculating the desired replica count |
| Boolean representing whether or not the autoscaler is able to fetch and update scales, as well as whether or not any backoff-related conditions would prevent scaling |
| Boolean representing whether or not the autoscaler is enabled (if it's able to calculate the desired scales) |
| Boolean representing whether or not the autoscaler is capped, either up or down, by the maximum or minimum replicas configured |
| Number of Kubernetes labels converted to Prometheus labels |
| The generation observed by the HorizontalPodAutoscaler controller |
Kubernetes metadata in APM-monitored applications
By linking your applications with Kubernetes, the following attributes are added to application trace and distributed trace:
nodeName
containerName
podName
clusterName
deploymentName
namespaceName