Announcing our brand new Issue Details View! Classify severity quicker and solve issues faster with all the information you need on a single page. Quickly answer questions such as "what other resources are impacted?", "what else is going wrong with this resource?", and much more with the new Issue Details Page. |
Kubernetes is the container orchestration platform of choice for many teams. In our ongoing efforts to bring the magic experience of Lumigo’s serverless capabilities to the world of containerized applications, we are delighted to share with you the Lumigo Kubernetes operator, a best-in-class operator to automatically trace your applications running on Kubernetes using the Lumigo OpenTelemetry distributions for Java, Node.js and Python.
We designed the Lumigo Kubernetes operator with ease of use and automation in mind: to get your applications traced with Lumigo on Kubernetes, all you have to do is install the Lumigo Kubernetes operator with Helm, and then create in the namespaces where you want applications traced, a Lumigo resource:
apiVersion: v1 kind: Secret metadata: name: lumigo-credentials stringData: # Kubectl won't allow you to deploy this dangling anchor. # Get the actual value from Lumigo following this documentation: https://docs.lumigo.io/docs/lumigo-tokens token: *lumigo-token # Example: t_123456789012345678901 --- apiVersion: operator.lumigo.io/v1alpha1 kind: Lumigo metadata: labels: app.kubernetes.io/name: lumigo app.kubernetes.io/instance: lumigo app.kubernetes.io/part-of: lumigo-operator name: lumigo spec: lumigoToken: secretRef: name: lumigo-credentials # This must match the name of the secret; the secret must be in the same namespace as this Lumigo custom resource key: token # This must match the key in the Kubernetes secret
(You find your Lumigo token in your Lumigo project.)
The Lumigo Kubernetes operator will update your deployments, daemonsets, replicasets, statefulsets, cronjobs and jobs to be traced with the Lumigo OpenTelemetry Distro for Java, Lumigo OpenTelemetry Distro for Node.js and Lumigo OpenTelemetry Distro for Python.
Monitoring applications on Kubernetes has never been easier ;-)
For more details on how to use the Lumigo Kubernetes Operator, refer to the documentation on GitHub. And if you are curios about how we managed to make it so easy, we have an in-depth look at the magic behind the Lumigo Kubernetes operator waiting for you on the Lumigo blog.
Many serverless and microservice applications are polyglots, with components written in different programming languages based on the task to accomplish, desired performance characteristics, libraries to reuse, developers' preferences and many other reasons. Lumigo now shows you the Lambda runtime and OpenTelemetry SDK language in the invocation and transactions graph, making it easier for you to contextualize the transaction and its components, and react faster to issues. Also, it is a nice splash of color ;-)
P.S. To keep transactions easy to read, the runtime will be hidden when, due to zoom level and screen size, graph nodes are less than 40px wide.
The Node.js 18 runtime brings the latest and greatest LTS version of Node.js to Lambda, and as of today, it is officially supported in Lumigo!
Node.js 18 itself brings a built-in test runner, the support for the Fetch API, and a few other quality-of-life improvements. But what makes the Node.js 18 runtime very interesting, is the adoption of the AWS SDK 3 for Javascript: it is a rewrite of the AWS SDK that focuses on first-class Typescript support and modularity.
Extolling the virtues of Typescript's static type system (as opposing to effectively not having static type checks like Javascript), is something that can get me going for a while, and I am sparing everyone it.
But modularity has implications for cold starts. It is a known fact that the size of the Lambda bundle affects cold starts. It is pretty much as the intuition goes: the more code to load into the Node runtime, the longer the startup.
Another nice thing that can speed up your Lambda functions is with AWS SDK 3 is enablement by default of the TCP connection re-use, which can shave significant amounts of time across invocations for Lambda instances that are kept busy.
Just turn on Lumigo autotracing as always. If you went ahead and adopted Node.js 18 a few days ago, and Lumigo told you the runtime is not supported, by now you will have noticed that Lumigo by now has set up autotracing for your Node.js 18 functions that you selected for autotracing in the UI, or using the `lumigo:auto-trace=true` AWS Tag. Nice and easy :-)
System Map, one of Lumigo's most useful features that visualizes distributed application architectures to help understand service dependencies, just got much better.
Today, we are introducing System Map autocomplete filters. Similarly to Explore's autocomplete filters, System Map filters make it easier to discover and inspect the resources that comprise your application.
For more information on System-Map, see the System Map documentation.
The AWS Cloud Development Kit (CDK) enables you to define your cloud application resources using familiar programming languages, mixing an imperative programming style with the declarative nature of the underpinning AWS CloudFormation.
Today, we are launching @lumigo/cdk-constructs-v2: 1st-party, officially supported constructs to add Lumigo autotracing for your Lambda functions in CDK applications!
The Lumigo construct can add Lumigo autotracing to all your Node.js and Python Lambda functions in a CDK app:
import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';
const app = new App();
// Add here stacks and constructs
new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')}).traceEverything(app);
app.synth();Tracing can also be added to all Lambda functions in a stack:
import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';
export class NodejsStack extends Stack {
constructor(scope: Construct, id: string, props: StackProps = {}) {
super(scope, id, props);
new Function(this, 'MyLambda', {
code: new InlineCode('foo'),
handler: 'index.handler',
runtime: Runtime.NODEJS_14_X,
});
}
}
const app = new App();
const stack = new NodejsStack(app, 'NodejsTestStack', {
env: {
region: 'eu-central-1',
}
});
new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')}).traceEverything(stack);
app.synth();You can also add tracing to Lambda functions one by one:
import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';
interface NodejsStackProps extends StackProps {
readonly lumigo: Lumigo;
}
export class NodejsStack extends Stack {
constructor(scope: Construct, id: string, props: NodejsStackProps = {}) {
super(scope, id, props);
const handler = new Function(this, 'MyLambda', {
code: new InlineCode('foo'),
handler: 'index.handler',
runtime: Runtime.NODEJS_14_X,
});
props.lumigo.traceLambda(handler);
}
}
const app = new App();
const lumigo = new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')});
const stack = new NodejsStack(app, 'NodejsTestStack', {
env: {
region: 'eu-central-1',
},
lumigo,
});
app.synth();Autotracing can be added to all Lambda functions declared as a Function from the aws-cdk-lib/aws-lambda package with a Node.js or Python runtime, NodejsFunction from the aws-cdk-lib/aws-lambda-nodejs and PythonFunction from the @aws-cdk/aws-lambda-python-alpha package.
Each version of the @lumigo/cdk-constructs-v2 package comes with a built-in list of latest Lambda layers as of the time of publishing. We regularly (in a matter of hours) publish new versions of the @lumigo/cdk-constructs-v2 package when we release a new layer. Reproducibility is very important for AWS CDK apps, and that is why the same version of the @lumigo/cdk-constructs-v2 will always generate the same CloudFormation. (Moreover, it is bad practice to invoke APIs during the "synth" phase in CDK.) So, to get the newest tracers deployed, update the version of the @lumigo/cdk-constructs-v2 package and run "cdk deploy" anew.
It also possible for you to manually specify which version of the layer to use:
import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';
const app = new App();
// Add here stacks and constructs
new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')}).traceEverything(app, {
lambdaNodejsLayerVersion: 42, // All Lambda functions with a supported Node.js runtime will use the layer v42
lambdaPythonLayerVersion: 1, // All Lambda functions with a supported Python runtime will use the layer v1
});
app.synth();You can indeed ask Lumigo to autotrace functions you manage with any of the tools out there that rely on CloudFormation (CDK itself, the Serverless framework, the AWS Serverless Application Model), but some may find that having Lumigo change infrastructure after deployment is not ideal in Infrastructure as Code (IaC) approaches. So, we are making it straightforward to add Lumigo tracing as an integral part of your IaC specification. This is why we built the Lumigo Serverless plugin before, and we are releasing the Lumigo CDK constructs today!
Well, we are just getting started. There is some really interesting stuff happening with Lumigo in and beyond Serverless in the next few months ;-)
Special thanks to Thorsten Höger of Taimos, AWS DevTools Hero and author of "The CDK Book", for his invaluable feedback during the ideation and first implementation of the @lumigo/cdk-constructs-v2 package.
Thorsten offers various service packages to help you level up your CDK usage, offers reviews of your CDK apps, and runs the Cloud Automation Weekly podcast, where I had the pleasure to be his first guest, discussing how I use daily AWS CDK in my Product Management work at Lumigo.
Healthchecks is a monitoring technique with a special place flavor: healthchecks are fired off at regular, frequent intervals (sometimes every 10 seconds, sometimes 1 minute) by orchestration platforms and monitoring tools. Most healthchecks are HTTP-based, and the returned HTTP response is checks based on the status code and (sometimes) content. But really, the only healthchecks a person needs to know about, are those that fail, which usually lead to containers being torn down and other disruptive infrastructure changes.
Given that Lumigo's pricing model is based on the amount of requests we process, the large amount of successful healthchecks that every container workload undergoes leads to undesirable consumption of quota, for data that is effectively not useful. Moreover, successful healthchecks lead to noise in the Explore and Transactions view, degrading the overall experience.
Luckily, one can often spot recognize HTTP requests that are healthchecks pretty easily! Both AWS ELB health-checks, as well as Kubernetes ones (including EKS), come in with specific User-Agent headers. Lumigo now automatically drops in the data processing pipeline in the Lumigo platform all the spans that:
User-Agent HTTP header with values that are known to be health-checks, specifically ELB-HealthChecker/* (AWS ELB, often used with Amazon ECS) and kube-probe/<kubelet_version> (Kubernetes, including Amazon EKS)500), usually something bad is about to happen to your containers :-)Nothing. It just works with every version of tracers we released so far for containers and all HTTP OpenTelemetry instrumentations we have ever seen. Enjoy :-)
P.S.: Matching health-checks by path (e.g., /health) sounds like a good solution on paper, but in practice it leads to very annoying false-positives (i.e., HTTP calls that are NOT related with health-checks). Moreover, healthcheck paths are configurable, and practitioners do make use for that configurability, which would lead to false negatives (health checks we let through). User-Agent headers, on the other hand, are far less often changed by healthcheck systems. User-agent matching, on the other hand, is usually rather reliable for this use-case.
We hear a lot from users that execution tags are one of Lumigo's most powerful features. Through execution tags, you can mark, filter and alert invocations based on data like on behalf of which of your customers did the invocation run. Developer Steve gave an excellent overview of execution tags in this Quick Bytes video.
Until now, execution tags were limited to the Node.js and Python Lumigo Lambda tracers. Today, any OpenTelemetry tracer, including but not limited to the Lumigo OpenTelemetry distributions, can send execution tags to Lumigo using OpenTelemetry's Span.setAttribute(key, value) API.
This is how you can set the execution tag foo to the value bar via the Lumigo OpenTelemetry Distributions for JS and Python:
// Typescript
import { trace } from '@opentelemetry/api';
// Note: 'trace.getActiveSpan()' is available from version 1.8.0 of the Lumigo OpenTelemetry Distro for JS
trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo','bar');// Javascript
const { trace } = require('@opentelemetry/api');
// Note: 'trace.getActiveSpan()' is available from version 1.8.0 of the Lumigo OpenTelemetry Distro for JS
trace.getActiveSpan().setAttribute('lumigo.execution_tags.foo','bar');// Python
from opentelemetry.trace import get_current_span
get_current_span().set_attribute('lumigo.execution_tags.foo','bar');Setting multiple values to the execution tag is also supported (this time, both bar and baz):
// Typescript
import { trace } from '@opentelemetry/api';
trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo',['bar','baz']);// Javascript
const { trace } = require('@opentelemetry/api');
trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo',['bar','baz']);
}// Python
from opentelemetry.trace import get_current_span
get_current_span().set_attribute('lumigo.execution_tags.foo',('bar','baz',)); # Both lists and tuples workThe APIs of OpenTelemetry SDKs for other programming languages differ slightly, but the gist of the matter is: if an OpenTelemetry tracer can send span attributes (and they all can :D), they can send execution tags to Lumigo!
Lambda and container tracers use different APIs to set execution tags, but you can filter by execution tags in Explore irrespective of their source:
Execution tags rely on the Span.setAttribute OpenTelemetry API, which is supported by virtually all OpenTelemetry SDKs. As far as looking up the current span is concerned, the trace.getActiveSpan() is available with the Lumigo OpenTelemetry Distro for JS version 1.8.0 and above. All versions of the Lumigo OpenTelemetry Distro for Python offer the opentelemetry.trace.get_current_span API.
* Execution tags documentation page
When we announced the support for the mongodb package in our OpenTelemetry Distro for JS, we heard a lot of feedback that can perhaps be best described as "Very cool, but what about Lambda Node.js?!?"
Fast forward to today: we have updated our Node.js Lambda tracer to support the mongodb version 4.x version that, among others, is compatible with MongoDB Atlas and AWS DocumentDB.
To trace mongodb v4.x, you need to:
@lumigo/tracer version 1.76.0 or above, or the Lumigo layer for Node.js in these versions or above (the actual version is dependent on the region).mongodb as an external package; refer to the MongoDB and Bundlers section of the Lumigo AWS Lambda Node.js documentation.Happy tracing :-)
Update: In the original post we pointed at the wrong version (1.75.1 instead of 1.76.0) of the @lumigo/tracer
You can now use Lumigo to easily create AWS Cloudwatch alarms and get notified in real-time in Slack, PagerDuty, and other tools when services like DynamoDB and SQS experience issues.
With Lumigo's new Cloudwatch metric alerts you'll no longer have to create alerts one-by-one in CloudWatch, and instead can configure multiple alerts at the same time, in just a few clicks. You can define dimensions and thresholds directly from the alert page, and setup notifications to your workflow tools without any need to integrate Cloudwatch manually.
To create a CloudWatch metric alert, go to the Alerts page and:
Once submitting the alert, Lumigo then creates the corresponding Alarms in CloudWatch.
The documentation about creating CloudWatch metric alerts with Lumigo is available here.
P.S. Ensure your IAM role includes the following permissions, or go to Settings > AWS and update to the latest IAM role.
{
"Effect" : "Allow",
"Action": [
"cloudwatch:PutMetricAlarm",
"cloudwatch:DeleteAlarms",
],
"Resources": "*"
}