We are excited to announce Programmatic error support for Kubernetes and ECS workloads. Programmatic Errors empower developers to customize, monitor, and troubleshoot non-fatal errors that may occur within their applications. With Programmatic Error support (through OpenTelemetry), you can now flag issues that may not interfere with the overall service, such as business logic errors.

Programmatic errors are created by adding span events with a custom attribute with the key name 'lumigo.type'.

For example, in Javascript, you can add a programmatic error in the following manner:

const { trace } = require('@opentelemetry/api');

trace.getActiveSpan()?.addEvent('<error-message>', {'lumigo.type': '<error-type>'});

To learn more about Programmatic errors and troubleshooting microservices with Lumigo, check out our docs page, and our OpenTelemetry distros for Java, Node.js, and Python 

The Python 3.10 runtime brings the latest and greatest LTS version of Python 3.10 to Lambda, and as of today, it is officially supported in Lumigo!

What's new in the Python 3.10 runtime

What makes the Python 3.10 runtime very interesting is the structural pattern matching addition allowing to use patterns to match against data structures such as lists, tuples, and dictionaries and run code based on the match. Structural pattern matching lets you write code that processes complex data structures more easily and can help you improve code readability and maintainability.

Python 3.10 has also improved error messages providing more context about the error, such as the line number, the location where the error occurred, and the exact nature of the error. Additionally, Python 3.10 error messages provide more helpful information about how to fix the error, such as suggestions for correct syntax or usage.

Other nice additions to Python 3.10 are a new syntax for parenthesized context managers, making it easier to read and write code that uses the “with” statement, which simplifies managing resources such as file handles or database connections, ensuring they are released correctly, writing union types as X | Y instead of the previous versions’ syntax of typing Union[X, Y] and use user-defined type guards to handle custom data types or to refine the types of built-in types that aim to reduce boilerplate code and improve readability and maintainability.

What do you need to do in Lumigo to use the Python 3.10 runtime?

Just turn on Lumigo autotracing as always. If you went ahead and adopted Python 3.10 already, and Lumigo told you the runtime is not supported, by now you will have noticed that Lumigo set up autotracing for your Python 3.10 functions that you selected for autotracing in the UI. Of course, also using the lumigo:auto-trace=true AWS Tag is supported. Nice and easy :-)

In many cases, alerts our users configure in production are resource-specific. For example, they'd like to know when a specific Lambda function is experiencing elevated error rates. But in some cases, users need to alert differently based on the specific request. For example, they might want an alert to be more sensitive when errors are happening for a specific tenant, or when a specific value is present in the request.

Today, Lumigo adds this capability with Execution Tag based alerts, enabling users to configure alerts based on the request payload, whether it is a function invocation or a container request. To configure Execution-Tags based alerts, open the Condition section in the alert policy, select Execution Tags, and enter the key and value.

Before setting up execution tag-based alerts, make sure you've added execution tags for the data you'd like to alert on. Click here for more details about execution tags and how to add them.

Kubernetes is the container orchestration platform of choice for many teams. In our ongoing efforts to bring the magic experience of Lumigo’s serverless capabilities to the world of containerized applications, we are delighted to share with you the Lumigo Kubernetes operator, a best-in-class operator to automatically trace your applications running on Kubernetes using the Lumigo OpenTelemetry distributions for Java, Node.js and Python.

We designed the Lumigo Kubernetes operator with ease of use and automation in mind: to get your applications traced with Lumigo on Kubernetes, all you have to do is install the Lumigo Kubernetes operator with Helm, and then create in the namespaces where you want applications traced, a Lumigo resource:

apiVersion: v1
kind: Secret
metadata:
name: lumigo-credentials
stringData:
# Kubectl won't allow you to deploy this dangling anchor.
# Get the actual value from Lumigo following this documentation: https://docs.lumigo.io/docs/lumigo-tokens
token: *lumigo-token # Example: t_123456789012345678901
---
apiVersion: operator.lumigo.io/v1alpha1
kind: Lumigo
metadata:
labels:
app.kubernetes.io/name: lumigo
app.kubernetes.io/instance: lumigo
app.kubernetes.io/part-of: lumigo-operator
name: lumigo
spec:
lumigoToken:
secretRef:
name: lumigo-credentials # This must match the name of the secret; the secret must be in the same namespace as this Lumigo custom resource
key: token # This must match the key in the Kubernetes secret

(You find your Lumigo token in your Lumigo project.)

The Lumigo Kubernetes operator will update your deployments, daemonsets, replicasets, statefulsets, cronjobs and jobs to be traced with the Lumigo OpenTelemetry Distro for Java, Lumigo OpenTelemetry Distro for Node.js and Lumigo OpenTelemetry Distro for Python.

Monitoring applications on Kubernetes has never been easier ;-)

For more details on how to use the Lumigo Kubernetes Operator, refer to the documentation on GitHub. And if you are curios about how we managed to make it so easy, we have an in-depth look at the magic behind the Lumigo Kubernetes operator waiting for you on the Lumigo blog.

Many serverless and microservice applications are polyglots, with components written in different programming languages based on the task to accomplish, desired performance characteristics, libraries to reuse, developers' preferences and many other reasons. Lumigo now shows you the Lambda runtime and OpenTelemetry SDK language in the invocation and transactions graph, making it easier for you to contextualize the transaction and its components, and react faster to issues. Also, it is a nice splash of color ;-)

P.S. To keep transactions easy to read, the runtime will be hidden when, due to zoom level and screen size, graph nodes are less than 40px wide.

The Node.js 18 runtime brings the latest and greatest LTS version of Node.js to Lambda, and as of today, it is officially supported in Lumigo!

What's new in the Node.js 18 runtime

Node.js 18 itself brings a built-in test runner, the support for the Fetch API, and a few other quality-of-life improvements. But what makes the Node.js 18 runtime very interesting, is the adoption of the AWS SDK 3 for Javascript: it is a rewrite of the AWS SDK that focuses on first-class Typescript support and modularity.

Extolling the virtues of Typescript's static type system (as opposing to effectively not having static type checks like Javascript), is something that can get me going for a while, and I am sparing everyone it.

But modularity has implications for cold starts. It is a known fact that the size of the Lambda bundle affects cold starts. It is pretty much as the intuition goes: the more code to load into the Node runtime, the longer the startup.

Another nice thing that can speed up your Lambda functions is with AWS SDK 3 is enablement by default of the TCP connection re-use, which can shave significant amounts of time across invocations for Lambda instances that are kept busy.

What do you need to do in Lumigo to use the Node.js 18 runtime?

Just turn on Lumigo autotracing as always. If you went ahead and adopted Node.js 18 a few days ago, and Lumigo told you the runtime is not supported, by now you will have noticed that Lumigo by now has set up autotracing for your Node.js 18 functions that you selected for autotracing in the UI, or using the `lumigo:auto-trace=true` AWS Tag. Nice and easy :-)

System Map, one of Lumigo's most useful features that visualizes distributed application architectures to help understand service dependencies, just got much better. 

Today, we are introducing System Map autocomplete filters. Similarly to Explore's autocomplete filters, System Map filters make it easier to discover and inspect  the resources that comprise your application.

For more information on System-Map, see the System Map documentation.

The AWS Cloud Development Kit (CDK) enables you to define your cloud application resources using familiar programming languages, mixing an imperative programming style with the declarative nature of the underpinning AWS CloudFormation.

Today, we are launching @lumigo/cdk-constructs-v2: 1st-party, officially supported constructs to add Lumigo autotracing for your Lambda functions in CDK applications!

What is supported today?

The Lumigo construct can add Lumigo autotracing to all your Node.js and Python Lambda functions in a CDK app:

import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';

const app = new App();

// Add here stacks and constructs

new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')}).traceEverything(app);

app.synth();

Tracing can also be added to all Lambda functions in a stack:

import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';

export class NodejsStack extends Stack {

  constructor(scope: Construct, id: string, props: StackProps = {}) {
    super(scope, id, props);

      new Function(this, 'MyLambda', {
        code: new InlineCode('foo'),
        handler: 'index.handler',
        runtime: Runtime.NODEJS_14_X,
      });
    }

}

const app = new App();

const stack = new NodejsStack(app, 'NodejsTestStack', {
  env: {
    region: 'eu-central-1',
  }
}); 

new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')}).traceEverything(stack);

app.synth();

You can also add tracing to Lambda functions one by one:

import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';
interface NodejsStackProps extends StackProps {
  readonly lumigo: Lumigo;
}
export class NodejsStack extends Stack {
    constructor(scope: Construct, id: string, props: NodejsStackProps = {}) {
        super(scope, id, props);
        const handler = new Function(this, 'MyLambda', {
            code: new InlineCode('foo'),
            handler: 'index.handler',
            runtime: Runtime.NODEJS_14_X,
        });
        props.lumigo.traceLambda(handler);
    }
}
const app = new App();

const lumigo = new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')});

const stack = new NodejsStack(app, 'NodejsTestStack', {
    env: {
        region: 'eu-central-1',
    },
    lumigo,
}); 

app.synth();

Autotracing can be added to all Lambda functions declared as a Function from the aws-cdk-lib/aws-lambda package with a Node.js or Python runtime, NodejsFunction from the aws-cdk-lib/aws-lambda-nodejs and PythonFunction from the @aws-cdk/aws-lambda-python-alpha package.

Which version of the tracer will be used?

Each version of the @lumigo/cdk-constructs-v2 package comes with a built-in list of latest Lambda layers as of the time of publishing. We regularly (in a matter of hours) publish new versions of the @lumigo/cdk-constructs-v2 package when we release a new layer. Reproducibility is very important for AWS CDK apps, and that is why the same version of the @lumigo/cdk-constructs-v2 will always generate the same CloudFormation. (Moreover, it is bad practice to invoke APIs during the "synth" phase in CDK.) So, to get the newest tracers deployed, update the version of the @lumigo/cdk-constructs-v2 package and run "cdk deploy" anew.

It also possible for you to manually specify which version of the layer to use:

import { Lumigo } from '@lumigo/cdk-constructs-v2';
import { App, SecretValue } from 'aws-cdk-lib';

const app = new App();
// Add here stacks and constructs

new Lumigo({lumigoToken:SecretValue.secretsManager('LumigoToken')}).traceEverything(app, {
    lambdaNodejsLayerVersion: 42,  // All Lambda functions with a supported Node.js runtime will use the layer v42
    lambdaPythonLayerVersion: 1,  // All Lambda functions with a supported Python runtime will use the layer v1
});

app.synth();

Why using a CDK construct when Lumigo has autotracing in the UI?

You can indeed ask Lumigo to autotrace functions you manage with any of the tools out there that rely on CloudFormation (CDK itself, the Serverless framework, the AWS Serverless Application Model), but some may find that having Lumigo change infrastructure after deployment is not ideal in Infrastructure as Code (IaC) approaches. So, we are making it straightforward to add Lumigo tracing as an integral part of your IaC specification. This is why we built the Lumigo Serverless plugin before, and we are releasing the Lumigo CDK constructs today!

Isn't 'Lumigo.traceEverything' a little bold as an API name? After all, it's only Lambda, no?

Well, we are just getting started. There is some really interesting stuff happening with Lumigo in and beyond Serverless in the next few months ;-)

Credits

Special thanks to Thorsten Höger of Taimos, AWS DevTools Hero and author of "The CDK Book", for his invaluable feedback during the ideation and first implementation of the @lumigo/cdk-constructs-v2 package.

Thorsten offers various service packages to help you level up your CDK usage, offers reviews of your CDK apps, and runs the Cloud Automation Weekly podcast, where I had the pleasure to be his first guest, discussing how I use daily AWS CDK in my Product Management work at Lumigo.

Health-checks are peculiar things

Healthchecks is a monitoring technique with a special place flavor: healthchecks are fired off at regular, frequent intervals (sometimes every 10 seconds, sometimes 1 minute) by orchestration platforms and monitoring tools. Most healthchecks are HTTP-based, and the returned HTTP response is checks based on the status code and (sometimes) content. But really, the only healthchecks a person needs to know about, are those that fail, which usually lead to containers being torn down and other disruptive infrastructure changes.

Issues with health-checks in Lumigo

Given that Lumigo's pricing model is based on the amount of requests we process, the large amount of successful healthchecks that every container workload undergoes leads to undesirable consumption of quota, for data that is effectively not useful. Moreover, successful healthchecks lead to noise in the Explore and Transactions view, degrading the overall experience.

Cutting the Gordian knot

Luckily, one can often spot recognize HTTP requests that are healthchecks pretty easily! Both AWS ELB health-checks, as well as Kubernetes ones (including EKS), come in with specific User-Agent headers. Lumigo now automatically drops in the data processing pipeline in the Lumigo platform all the spans that:

1. Carry the User-Agent HTTP header with values that are known to be health-checks, specifically ELB-HealthChecker/* (AWS ELB, often used with Amazon ECS) and kube-probe/<kubelet_version> (Kubernetes, including Amazon EKS)
2. Return an HTTP status code that denotes a successful response (a.k.a.: `2xx` like `200 OK`, `201 Accepted`, etc.). This is because if a Health-check fails (e.g. returning HTTP status code 500), usually something bad is about to happen to your containers :-)

What do you need to do on your end?

Nothing. It just works with every version of tracers we released so far for containers and all HTTP OpenTelemetry instrumentations we have ever seen. Enjoy :-)

P.S.: Matching health-checks by path (e.g., /health) sounds like a good solution on paper, but in practice it leads to very annoying false-positives (i.e., HTTP calls that are NOT related with health-checks). Moreover, healthcheck paths are configurable, and practitioners do make use for that configurability, which would lead to false negatives (health checks we let through). User-Agent headers, on the other hand, are far less often changed by healthcheck systems. User-agent matching, on the other hand, is usually rather reliable for this use-case.