Skip to content

Latest commit

 

History

History
609 lines (465 loc) · 24.1 KB

0474-event-bridge-scheduler-l2.md

File metadata and controls

609 lines (465 loc) · 24.1 KB

Event Bridge Scheduler L2

  • Original Author(s):: @filletofish, @Jacco
  • Tracking Issue: #473
  • API Bar Raiser: @kaizencc

Write one sentence which is a brief description of the feature. It should describe:

  • What is the user pain we are solving?
  • How does it impact users?

Library aws-events-scheduler contains L2 CDK constructs for creating, run, and manage scheduled tasks at scale with Amazon Event Bridge Scheduler.

Working Backwards

This section should contain one or more "artifacts from the future", as if the feature was already released and we are publishing its CHANGELOG, README, CONTRIBUTING.md and optionally a PRESS RELEASE. This is the most important section of your RFC. It's a powerful thought exercise which will challenge you to truly think about this feature from a user's point of view.

Choose one or more of the options below:

  • CHANGELOG: Write the changelog entry for this feature in conventional form (e.g. feat(eks): cluster tags). If this change includes a breaking change, include a BREAKING CHANGE clause with information on how to migrate. If migration is complicated, refer to a fictional GitHub issue and add its contents here.

  • README: If this is a new feature, write the README section which describes this new feature. It should describe the feature and walk users through usage examples and description of the various options and behavior.

  • PRESS RELEASE: If this is a major feature (~6 months of work), write the press release which announces this feature. The press release is a single page that includes 7 paragraphs: (1) summary, (2) problem, (3) solution, (4) leader quote, (5) user experience, (6) customer testimonial and (7) one sentence call to action.

CHANGELOG:

feat(events-scheduler): Event Bridge Scheduler L2 constructs

README:

See below

Amazon EventBridge Scheduler Contruct Library

Amazon EventBridge Scheduler is a feature from Amazon EventBridge that allows you to create, run, and manage scheduled tasks at scale. With EventBridge Scheduler, you can schedule one-time or recurrently tens of millions of tasks across many AWS services without provisioning or managing underlying infrastructure.

  1. Schedule: A schedule is the main resource you create, configure, and manage using Amazon EventBridge Scheduler. Every schedule has a schedule expression that determines when, and with what frequency, the schedule runs. EventBridge Scheduler supports three types of schedules: rate, cron, and one-time schedules. When you create a schedule, you configure a target for the schedule to invoke.
  2. Targets: A target is an API operation that EventBridge Scheduler calls on your behalf every time your schedule runs. EventBridge Scheduler supports two types of targets: templated targets and universal targets. Templated targets invoke common API operations across a core groups of services. For example, EventBridge Scheduler supports templated targets for invoking AWS Lambda Function or starting execution of Step Function state machine. For API operations that are not supported by templated targets you can use customizeable universal targets. Universal targets support calling more than 6,000 API operations across over 270 AWS services.
  3. Schedule Group: A schedule group is an Amazon EventBridge Scheduler resource that you use to organize your schedules. Your AWS account comes with a default scheduler group. A new schedule will always be added to a scheduling group. If you do not provide a scheduling group to add to, it will be added to the default scheduling group. You can create up to 500 schedule groups in your AWS account. Groups can be used to organize the schedules logically, access the schedule metrics and manage permissions at group granularity (see details below). Scheduling groups support tagging: with EventBridge Scheduler, you apply tags to schedule groups, not to individual schedules to organize your resources.

This module is part of the AWS Cloud Development Kit project. It allows you to define Event Bridge Schedules.

Defining a schedule

const target = new targets.LambdaInvoke(props.func, {
    input: ScheduleTargetInput.fromObject({
        "payload": "useful",
    }),
});

const schedule = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.rate(Duration.minutes(10)),
    target,
    description: 'This is a test schedule that invokes lambda function every 10 minutes.',
});

Schedule Expressions

You can choose from three schedule types when configuring your schedule: rate-based, cron-based, and one-time schedules.

Both rate-based and cron-based schedules are recurring schedules. You can configure each recurring schedule type using a schedule expression. For cron- based schedule you can specify a time zone in which EventBridge Scheduler evaluates the expression.

const rateBasedSchedule = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.rate(Duration.minutes(10)),
    target,
    description: 'This is a test rate-based schedule',
});

const cronBasedSchedule = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.cron({
        minute: '0',
        hour: '23',
        day: '20',
        month: '11',
        timeZone: TimeZone.AMERICA_NEW_YORK,
    }),
    target,
    description: 'This is a test cron-based schedule that will run at 11:00 PM, on day 20 of the month, only in November in New York timezone',
});

A one-time schedule is a schedule that invokes a target only once. You configure a one-time schedule when by specifying the time of the day, date, and time zone in which EventBridge Scheduler evaluates the schedule.

const oneTimeSchedule = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.at(
        new Date(2022, 10, 20, 19, 20, 23)
        TimeZone.AMERICA_NEW_YORK
    )
    target,
    description: 'This is a one-time schedule in New York timezone',
});

Grouping Schedules

Your AWS account comes with a default scheduler group, which you can access in CDK with:

const defaultGroup = Group.fromDefaultGroup(this, "DefaultGroup");

When creating a new schedule, you can also add the schedule to a custom scheduling group managed by you:

const group = new Group(this, "Group", {
    groupName: "MyGroup",
});

const target = new targets.LambdaInvoke(props.func, {
    input: ScheduleTargetInput.fromObject({
        "payload": "useful",
    }),
});

const schedule1 = new Schedule(this, 'Schedule1', {
    scheduleExpression: ScheduleExpression.rate(Duration.minutes(10)),
    target,
});

const schedule2 = new Schedule(this, 'Schedule2', {
    scheduleExpression: ScheduleExpression.rate(Duration.minutes(5)),
    target,
});

group.addSchedules(schedule1, schedule2);

// You can also assign groups in a schedule constructor:
const schedule3 = new Schedule(this, 'Schedule2', {
    scheduleExpression: ScheduleExpression.rate(Duration.minutes(5)),
    group,
    target,
});

Scheduler Targets

The @aws-cdk/aws-schedule-targets module includes classes that implement the IScheduleTarget interface for various AWS services. EventBridge Scheduler supports two types of targets: templated targets invoke common API operations across a core groups of services, and customizeable universal targets that you can use to call more than 6,000 operations across over 270 services.

Templated targets are a set of common API operations across a group of core AWS services such as Amazon SQS, Lambda, and Step Functions. The module contains CDK constructs for templated targets. For example, you can use construct targets.LambdaInvoke to target Lambda's Invoke API operation by providing a lambda.IFunction, or use targets.SqsSendMessage with a sqs.IQueue to send a message to SQS queue.

The following templated targets are supported:

  1. targets.CodeBuildStartBuild: Start an AWS CodeBuild build
  2. targets.CodePipelineStartPipelineExecution: Start an AWS CodePipeline pipeline execution
  3. targets.StepFunctionsStartExecution: Trigger an AWS Step Functions state machine
  4. targets.KinesisDataFirehosePutRecord: Put a record to a Kinesis Firehose
  5. targets.KinesisStreamPutRecord: Put a record to a Kinesis Stream
  6. targets.InspectorStartAssessmentRun: Start an AWS Inspector assessment run
  7. targets.EventBridgePutEvents: Put an event to an AWS Event Bridge Event Bus
  8. targets.EcsTask: Start a task on an Amazon ECS cluster
  9. targets.SageMakerStartPipelineExecution: Start Amazon SageMaker pipeline
  10. targets.LambdaInvoke: Invoke an AWS Lambda function
  11. targets.SqsSendMessage: Send a message to an Amazon SQS Queue
  12. targets.SnsPublish: Publish a message into an SNS topic

A universal target is a customizable set of parameters that allow you to invoke a wider set of API operation for many AWS services. To create a universal target you need to specify input, service name and service action. See full list of support universal targets.

For example, to create a schedule that will create a SQS queue every hour:

const input = ScheduleTargetInput.fromObject({
    'QueueName': 'MyQueue'
});

const schedule = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.rate(Duration.hours(1)),
    target: new targets.Universal('sqs', 'CreateQueue', { input: input })
});

Input

Target can be invoked with a custom input. Class ScheduleTargetInput supports free form text input and JSON-formatted object input:

const input = ScheduleTargetInput.fromObject({
    'QueueName': 'MyQueue'
});

You can include context attributes in your target payload. EventBridge Scheduler will replace each keyword with its respective value and deliver it to the target. See full list of supported context attributes:

  1. ContextAttribute.scheduleArn() – The ARN of the schedule.
  2. ContextAttribute.scheduledTime() – The time you specified for the schedule to invoke its target, for example, 2022-03-22T18:59:43Z.
  3. ContextAttribute.executionId() – The unique ID that EventBridge Scheduler assigns for each attempted invocation of a target, for example, d32c5kddcf5bb8c3.
  4. ContextAttribute.attemptNumber() – A counter that identifies the attempt number for the current invocation, for example, 1.
const text = `Attempt number: ${ContextAttribute.attemptNumber}`;
const input = scheduler.ScheduleTargetInput.fromInput(text);

Specifying Execution Role

An execution role is an IAM role that EventBridge Scheduler assumes in order to interact with other AWS services on your behalf.

The classes for templated schedule targets automatically create an IAM role with all the minimum necessary permissions to interact with the templated target. If you wish you may specify your own IAM role, then the templated targets will grant minimal required permissions. For example: for invoking Lambda function target LambdaInvoke will grant execution IAM role permission to lambda:InvokeFunction.

import * as iam from '@aws-cdk/aws-iam';

declare const fn: lambda.Function;

const role = new iam.Role(this, 'Role', {
  assumedBy: new iam.ServicePrincipal('scheduler.amazonaws.com'),
});

const target = new targets.LambdaInvoke({
    input: ScheduleTargetInput.fromObject({
        "payload": "useful"
    }),
    role,
}, fn);

Universal target automatically create an IAM role if you do not specify your own IAM role. However, in comparison with templated targets, for universal targets you must grant the required IAM permissions yourself.

const input = ScheduleTargetInput.fromObject({
    'QueueName': 'MyQueue'
});

const target = new targets.Universal('sqs', 'CreateQueue', { input: input });

target.role.addToPolicy(new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    resources: '*',
    actions: ['sqs:CreateQueue']
}));

const schedule = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.rate(Duration.hours(1)),
    target
});

Cross-account and cross-region targets

Executing cross-account and cross-region targets are not supported yet.

Specifying Encryption key

EventBridge Scheduler integrates with AWS Key Management Service (AWS KMS) to encrypt and decrypt your data using an AWS KMS key. EventBridge Scheduler supports two types of KMS keys: AWS owned keys, and customer managed keys.

By default, all events in Scheduler are encrypted with a key that AWS owns and manages. If you wish you can also provide a customer managed key to encrypt and decrypt the payload that your schedule delivers to its target. Target classes will automatically add AWS KMS Decrypt permission to your schedule's execution role permissions policy

import * as kms from '@aws-cdk/aws-kms';

const key = new kms.Key(this, 'Key');

const schedule = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.rate(Duration.minutes(10)),
    target,
    key: key,
});

See: Data Protection in the Amazon EventBridge Scheduler User Guide.

Error-handling

You can configure how your schedule handles failures, when EventBridge Scheduler is unable to deliver an event successfully to a target, by using two primary mechanisms: a retry policy, and a dead-letter queue (DLQ).

A retry policy determines the number of times EventBridge Scheduler must retry a failed event, and how long to keep an unprocessed event.

A DLQ is a standard Amazon SQS queue EventBridge Scheduler uses to deliver failed events to, after the retry policy has been exhausted. You can use a DLQ to troubleshoot issues with your schedule or its downstream target. If you've configured a retry policy for your schedule, EventBridge Scheduler delivers the dead-letter event after exhausting the maximum number of retries you set in the retry policy.

const dlq = new Queue(this, "DLQ", {
      queueName: 'MyDLQ',
});

const target = new targets.LambdaInvoke(props.func, {
    deadLetterQueue: dlq,
    maximumEventAge: Duration.minutes(1),
    maximumRetryAttempts: 3
});

Overriding Target Properties

If you wish to reuse the same target in multiple schedules, you can override target properties like input, maximumRetryAttempts and maximumEventAge when creating a Schedule:

const target = new targets.LambdaInvoke(props.func, {
    input: ScheduleTargetInput.fromObject({
        "payload": "useful"
    }),
});

const schedule1 = new Schedule(this, 'Schedule', {
    scheduleExpression: ScheduleExpression.cron({ day: '20' }),
    target,
});

const schedule2 = new Schedule(this, 'Schedule2', {
    scheduleExpression: ScheduleExpression.cron({ day: '5' })
    target,
    targetOverrides: {
        input: ScheduleTargetInput.fromText("Overriding Target Input")
    }
});

Monitoring

You can monitor Amazon EventBridge Scheduler using CloudWatch, which collects raw data and processes it into readable, near real-time metrics. EventBridge Scheduler emits a set of metrics for all schedules, and an additional set of metrics for schedules that have an associated dead-letter queue (DLQ). If you configure a DLQ for your schedule, EventBridge Scheduler publishes additional metrics when your schedule exhausts its retry policy.

Metrics for all schedules

Class Schedule provides static methods for accessing all schedules metrics with default configuration, such as metricAllErrors for viewing errors when executing targets.

import * as cloudwatch from '@aws-cdk/aws-cloudwatch';

new Alarm(this, 'SchedulesErrorAlarm', {
  metric: Schedule.metricAllErrors(),
  threshold: 0
});

Metrics for a Group

To view metrics for a specific group you can use methods on class Group:

const group = new Group(this, "Group", {
    groupName: "MyGroup"
});

new Alarm(this, 'MyGroupErrorAlarm', {
  metric: group.metricAllErrors(),
  threshold: 0
});

See full list of metrics and their description at Monitoring Using CloudWatch Metrics in the AWS Event Bridge Scheduler User Guide.

Ticking the box below indicates that the public API of this RFC has been signed-off by the API bar raiser (the status/api-approved label was applied to the RFC pull request):

[x] Signed-off by API Bar Raiser @kaizencc

Public FAQ

This section should include answers to questions readers will likely ask about this release. Similar to the "working backwards", this section should be written in a language as if the feature is now released.

The template includes a some common questions, feel free to add any questions that might be relevant to this feature or omit questions that you feel are not applicable.

What are we launching today?

What exactly are we launching? Is this a new feature in an existing module? A new module? A whole framework? A change in the CLI?

We are launching a new module (@aws-cdk/aws-eventbridge-scheduler) that contains L2 constructs for managing Amazon EventBridge Scheduler schedules and targets.

With EventBridge Scheduler L2 CDK constructs, you can schedule in code one-time or recurrently tens of millions of tasks across many AWS services without provisioning or managing underlying infrastructure.

Out of the box, we are launching with 12 Scheduler Targets for AWS service destinations (LambdaInvoke, SqsSendMessage, StepFunctionsStartExecution), as well as a Universal Scheduler Target so that customers can connect to other AWS services. These targets are located in a secondary module (@aws-cdk/aws-eventbridge-scheduler-targets).

Why should I use this feature?

Describe use cases that are addressed by this feature.

EventBridge Scheduler is a tool that can be used to automate the creation of schedules for various tasks, such as sending reminders for tasks, starting and stopping Amazon EC2 instances, and managing subscription-based services. It can be useful for companies of different sizes and types, such as a task management system, a large organization with multiple AWS accounts, and SaaS providers. EventBridge Scheduler can help reduce costs, respect time zones, and manage scheduled tasks more efficiently. Using EventBridge Scheduler with CDK Constructs smooths many configuration edges and provides seamless integrations with your existing infrastructure as code.

Internal FAQ

The goal of this section is to help decide if this RFC should be implemented. It should include answers to questions that the team is likely ask. Contrary to the rest of the RFC, answers should be written "from the present" and likely discuss design approach, implementation plans, alternative considered and other considerations that will help decide if this RFC should be implemented.

Why are we doing this?

What is the motivation for this change?

The tracking Github issue for the module has the 19 +1s indicating that customers want L2 CDK Construct support for this service.

Here is how L2 CDK constructs for Event Scheduler can improve your AWS CDK development experience:

  1. CDK L2 Constructs can simplify the process of setting up permissions for invoking targets, such as AWS Lambda functions, by automatically configuring required permissions for the targets.
  2. Encrypt data by providing pre-configured encryption settings that you can use as part of your AWS infrastructure as code. This can make it easier to ensure that your data is encrypted at rest and in transit, and can help you meet compliance requirements.
  3. Add metrics and permissions to your AWS EventBridge schedules. You can use constructs to define CloudWatch alarms that monitor schedule activity, and IAM permissions that control access to schedules. This can make it easier to manage and monitor your schedules, as well as provide more granular control over who can modify them.
  4. Constructs provide pre-configured targets for common AWS services, such as Lambda functions and SQS queues, as well as a universal target that can be used to connect to other AWS services.

Why should we not do this?

Is there a way to address this use case with the current product? What are the downsides of implementing this feature?

We are not confident that the service API is fully set in stone and implementing an L2 on top of the current L1 may be setting us up for changes in the future. We are reaching out to the service team to get their input and plans for the service.

It’s a large effort (X devs * 1 week) to invest in a module when we have other pressing projects. However, the bulk of the effort has been spent already since we have fairly robust prototypes already implemented.

What is the technical solution (design) of this feature?

classDiagram
    ScheduleExpression <|-- Schedule : scheduleExpression
    Key <|-- Schedule : key
    ScheduleTargetBase <|-- Schedule : target
    Group <|-- Schedule : group
    Input <|-- ScheduleTargetBase : input
    Targets <.. ScheduleTargetBase : implements

    class Schedule {
        + String name
        + Duration flexibleTimeWindow
        + Date startDate
        + Date endDate
        + Boolean disabled

        + <> targetOverrideProps

        + grant(..)
        + grantRead(..)
        + grantWrite(..)
        + Metric metric..()
    }
    class ScheduleExpression {
        + ScheduleExpression at(date: Date, timezone? Timezone)
        + ScheduleExpression rate(duration: Duration)
        + ScheduleExpression cron(options: CronOptionsWithTimezone)
        + ScheduleExpression expression(..)
    }

    class Group {
        + String name
        + addSchedules(..Schedule)

        + Metric metric..()
        + grant(..)
        + grantRead(..)
        + grantWrite(..)
    }

    class ScheduleTargetBase {
        + sqs.IQueue deadLetterQueue
        + Duration maximumEventAge
        + Number maximumRetryAttempts
        + Role role
    }

    class Input {
        + Input fromText(..)
        + Input fromMultilineText(..)
        + Input fromObject(..)
        + Input fromInput(..)
    }
Loading

L2 Constructs Class Diagram

Prototype at github.com/filletofish/cdk-eb-scheduler.

Is this a breaking change?

No.

What alternative solutions did you consider?

Briefly describe alternative approaches that you considered. If there are hairy details, include them in an appendix.

  1. We have discussed an opportunity of reusing existing events.targets module. See RFC comments.

What are the drawbacks of this solution?

Describe any problems/risks that can be introduced if we implement this RFC.

N/A

What is the high-level project plan?

Describe your plan on how to deliver this feature from prototyping to GA. Especially think about how to "bake" it in the open and get constant feedback from users before you stabilize the APIs.

If you have a project board with your implementation plan, this is a good place to link to it.

Most part of the L2 constructs is already implemented. We have a set of relatively (1-2 day of work) todos that can be implemented in parallel:

  1. Decide what removal policy for Group should be
  2. Support and Test Encrypted Schedules
  3. Decide or clean up how IAM Roles are used in Schedules and Targets
  4. Check why name generation fails if env is not passed to the Stack
  5. Unit tests for class schedule
  6. Unit tests for class targets
  7. Unit tests for classes: groups, schedule expressions, input
  8. Methods for Granting Schedule Group Management IAM permissions
  9. Integration tests

Are there any open issues that need to be addressed later?

Describe any major open issues that this RFC did not take into account. Once the RFC is approved, create GitHub issues for these issues and update this RFC of the project board with these issue IDs.

N/A

Appendix

N/A