Skip navigation links

Package software.amazon.awscdk.pipelines

CDK Pipelines

See: Description

Package software.amazon.awscdk.pipelines Description

CDK Pipelines

---

cdk-constructs: Developer Preview

The APIs of higher level constructs in this module are in developer preview before they become stable. We will only make breaking changes to address unforeseen API issues. Therefore, these APIs are not subject to Semantic Versioning, and breaking changes will be announced in release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

A construct library for painless Continuous Delivery of CDK applications.

Developer Preview

This module is in developer preview. We may make breaking changes to address unforeseen API issues. Therefore, these APIs are not subject to Semantic Versioning, and breaking changes will be announced in release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

At a glance

Defining a pipeline for your application is as simple as defining a subclass of Stage, and calling pipeline.addApplicationStage() with instances of that class. Deploying to a different account or region looks exactly the same, the CDK Pipelines library takes care of the details.

(Note that have to bootstrap all environments before the following code will work, see the section CDK Environment Bootstrapping below). 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 /** The stacks for our app are defined in my-stacks.ts.  The internals of these
   * stacks aren't important, except that DatabaseStack exposes an attribute
   * "table" for a database table it defines, and ComputeStack accepts a reference
   * to this table in its properties.
   * /
 import lib.my.stacks.DatabaseStack;
 import lib.my.stacks.ComputeStack;
 
 import software.amazon.awscdk.core.Construct;
 import software.amazon.awscdk.core.Stage;
 import software.amazon.awscdk.core.Stack;
 import software.amazon.awscdk.core.StackProps;
 import software.amazon.awscdk.core.StageProps;
 import software.amazon.awscdk.pipelines.CdkPipeline;
 import software.amazon.awscdk.services.codepipeline.*;
 
 /**
  * Your application
  *
  * May consist of one or more Stacks (here, two)
  *
  * By declaring our DatabaseStack and our ComputeStack inside a Stage,
  * we make sure they are deployed together, or not at all.
  * /
 public class MyApplication extends Stage {
     public MyApplication(Construct scope, String id) {
         this(scope, id, null);
     }
 
     public MyApplication(Construct scope, String id, StageProps props) {
         super(scope, id, props);
 
         Object dbStack = new DatabaseStack(this, "Database");
         ComputeStack.Builder.create(this, "Compute")
                 .table(dbStack.getTable())
                 .build();
     }
 }
 
 /**
  * Stack to hold the pipeline
  * /
 public class MyPipelineStack extends Stack {
     public MyPipelineStack(Construct scope, String id) {
         this(scope, id, null);
     }
 
     public MyPipelineStack(Construct scope, String id, StackProps props) {
         super(scope, id, props);
 
         Artifact sourceArtifact = new Artifact();
         Artifact cloudAssemblyArtifact = new Artifact();
 
         CdkPipeline pipeline = new CdkPipeline(this, "Pipeline", new CdkPipelineProps());
 
         // Do this as many times as necessary with any account and region
         // Account and region may different from the pipeline's.
         pipeline.addApplicationStage(new MyApplication(this, "Prod", new StageProps()
                 .env(new Environment()
                         .account("123456789012")
                         .region("eu-west-1"))));
     }
 }

The pipeline is self-mutating, which means that if you add new application stages in the source code, or new stacks to MyApplication, the pipeline will automatically reconfigure itself to deploy those new stages and stacks.

CDK Versioning

This library uses prerelease features of the CDK framework, which can be enabled by adding the following to cdk.json: 

 {
   // ...
   "context": {
     "@aws-cdk/core:newStyleStackSynthesis": true
   }
 }

A note on cost

By default, the CdkPipeline construct creates an AWS Key Management Service (AWS KMS) Customer Master Key (CMK) for you to encrypt the artifacts in the artifact bucket, which incurs a cost of $1/month. This default configuration is necessary to allow cross-account deployments.

If you do not intend to perform cross-account deployments, you can disable the creation of the Customer Master Keys by passing crossAccountKeys: false when defining the Pipeline: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object pipeline = CdkPipeline.Builder.create(this, "Pipeline")
         .crossAccountKeys(false)
         .build();

Defining the Pipeline (Source and Synth)

The pipeline is defined by instantiating CdkPipeline in a Stack. This defines the source location for the pipeline as well as the build commands. For example, the following defines a pipeline whose source is stored in a GitHub repository, and uses NPM to build. The Pipeline will be provisioned in account 111111111111 and region eu-west-1: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 public class MyPipelineStack extends Stack {
     public MyPipelineStack(Construct scope, String id) {
         this(scope, id, null);
     }
 
     public MyPipelineStack(Construct scope, String id, StackProps props) {
         super(scope, id, props);
 
         Object sourceArtifact = new Artifact();
         Object cloudAssemblyArtifact = new Artifact();
 
         Object pipeline = CdkPipeline.Builder.create(this, "Pipeline")
                 .pipelineName("MyAppPipeline")
                 .cloudAssemblyArtifact(cloudAssemblyArtifact)
 
                 .sourceAction(GitHubSourceAction.Builder.create()
                         .actionName("GitHub")
                         .output(sourceArtifact)
                         .oauthToken(SecretValue.secretsManager("GITHUB_TOKEN_NAME"))
                         // Replace these with your actual GitHub project name
                         .owner("OWNER")
                         .repo("REPO")
                         .branch("main")
                         .build())
 
                 .synthAction(SimpleSynthAction.standardNpmSynth(Map.of(
                         "sourceArtifact", sourceArtifact,
                         "cloudAssemblyArtifact", cloudAssemblyArtifact,
 
                         // Optionally specify a VPC in which the action runs
                         "vpc", new Vpc(this, "NpmSynthVpc"),
 
                         // Use this if you need a build step (if you're not using ts-node
                         // or if you have TypeScript Lambdas that need to be compiled).
                         "buildCommand", "npm run build")))
                 .build();
     }
 }
 
 Object app = new App();
 MyPipelineStack.Builder.create(app, "PipelineStack")
         .env(Map.of(
                 "account", "111111111111",
                 "region", "eu-west-1"))
         .build();

If you prefer more control over the underlying CodePipeline object, you can create one yourself, including custom Source and Build stages: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object codePipeline = Pipeline.Builder.create(pipelineStack, "CodePipeline")
         .stages(asList(Map.of(
                 "stageName", "CustomSource",
                 "actions", asList(...)), Map.of(
                 "stageName", "CustomBuild",
                 "actions", asList(...))))
         .build();
 
 Object app = new App();
 Object cdkPipeline = CdkPipeline.Builder.create(app, "CdkPipeline")
         .codePipeline(codePipeline)
         .cloudAssemblyArtifact(cloudAssemblyArtifact)
         .build();

Initial pipeline deployment

You provision this pipeline by making sure the target environment has been bootstrapped (see below), and then executing deploying the PipelineStack once. Afterwards, the pipeline will keep itself up-to-date.

Important: be sure to git commit and git push before deploying the Pipeline stack using cdk deploy!

The reason is that the pipeline will start deploying and self-mutating right away based on the sources in the repository, so the sources it finds in there should be the ones you want it to find.

Run the following commands to get the pipeline going: 

 $ git commit -a
 $ git push
 $ cdk deploy PipelineStack

Administrative permissions to the account are only necessary up until this point. We recommend you shed access to these credentials after doing this.

Sources

Any of the regular sources from the @aws-cdk/aws-codepipeline-actions module can be used.

Synths

You define how to build and synth the project by specifying a synthAction. This can be any CodePipeline action that produces an artifact with a CDK Cloud Assembly in it (the contents of the cdk.out directory created when cdk synth is called). Pass the output artifact of the synth in the Pipeline's cloudAssemblyArtifact property.

SimpleSynthAction is available for synths that can be performed by running a couple of simple shell commands (install, build, and synth) using AWS CodeBuild. When using these, the source repository does not need to have a buildspec.yml. An example of using SimpleSynthAction to run a Maven build followed by a CDK synth: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object pipeline = CdkPipeline.Builder.create(this, "Pipeline")
         // ...
         .synthAction(SimpleSynthAction.Builder.create()
                 .sourceArtifact(sourceArtifact)
                 .cloudAssemblyArtifact(cloudAssemblyArtifact)
                 .installCommands(asList("npm install -g aws-cdk"))
                 .buildCommands(asList("mvn package"))
                 .synthCommand("cdk synth")
                 .build())
         .build();

Available as factory functions on SimpleSynthAction are some common convention-based synth:

If you need a custom build/synth step that is not covered by SimpleSynthAction, you can always add a custom CodeBuild project and pass a corresponding CodeBuildAction to the pipeline.

Adding Application Stages

To define an application that can be added to the pipeline integrally, define a subclass of Stage. The Stage can contain one or more stack which make up your application. If there are dependencies between the stacks, the stacks will automatically be added to the pipeline in the right order. Stacks that don't depend on each other will be deployed in parallel. You can add a dependency relationship between stacks by calling stack1.addDependency(stack2).

Stages take a default env argument which the Stacks inside the Stage will fall back to if no env is defined for them.

An application is added to the pipeline by calling addApplicationStage() with instances of the Stage. The same class can be instantiated and added to the pipeline multiple times to define different stages of your DTAP or multi-region application pipeline: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 // Testing stage
 pipeline.addApplicationStage(MyApplication.Builder.create(this, "Testing")
         .env(Map.of("account", "111111111111", "region", "eu-west-1"))
         .build());
 
 // Acceptance stage
 pipeline.addApplicationStage(MyApplication.Builder.create(this, "Acceptance")
         .env(Map.of("account", "222222222222", "region", "eu-west-1"))
         .build());
 
 // Production stage
 pipeline.addApplicationStage(MyApplication.Builder.create(this, "Production")
         .env(Map.of("account", "333333333333", "region", "eu-west-1"))
         .build());

Be aware that adding new stages via addApplicationStage() will automatically add them to the pipeline and deploy the new stacks, but removing them from the pipeline or deleting the pipeline stack will not automatically delete deployed application stacks. You must delete those stacks by hand using the AWS CloudFormation console or the AWS CLI.

More Control

Every Application Stage added by addApplicationStage() will lead to the addition of an individual Pipeline Stage, which is subsequently returned. You can add more actions to the stage by calling addAction() on it. For example: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object testingStage = pipeline.addApplicationStage(MyApplication.Builder.create(this, "Testing")
         .env(Map.of("account", "111111111111", "region", "eu-west-1"))
         .build());
 
 // Add a action -- in this case, a Manual Approval action
 // (for illustration purposes: testingStage.addManualApprovalAction() is a
 // convenience shorthand that does the same)
 testingStage.addAction(ManualApprovalAction.Builder.create()
         .actionName("ManualApproval")
         .runOrder(testingStage.nextSequentialRunOrder())
         .build());

You can also add more than one Application Stage to one Pipeline Stage. For example: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 // Create an empty pipeline stage
 Object testingStage = pipeline.addStage("Testing");
 
 // Add two application stages to the same pipeline stage
 testingStage.addApplication(MyApplication1.Builder.create(this, "MyApp1")
         .env(Map.of("account", "111111111111", "region", "eu-west-1"))
         .build());
 testingStage.addApplication(MyApplication2.Builder.create(this, "MyApp2")
         .env(Map.of("account", "111111111111", "region", "eu-west-1"))
         .build());

Even more, adding a manual approval action or reserving space for some extra sequential actions between 'Prepare' and 'Execute' ChangeSet actions is possible. 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 pipeline.addApplicationStage(new MyApplication(this, "Production"), Map.of(
         "manualApprovals", true,
         "extraRunOrderSpace", 1));

Adding validations to the pipeline

You can add any type of CodePipeline Action to the pipeline in order to validate the deployments you are performing.

The CDK Pipelines construct library comes with a ShellScriptAction which uses AWS CodeBuild to run a set of shell commands (potentially running a test set that comes with your application, using stack outputs of the deployed stacks).

In its simplest form, adding validation actions looks like this: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object stage = pipeline.addApplicationStage(new MyApplication());
 
 stage.addActions(ShellScriptAction.Builder.create()
         .actionName("MyValidation")
         .commands(asList("curl -Ssf https://my.webservice.com/"))
         // Optionally specify a VPC if, for example, the service is deployed with a private load balancer
         .vpc(vpc)
         // Optionally specify SecurityGroups
         .securityGroups(securityGroups)
         // Optionally specify a BuildEnvironment
         .environment(environment)
         .build());

Using CloudFormation Stack Outputs in ShellScriptAction

Because many CloudFormation deployments result in the generation of resources with unpredictable names, validations have support for reading back CloudFormation Outputs after a deployment. This makes it possible to pass (for example) the generated URL of a load balancer to the test set.

To use Stack Outputs, expose the CfnOutput object you're interested in, and call pipeline.stackOutput() on it: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 public class MyLbApplication extends Stage {
     public final Object loadBalancerAddress;
 
     public MyLbApplication(Construct scope, String id) {
         this(scope, id, null);
     }
 
     public MyLbApplication(Construct scope, String id, StageProps props) {
         super(scope, id, props);
 
         Object lbStack = new LoadBalancerStack(this, "Stack");
 
         // Or create this in `LoadBalancerStack` directly
         this.loadBalancerAddress = CfnOutput.Builder.create(lbStack, "LbAddress")
                 .value(String.format("https://%s/", lbStack.loadBalancer.getLoadBalancerDnsName()))
                 .build();
     }
 }
 
 MyLbApplication lbApp = MyLbApplication.Builder.create(this, "MyApp")
         .env(Map.of())
         .build();
 Object stage = pipeline.addApplicationStage(lbApp);
 stage.addActions(ShellScriptAction.Builder.create()
         // ...
         .useOutputs(Map.of(
                 // When the test is executed, this will make $URL contain the
                 // load balancer address.
                 "URL", pipeline.stackOutput(lbApp.getLoadBalancerAddress())))
         .build());

Using additional files in Shell Script Actions

As part of a validation, you probably want to run a test suite that's more elaborate than what can be expressed in a couple of lines of shell script. You can bring additional files into the shell script validation by supplying the additionalArtifacts property.

Here are some typical examples for how you might want to bring in additional files from several sources:

Controlling IAM permissions

IAM permissions can be added to the execution role of a ShellScriptAction in two ways.

Either pass additional policy statements in the rolePolicyStatements property: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 ShellScriptAction.Builder.create()
         // ...
         .rolePolicyStatements(asList(
             PolicyStatement.Builder.create()
                     .actions(asList("s3:GetObject"))
                     .resources(asList("*"))
                     .build()))
         .build();

The Action can also be used as a Grantable after having been added to a Pipeline: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object action = ShellScriptAction.Builder.create().build();
 pipeline.addStage('Test').addActions(action);
 
 bucket.grantRead(action);

Additional files from the source repository

Bringing in additional files from the source repository is appropriate if the files in the source repository are directly usable in the test (for example, if they are executable shell scripts themselves). Pass the sourceArtifact: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object sourceArtifact = new Artifact();
 
 Object pipeline = CdkPipeline.Builder.create(this, "Pipeline").build();
 
 Object validationAction = ShellScriptAction.Builder.create()
         .actionName("TestUsingSourceArtifact")
         .additionalArtifacts(asList(sourceArtifact))
 
         // 'test.sh' comes from the source repository
         .commands(asList("./test.sh"))
         .build();

Additional files from the synth step

Getting the additional files from the synth step is appropriate if your tests need the compilation step that is done as part of synthesis.

On the synthesis step, specify additionalArtifacts to package additional subdirectories into artifacts, and use the same artifact in the ShellScriptAction's additionalArtifacts: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 // If you are using additional output artifacts from the synth step,
 // they must be named.
 Object cloudAssemblyArtifact = new Artifact("CloudAsm");
 Object integTestsArtifact = new Artifact("IntegTests");
 
 Object pipeline = CdkPipeline.Builder.create(this, "Pipeline")
         .synthAction(SimpleSynthAction.standardNpmSynth(Map.of(
                 "sourceArtifact", sourceArtifact,
                 "cloudAssemblyArtifact", cloudAssemblyArtifact,
                 "buildCommands", asList("npm run build"),
                 "additionalArtifacts", asList(Map.of(
                         "directory", "test",
                         "artifact", integTestsArtifact)))))
         .build();
 
 Object validationAction = ShellScriptAction.Builder.create()
         .actionName("TestUsingBuildArtifact")
         .additionalArtifacts(asList(integTestsArtifact))
         // 'test.js' was produced from 'test/test.ts' during the synth step
         .commands(asList("node ./test.js"))
         .build();

Add Additional permissions to the CodeBuild Project Role for building and synthesizing

You can customize the role permissions used by the CodeBuild project so it has access to the needed resources. eg: Adding CodeArtifact repo permissions so we pull npm packages from the CA repo instead of NPM. 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 public class MyPipelineStack extends Stack {
     public MyPipelineStack(Construct scope, String id) {
         this(scope, id, null);
     }
 
     public MyPipelineStack(Construct scope, String id, StackProps props) {
         Object pipeline = CdkPipeline.Builder.create(this, "Pipeline")
                 (SpreadAssignment ...
                       synthAction
                   synthAction).SimpleSynthAction(SimpleSynthAction).(.standardNpmSynth(Map.of(
                         "sourceArtifact", sourceArtifact,
                         "cloudAssemblyArtifact", cloudAssemblyArtifact,
 
                         // Use this to customize and a permissions required for the build
                         // and synth
                         "rolePolicyStatements", asList(
                             PolicyStatement.Builder.create()
                                     .actions(asList("codeartifact:*", "sts:GetServiceBearerToken"))
                                     .resources(asList("arn:codeartifact:repo:arn"))
                                     .build()),
 
                         // Then you can login to codeartifact repository
                         // and npm will now pull packages from your repository
                         // Note the codeartifact login command requires more params to work.
                         "buildCommands", asList("aws codeartifact login --tool npm", "npm run build"))))
                 .build();
     }
 }

Developing the pipeline

The self-mutation feature of the CdkPipeline might at times get in the way of the pipeline development workflow. Each change to the pipeline must be pushed to git, otherwise, after the pipeline was updated using cdk deploy, it will automatically revert to the state found in git.

To make the development more convenient, the self-mutation feature can be turned off temporarily, by passing selfMutating: false property, example: 

 // Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
 Object pipeline = CdkPipeline.Builder.create(this, "Pipeline")
         .selfMutating(false)...
         .build();

CDK Environment Bootstrapping

An environment is an (account, region) pair where you want to deploy a CDK stack (see Environments in the CDK Developer Guide). In a Continuous Deployment pipeline, there are at least two environments involved: the environment where the pipeline is provisioned, and the environment where you want to deploy the application (or different stages of the application). These can be the same, though best practices recommend you isolate your different application stages from each other in different AWS accounts or regions.

Before you can provision the pipeline, you have to bootstrap the environment you want to create it in. If you are deploying your application to different environments, you also have to bootstrap those and be sure to add a trust relationship.

This library requires a newer version of the bootstrapping stack which has been updated specifically to support cross-account continous delivery. In the future, this new bootstrapping stack will become the default, but for now it is still opt-in.

The commands below assume you are running cdk bootstrap in a directory where cdk.json contains the "@aws-cdk/core:newStyleStackSynthesis": true setting in its context, which will switch to the new bootstrapping stack automatically.

If run from another directory, be sure to run the bootstrap command with the environment variable CDK_NEW_BOOTSTRAP=1 set.

To bootstrap an environment for provisioning the pipeline: 

 $ env CDK_NEW_BOOTSTRAP=1 npx cdk bootstrap \
     [--profile admin-profile-1] \
     --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \
     aws://111111111111/us-east-1

To bootstrap a different environment for deploying CDK applications into using a pipeline in account 111111111111: 

 $ env CDK_NEW_BOOTSTRAP=1 npx cdk bootstrap \
     [--profile admin-profile-2] \
     --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \
     --trust 11111111111 \
     aws://222222222222/us-east-2

These command lines explained:

Security tip: we recommend that you use administrative credentials to an account only to bootstrap it and provision the initial pipeline. Otherwise, access to administrative credentials should be dropped as soon as possible.


On the use of AdministratorAccess: The use of the AdministratorAccess policy ensures that your pipeline can deploy every type of AWS resource to your account. Make sure you trust all the code and dependencies that make up your CDK app. Check with the appropriate department within your organization to decide on the proper policy to use.

If your policy includes permissions to create on attach permission to a role, developers can escalate their privilege with more permissive permission. Thus, we recommend implementing permissions boundary in the CDK Execution role. To do this, you can bootstrap with the --template option with a customized template that contains a permission boundary.

Migrating from old bootstrap stack

The bootstrap stack is a CloudFormation stack in your account named CDKToolkit that provisions a set of resources required for the CDK to deploy into that environment.

The "new" bootstrap stack (obtained by running cdk bootstrap with CDK_NEW_BOOTSTRAP=1) is slightly more elaborate than the "old" stack. It contains:

It is possible and safe to migrate from the old bootstrap stack to the new bootstrap stack. This will create a new S3 file asset bucket in your account and orphan the old bucket. You should manually delete the orphaned bucket after you are sure you have redeployed all CDK applications and there are no more references to the old asset bucket.

Security Tips

It's important to stay safe while employing Continuous Delivery. The CDK Pipelines library comes with secure defaults to the best of our ability, but by its very nature the library cannot take care of everything.

We therefore expect you to mind the following:

Troubleshooting

Here are some common errors you may encounter while using this library.

Pipeline: Internal Failure

If you see the following error during deployment of your pipeline: 

 CREATE_FAILED  | AWS::CodePipeline::Pipeline | Pipeline/Pipeline
 Internal Failure

There's something wrong with your GitHub access token. It might be missing, or not have the right permissions to access the repository you're trying to access.

Key: Policy contains a statement with one or more invalid principals

If you see the following error during deployment of your pipeline: 

 CREATE_FAILED | AWS::KMS::Key | Pipeline/Pipeline/ArtifactsBucketEncryptionKey
 Policy contains a statement with one or more invalid principals.

One of the target (account, region) environments has not been bootstrapped with the new bootstrap stack. Check your target environments and make sure they are all bootstrapped.

is in ROLLBACK_COMPLETE state and can not be updated

If you see the following error during execution of your pipeline: 

 Stack ... is in ROLLBACK_COMPLETE state and can not be updated. (Service:
 AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request
 ID: ...)

The stack failed its previous deployment, and is in a non-retryable state. Go into the CloudFormation console, delete the stack, and retry the deployment.

Cannot find module 'xxxx' or its corresponding type declarations

You may see this if you are using TypeScript or other NPM-based languages, when using NPM 7 on your workstation (where you generate package-lock.json) and NPM 6 on the CodeBuild image used for synthesizing.

It looks like NPM 7 has started writing less information to package-lock.json, leading NPM 6 reading that same file to not install all required packages anymore.

Make sure you are using the same NPM version everywhere, either downgrade your workstation's version or upgrade the CodeBuild version.

Current Limitations

Limitations that we are aware of and will address:

Known Issues

There are some usability issues that are caused by underlying technology, and cannot be remedied by CDK at this point. They are reproduced here for completeness.

Skip navigation links

Copyright © 2021. All rights reserved.