• Products
  • Documentation
  • Resources

Deploy ImageLabeller with Bitbucket

To demonstrate how to develop, deploy, and manage applications using Jira Software and various connected tools, our team created ImageLabeller, a simple demo application built on AWS that uses machine learning to apply labels to images.

This page covers how to deploy ImageLabeller with Bitbucket. Before you begin, we recommend reading the ImageLabeller architecture and AWS SageMaker setup pages for context.

Prerequisites

If you don’t already have SSH configured for your Bitbucket account, follow these instructions.

Public facing Github repositories with ImageLabeller code

https://github.com/AtlassianOpenDevOpsGuides

Bitbucket pipelines demo video

https://www.youtube.com/watch?v=QccHnSWE_BM

Create a repository for AWS S3 infrastructure

A standard developer loop typically has a developer picking up a task from Jira, moving it to work in progress, and then completing the development work. The Jira issue ID is the key which ties the development work to the Jira issue. It is the core integration component between the two systems.

From Jira, create a new issue for adding an AWS S3 infrastructure repository to Bitbucket. Take note of the issue ID. IM-5 in this example.

Create a new issue on Jira board

Go to Bitbucket > Create > Repository.

Create repository in Bitbucket

Select the appropriate Workspace and Project. Set the default branch name to mainline. Click Create repository to proceed.

Select Workspace in Project in modal

In your terminal, go to your s3_infra repository and run the following to push your AWS CloudFormation template.yml file to Bitbucket.

1 2 3 4 5 git add --all git commit -m "IM-5 add s3_infra repository to Bitbucket" git remote add origin git@bitbucket.org:pmmquickstartguides01/s3_infra.git git branch -m mainline git push -u origin mainline

Enable Bitbucket pipelines

Go to Repository settings > Settings > Enable Pipelines.

Enable pipelines in Bitbucket

Add an AWS access key repository variables

Go to Repository variables, enter your AWS access key ID, and click Add. Then enter your AWS secret access key and click Add.

Enter your AWS key ID

Set up deployment environments

Click Deployments.

Deployments tab

Click add environment to add new environments. There is a Test environment in US-WEST-1, a Staging environment in US-EAST-2, and three Production environments in US-WEST-2, US-EAST-1, and CA-CENTRAL-1 in this example.

add deployment environment

bitbucket-pipelines.yml for deploying to AWS

Go to your s3_infra repository in your terminal and create a branch named after your Jira issue ID.

1 git checkout -b IM-5

Create a bitbucket-pipelines.yml file with the following yaml. This defines a deployment workflow for your Test, Staging, and Production environments.

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 definitions: steps: - step: &deploy-test-usw1 name: Deploy Test us-west-1 script: - pipe: atlassian/aws-cloudformation-deploy:0.10.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-1' STACK_NAME: 'OpenDevOpsS3Infra' CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'] WAIT: 'true' TEMPLATE: 'template.yml' - step: &deploy-staging-use2 name: Deploy Staging us-east-2 script: - pipe: atlassian/aws-cloudformation-deploy:0.10.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-east-2' STACK_NAME: 'OpenDevOpsS3Infra' CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'] WAIT: 'true' TEMPLATE: 'template.yml' - step: &deploy-production-usw2 name: Deploy Production us-west-2 script: - pipe: atlassian/aws-cloudformation-deploy:0.10.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-2' STACK_NAME: 'OpenDevOpsS3Infra' CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'] WAIT: 'true' TEMPLATE: 'template.yml' - step: &deploy-production-use1 name: Deploy Production us-east-1 script: - pipe: atlassian/aws-cloudformation-deploy:0.10.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-east-1' STACK_NAME: 'OpenDevOpsS3Infra' CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'] WAIT: 'true' TEMPLATE: 'template.yml' - step: &deploy-production-cac1 name: Deploy Production ca-central-1 script: - pipe: atlassian/aws-cloudformation-deploy:0.10.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'ca-central-1' STACK_NAME: 'OpenDevOpsS3Infra' CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'] WAIT: 'true' TEMPLATE: 'template.yml' pipelines: default: - step: <<: *deploy-test-usw1 deployment: Test us-west-1 - step: <<: *deploy-staging-use2 deployment: Staging us-east-2 branches: mainline: - step: <<: *deploy-production-usw2 deployment: Production us-west-2 - step: <<: *deploy-production-use1 deployment: Production us-east-1 - step: <<: *deploy-production-cac1 deployment: Production ca-central-1

Understanding a bitbucket-pipelines.yml file

Definitions and steps

Define a set of steps in the definitions section. Each step has an alias that is referenced throughout the bitbucket-pipelines.yml file, a name that shows up in the Bitbucket deployment screen, and a script. A script is a collection of one or more commands.

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 definitions: steps: - step: &deploy-test-usw1 name: Deploy Test us-west-1 script: - pipe: atlassian/aws-cloudformation-deploy:0.10.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-1' STACK_NAME: 'OpenDevOpsS3Infra' CAPABILITIES: ['CAPABILITY_IAM', 'CAPABILITY_AUTO_EXPAND'] WAIT: 'true' TEMPLATE: 'template.yml'

Pipes

This step uses the atlassian/aws-cloudformation-deploy pipe to deploy the AWS S3 buckets defined in the s3_infra template.yml file.

1 - pipe: atlassian/aws-cloudformation-deploy:0.10.0

The set of available pipes is found here. The documentation for the atlassian/aws-cloudformation-deploy pipe is found here.

Pipelines

Pipelines run a set of steps. The default pipeline is the set of steps that is run for branches that are not explicitly named under pipelines. This default pipeline runs steps that deploy to Test and Staging environments.

1 2 3 4 5 6 7 8 pipelines: default: - step: <<: *deploy-test-usw1 deployment: Test us-west-1 - step: <<: *deploy-staging-use2 deployment: Staging us-east-2

Deployments enable integration between Jira and Bitbucket.

1 deployment: Test us-west-1

You can define a set of steps that run for specific, named branch under branches. The snippet below defines a set of steps for the mainline branch.

1 2 3 4 5 6 7 8 9 10 11 branches: mainline: - step: <<: *deploy-production-usw2 deployment: Production us-west-2 - step: <<: *deploy-production-use1 deployment: Production us-east-1 - step: <<: *deploy-production-cac1 deployment: Production ca-central-1

Read this reference article for more information on this topic.

Pushing to a feature branch

Run the following from the command line to push your changes to the IM-5 branch of your s3_infra repository. Include the Jira issue ID in commit messages, and branch names to enable the Jira Bitbucket integration to keep track of what is happening in your project.

1 2 3 git add --all git commit -m "IM-5 add bitbucket-pipelines.yml" git push -u origin IM-5

Click Pipelines, then IM-5 to see the running pipeline.

see the running pipeline

Click the pipeline itself to see details of the execution. The pipeline ran steps toscreenshot_s deploy to a Test environment in us-west-1 and a Staging environment in us-east-2.

see details of the execution

Create a pull request

To create a pull request click Pull requests, then Create pull request.

create pull request

Choose your feature branch as the source branch, check the Close branch checkbox, then click Create pull request.

create pull request modal

Review the code changes, then Approve and Merge the pull request.

approve and merge request

Clicking Merge opens the merge pull request screen. Check the Transition issue check box and click Merge.

transition issue check box

Click Pipelines to monitor the mainline pipeline.

monitor the main pipeline

The IM-5 branch is gone. The mainline branch is left, and a pipeline is running. Click the pipeline.

pipeline is running

The pipeline ran steps to deploy to Production environments in us-west-2, us-east-1, and ca-central-1. It is possible to rerun a pipeline by clicking the Rerun button if a pipeline fails.


Rollback a bad deployment

To rollback a deployment click Deployments.

deployments

Click the environment you want to rollback to get a list of historical deployments, choose the version you want to rollback to, then click Redeploy.

choose the version you want to deploy

Verify the change is correct, and click Redeploy.

verify the change is correct

Only the chosen environment is redeployed.

the chosen environment is redeployed

Create a repository for SubmitImage AWS Lambda

Go to Jira and create a Jira issue for adding a SubmitImage repository to Bitbucket. Take note of the Jira issue ID. In this example it is IM-6.

create a jira issue

Go to Bitbucket, and click Create, then Repository.

create repository

Select the appropriate Workspace, and Project. Set the Default branch name to mainline. Click Create repository to proceed.

select the workspace and project

In your terminal go to your SubmitImage repository, and run the following to push your AWS Lambda code to Bitbucket. Include the Jira issue ID in commit messages, and branch names to enable the Jira Bitbucket integration to keep track of what is happening in your project.

1 2 3 4 5 git add --all git commit -m "IM-6 add SubmitImage to Bitbucket" git remote add origin git@bitbucket.org:pmmquickstartguides01/submitimage.git git branch -m mainline git push -u origin mainline

Enable pipelines

Go to Repository settings, click Settings, then Enable pipelines.

enable pipelines

Add repository variables

Click Repository variables to add your AWS access key ID, AWS secret access key, and AWS account ID. Give the IAM user associated with the AWS access key AdministratorAccess. You can opt to use more fine grained access control by choosing individual AWS access policies.

repository variables

Add deployment environments

Click Deployments, then add environment to add new environments. There is a Test environment in us-west-1, a Staging environment in us-east-2, and three Production environments in us-west-2, us-east-1 and ca-central-1 in this example.

add deployment environment

Create SSH keys

Go to your SubmitImage repository in Bitbucket, and click Repository settings, then SSH keys, then Generate keys.

create SSH keys

You will need this SSH key when you create the SystemTests repository.

SSH key details

bitbucket-pipelines.yml for deploying to AWS

Go to your SubmitImage repository in your terminal, and create a branch named after your Jira issue ID.

1 git checkout -b IM-6

Create a bitbucket-pipelines.yml file with the following yaml. This defines a Bitbucket pipeline for your Test, Staging, and Production environments.

 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 definitions: steps: -step: &rununittests name: run unit tests image: golang:buster script: - cd submitImage - go test ./opendevopslambda/... -step: &deploy-test-usw1 name: Deploy Test us-west-1 image: amazon/aws-sam-cli-build-image-provided script: - curl https://dl.google.com/go/go1.16.3.linux-amd64.tar.gz -o go1.16.3.tar.gz - rm -rf /usr/local/go - tar -C /usr/local -xzf go1.16.3.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version - make - ls -lah /opt/atlassian/pipelines/agent/build/build/ - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-1' STACK_NAME: 'OpenDevOpsSubmitImage' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-west-1-${AWS_ACCOUNT_ID}/submit-image-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-west-1-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'build/template.yaml' -step: &integration-test-usw1 name: Integration test usw1 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-west-1 -step: &deploy-staging-use2 name: Deploy Staging us-east-2 image: amazon/aws-sam-cli-build-image-provided script: - curl https://dl.google.com/go/go1.16.3.linux-amd64.tar.gz -o go1.16.3.tar.gz - rm -rf /usr/local/go - tar -C /usr/local -xzf go1.16.3.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version - make - ls -lah /opt/atlassian/pipelines/agent/build/build/ - BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" - chmod -R o+rw $BITBUCKET_PIPE_SHARED_STORAGE_DIR - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-east-2' STACK_NAME: 'OpenDevOpsSubmitImage' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-east-2-${AWS_ACCOUNT_ID}/submit-image-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-east-2-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'build/template.yaml' -step: &integration-test-use2 name: Integration test use2 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-east-2 -step: &deploy-production-usw2 name: Deploy Production us-west-2 image: amazon/aws-sam-cli-build-image-provided script: - curl https://dl.google.com/go/go1.16.3.linux-amd64.tar.gz -o go1.16.3.tar.gz - rm -rf /usr/local/go - tar -C /usr/local -xzf go1.16.3.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version - make - ls -lah /opt/atlassian/pipelines/agent/build/build/ - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-2' STACK_NAME: 'OpenDevOpsSubmitImage' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-west-2-${AWS_ACCOUNT_ID}/submit-image-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-west-2-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'build/template.yaml' -step: &integration-test-usw2 name: Integration test usw2 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-west-2 -step: &deploy-production-use1 name: Deploy Production us-east-1 image: amazon/aws-sam-cli-build-image-provided script: - curl https://dl.google.com/go/go1.16.3.linux-amd64.tar.gz -o go1.16.3.tar.gz - rm -rf /usr/local/go - tar -C /usr/local -xzf go1.16.3.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version - make - ls -lah /opt/atlassian/pipelines/agent/build/build/ - BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" - chmod -R o+rw $BITBUCKET_PIPE_SHARED_STORAGE_DIR - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-east-1' STACK_NAME: 'OpenDevOpsSubmitImage' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-east-1-${AWS_ACCOUNT_ID}/submit-image-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-east-1-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'build/template.yaml' -step: &integration-test-use1 name: Integration test use1 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-east-1 -step: &deploy-production-cac1 name: Deploy Production ca-central-1 image: amazon/aws-sam-cli-build-image-provided script: - curl https://dl.google.com/go/go1.16.3.linux-amd64.tar.gz -o go1.16.3.tar.gz - rm -rf /usr/local/go - tar -C /usr/local -xzf go1.16.3.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version - make - ls -lah /opt/atlassian/pipelines/agent/build/build/ - BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" - chmod -R o+rw $BITBUCKET_PIPE_SHARED_STORAGE_DIR - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'ca-central-1' STACK_NAME: 'OpenDevOpsSubmitImage' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-ca-central-1-${AWS_ACCOUNT_ID}/submit-image-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-ca-central-1-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'build/template.yaml' -step: &integration-test-cac1 name: Integration test cac1 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=ca-central-1 pipelines: default: - step: *rununittests - step: <<: *deploy-test-usw1 deployment: Test us-west-1 # - step: *integration-test-usw1 - step: <<: *deploy-staging-use2 deployment: Staging us-east-2 # - step: *integration-test-use2 branches: mainline: - step: <<: *deploy-production-usw2 deployment: Production us-west-2 # - step: *integration-test-usw2 - step: <<: *deploy-production-use1 deployment: Production us-east-1 # - step: *integration-test-use1 - step: <<: *deploy-production-cac1 deployment: Production ca-central-1 # - step: *integration-test-cac1

The execution of the integration tests is commented out for now. The system tests will only pass when the entire application is deployed. Uncomment the integration test steps in your repository, and do another push to run the deployment pipeline after all components of ImageLabeller are deployed.

Understanding a bitbucket-pipelines.yml file

This step runs unit tests from the SubmitImage code base.

1 2 3 4 5 6 -step: &rununittests name: run unit tests image: golang:buster script: - cd submitImage - go test ./opendevopslambda/...

This step uses AWS SAM to deploy your SubmitImage AWS Lambda to us-west-2. The documentation for this pipe can be found here.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 -step: &deploy-usw2-prod name: deploy us-west-2 prod image: amazon/aws-sam-cli-build-image-provided script: - curl https://dl.google.com/go/go1.16.3.linux-amd64.tar.gz -o go1.16.3.tar.gz - rm -rf /usr/local/go - tar -C /usr/local -xzf go1.16.3.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version - make - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-2' STACK_NAME: 'OpenDevOpsSubmitImage' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-west-2-${AWS_ACCOUNT_ID}/submit-image-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-west-2-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'build/template.yaml'

This step clones the SystemTests repository, and runs integrations tests in us-west-2.

1 2 3 4 5 6 7 -step: &integration-test-usw2 name: Integration test usw2 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-west-2

Pushing to a feature branch

From the command line run the following to push your changes to the IM-8 branch of your SubmitImage repository.

1 2 3 git add --all git commit -m "IM-6 add bitbucket-pipelines.yml to SubmitImage" git push -u origin IM-6

Click Pipelines, then IM-6 to see the running pipeline.

screenshot of running pipeline

Create a pull request

To create a pull request click Pull requests, then Create pull request. Finish the pull request, and click Pipelines to see the Production deployment.

screenshot of production deployment in bitbucket pipelines

Create a repository for InvokeLabeller AWS Lambda

Go to Jira and create a Jira issue for adding an InvokeLabeller repository to Bitbucket. Take note of the Jira issue ID. In this example it is IM-10.

screenshot of jira issue to add invokelabeller AWS lambda

Go to Bitbucket, and click Create, then Repository. Select the appropriate Workspace, and Project. Set the Default branch name to mainline. Click Create repository to proceed.

screenshot of creating bitbucket repository "invokelabeller"

In your terminal go to your InvokeLabeller repository, and run the following to push your AWS Lambda code to Bitbucket. Include the Jira issue ID in commit messages, and branch names to enable the Jira Bitbucket integration to keep track of what is happening in your project.

1 2 3 4 5 git add --all git commit -m "IM-10 add InvokeLabeller to Bitbucket" git remote add origin git@bitbucket.org:pmmquickstartguides01/invokelabeller.git git branch -m mainline git push -u origin mainline

Enable pipelines

Go to Repository settings, click Settings, then Enable pipelines.

enable pipelines

Add repository variables

Click Repository variables to add your AWS access key ID, AWS secret access key, and AWS account ID. Give the IAM user associated with the AWS access key AdministratorAccess. You can opt to use more fine grained access control by choosing individual AWS access policies, but the details are left to the reader.

repository variables

Add deployment environments

Click Deployments, then add environment to add new environments. There is a Test environment in us-west-1, a Staging environment in us-east-2, and three Production environments in us-west-2, us-east-1 and ca-central-1 in this example.

add deployment environment

Create SSH keys

Go to your SubmitImage repository in Bitbucket, and click Repository settings, then SSH keys, then Generate keys.

SSH key details

bitbucket-pipelines.yml for deploying to AWS

Go to your InvokeLabeller repository in your terminal, and create a branch named after your Jira issue ID.

1 git checkout -b IM-10

Create a bitbucket-pipelines.yml file with the following yaml. This defines a Bitbucket pipeline for your Test, Staging, and Production environments.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 definitions: steps: -step: &rununittests name: run unit tests image: python:rc-buster script: - pip3 install pytest - pip3 install moto - pip3 install -r tst/requirements.txt --user - python3 -m pytest -v tst/unit --junitxml=test-reports/report.xml -step: &deploy-usw1-test name: deploy us-west-1 test image: amazon/aws-sam-cli-build-image-python3.8 script: - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-1' STACK_NAME: 'OpenDevOpsImageLabeller' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-west-1-${AWS_ACCOUNT_ID}/image-labeller-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-west-1-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'template.yml' -step: &integration-test-usw1 name: integration test usw1 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-west-1 -step: &deploy-use2-staging name: deploy us-east-2 staging image: amazon/aws-sam-cli-build-image-python3.8 script: - BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" - chmod -R o+rw $BITBUCKET_PIPE_SHARED_STORAGE_DIR - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-east-2' STACK_NAME: 'OpenDevOpsImageLabeller' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-east-2-${AWS_ACCOUNT_ID}/image-labeller-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-east-2-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'template.yml' -step: &integration-test-use2 name: integration test use2 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-east-2 -step: &deploy-usw2-prod name: deploy us-west-2 prod image: amazon/aws-sam-cli-build-image-python3.8 script: - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-west-2' STACK_NAME: 'OpenDevOpsImageLabeller' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-west-2-${AWS_ACCOUNT_ID}/image-labeller-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-west-2-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'template.yml' -step: &integration-test-usw2 name: integration test usw2 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-west-2 -step: &deploy-use1-prod name: deploy us-east-1 prod image: amazon/aws-sam-cli-build-image-python3.8 script: - BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" - chmod -R o+rw $BITBUCKET_PIPE_SHARED_STORAGE_DIR - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'us-east-1' STACK_NAME: 'OpenDevOpsImageLabeller' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-us-east-1-${AWS_ACCOUNT_ID}/image-labeller-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-us-east-1-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'template.yml' -step: &integration-test-use1 name: integration test use1 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-east-1 -step: &deploy-cac1-prod name: deply ca-central-1 prod image: amazon/aws-sam-cli-build-image-python3.8 script: - BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" - chmod -R o+rw $BITBUCKET_PIPE_SHARED_STORAGE_DIR - pipe: atlassian/aws-sam-deploy:1.2.0 variables: AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION: 'ca-central-1' STACK_NAME: 'OpenDevOpsImageLabeller' CAPABILITIES: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND' ] TEMPLATE: 'https://s3.amazonaws.com/open-devops-code-ca-central-1-${AWS_ACCOUNT_ID}/image-labeller-packaged.yml' WAIT: 'true' DEBUG: 'true' S3_BUCKET: 'open-devops-code-ca-central-1-${AWS_ACCOUNT_ID}' SAM_TEMPLATE: 'template.yml' -step: &integration-test-cac1 name: integration test cac1 image: golang:buster script: - git clone git@bitbucket.org:pmmquickstartguides01/systemtests.git - cd systemtests - go test -v ./... -aws_region=ca-central-1 pipelines: default: - step: *rununittests - step: <<: *deploy-usw1-test deployment: us-west-1 Test # - step: *integration-test-usw1 - step: <<: *deploy-use2-staging deployment: us-east-2 Staging # - step: *integration-test-use2 branches: mainline: - step: <<: *deploy-usw2-prod deployment: us-west-2 Prod # - step: *integration-test-usw2 - step: <<: *deploy-use1-prod deployment: us-east-1 Prod # - step: *integration-test-use1 - step: <<: *deploy-cac1-prod deployment: ca-central-1 Prod # - step: *integration-test-cac1

The execution of the integration tests is commented out for now. The system tests will only pass when the entire application is deployed. Uncomment the integration test steps in your repository, and do another push to run the deployment pipeline after all components of ImageLabeller are deployed.

Understanding a bitbucket-pipelines.yml file

This step runs unit tests from the InvokeLabeller code base.

1 2 3 4 5 6 -step: &rununittests name: run unit tests image: golang:buster script: - cd submitImage - go test ./opendevopslambda/...

Update src/app.py with AWS SageMaker endpoint

Open InvokeLabeller’s src/app.py file and look for query_endpoint. Change the endpoint_name, and client region_name to match your AWS SageMaker notebook.

1 2 3 4 5 6 def query_endpoint(img): endpoint_name = 'jumpstart-dft-image-labeller-endpoint' client = boto3.client(service_name='runtime.sagemaker', region_name='us-west-1') response = client.invoke_endpoint(EndpointName=endpoint_name, ContentType='application/x-image', Body=img) model_predictions = json.loads(response['Body'].read())['predictions'][0] return model_predictions

Pushing to a feature branch

From the command line run the following to push yourchanges to the IM-10 branch of your InvokeLabeller repository.

1 2 3 git add --all git commit -m "IM-10 add bitbucket-pipelines.yml to InvokeLabeller" git push -u origin IM-10

Click Pipelines, then IM-10 to see the running pipeline.

screenshot of running pipeline in bitbucket

Create a pull request

To create a pull request click Pull requests, then Create pull request. Finish the pull request, and click Pipelines to see the Production deployment.

screenshot of production deployment for invokelabeller in bitbucket

Create a repository for System Tests

Go to Jira, and create a new issue for adding a SystemTests repository to Bitbucket. Make note of the issue ID. IM-7 in this example.

Create a jira issue for system tests

Go to Bitbucket, and click Create, then Repository.

Create a Bitbucket repository for SystemTests

Select the appropriate Workspace, and Project. Set the Default branch name to mainline. Click Create repository to proceed.

Create a new Bitbucket repository for SystemTests

In your terminal go to your SystemTests repository, and run the following to push your code to Bitbucket.

1 2 3 4 5 git add --all git commit -m "IM-7 add SystemTests repository to Bitbucket" git remote add origin git@bitbucket.org:pmmquickstartguides01/systemtests.git git branch -M mainline git push -u origin mainline

The systemTests repository doesn’t need a bitbucket-pipelines.yml file. It has no pipeline of its own since it provides tests for other pipelines to run.

Add SSH keys from SubmitImage, GetImageLabel, and InvokeLabeller

Click Repository settings, then Access keys.

Add SSH access keys to SystemTests

Click Add Key, paste the SSH key copied from SubmitImage, GetImageLabel, or InvokeLabeller, then click Add SSH key.

Add SSH key

If you’ve made this far, congratulations! You just deployed ImageLabeller. The next step is to set up monitoring ImageLabeller with Opsgenie.

 

Last modified on Sep 29, 2021
Cached at 3:23 AM on Oct 19, 2021 |

Additional Help