• Products
  • Documentation
  • Resources

Deploy ImageLabeller with Gitlab

To demonstrate how to develop, deploy, and manage applications using Jira Software and various connected tools, our team created ImageLabeller, a simple demo application built on AWS that uses machine learning to apply labels to images.

This page covers how to deploy ImageLabeller with Gitlab. Before you begin, we recommend reading the ImageLabeller architecture and AWS SageMaker setup pages for context.

Prerequisites

If you don’t already have a Gitlab group, follow the steps in this Gitlab guide to create one from scratch.

Public facing Github repositories with ImageLabeller code

https://github.com/AtlassianOpenDevOpsGuides

Integrate Jira and GitLab

From Jira click Board, then Apps, then GitLab.

screenshot of "gitlab" highlighted in jira's app menu

Click Get it now.

screenshot of gitlab app summary in jira

Click Apps, then Manage your apps.

screenshot of "manage apps" highlighted in jira dropdown

Expand GitLab for Jira.

screenshot of "gitlab for jira" expanded

Click Add namespace.

screenshot of gitlab configuration in jira

Select your existing namespace, and click Link. This guide assumes you already have an existing GitLab account, and GitLab group.

screenshot of gitlab link namespace modal

Add SSH Key to GitLab

Click on your profile icon in the top right corner, and click preferences.

screenshot of "preferences" highlighted in gitlab repository dropdown

Click SSH Keys and follow the instructions to generate a new SSH key or use an existing SSH key.

Create a repository for AWS S3 infrastructure

A standard developer loop typically has a developer picking up a task from Jira, moving it to work in progress, and then doing the development work. The Jira issue ID is the key which ties the development work to the Jira issue. It is the core integration component between the two systems.

Go to Jira, and create a new issue for adding an AWS S3 infrastructure repository to GitLab. Make note of the issue ID. IM-5 in this example.

screenshot of creating jira issue to add s3_infra repository in gitlab

Go to GitLab and click New project.

screenshot of projects dashboard in gitlab

Click Create blank project.

screenshot of create new project options in gitlab

Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

screenshot of new project details form in gitlab

In your terminal go to your s3_infra repository, and run the following to push your AWS CloudFormation template.yml file to GitLab.

1 2 3 4 5 git add --all git commit -m "IM-5 add s3_infra repository to gitlab" git remote add origin git@gitlab.com:pmmquickstartguides/s3_infra.git git branch -m mainline git push -u origin mainline

Add AWS access key

Click Settings, then CI/CD. Scroll down, and expand Variables. Click Add variable.

screenshot of CI/CD variable page in gitlab

Create two variables. One for your AWS access key ID, and one for your AWS secret access key.

screenshot of add variables form in gitlab

Protect the variables so they are only used by pipelines running on protected branches, and tags. Give the IAM user associated with the AWS access key AdministratorAccess. You can also opt to use more fine grained access control by choosing individual AWS access policies.

screenshot of more access controls option in gitlab

Configure protected branches for access to protected variables

Click Settings, then Repository. Scroll down, and expand Protected branches.

Enter your Jira issue ID prefix and a *.

The Jira issue IDs are like IM-5, and IM-6 in this example; The prefix is IM-.

Enter IM-*, and click Protect.

screenshot of protected branches in gitlab

You will see mainline, and IM-* as protected branches.

Setup deployment environments

Click Deployments, then Environments. Click New environment to add new environments. There are Test environments in US-WEST-1, and US-EAST-2, and Production environments in US-WEST-2, US-EAST-1, and CA-CENTRAL-1 in this example.

screenshot of deployment environments in gitlab

.gitlab-ci.yml for deploying to AWS

Go to your s3_infra repository in your terminal, and create a branch named after your Jira issue ID.

1 git checkout -b IM-5

Create a .gitlab-ci.yml file with the following yaml. This defines a deployment workflow for your Test, Staging, and Production environments.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 stages: - merge-request - test-us-west-1 - test-us-east-2 - production-us-west-2 - production-us-east-1 - production-ca-central-1 merge-request-pipeline-job: stage: merge-request rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" script: - echo "This pipeline always succeeds and enables merges during merge requests" - echo true deploy-test-us-west-1: stage: test-us-west-1 environment: test-us-west-1 rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - aws cloudformation deploy --region us-west-1 --template-file template.yml --stack-name OpenDevOpsS3Infra deploy-test-us-east-2: stage: test-us-east-2 environment: test-us-east-2 rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - aws cloudformation deploy --region us-east-2 --template-file template.yml --stack-name OpenDevOpsS3Infra deploy-production-us-west-2: stage: production-us-west-2 environment: production-us-west-2 rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - aws cloudformation deploy --region us-west-2 --template-file template.yml --stack-name OpenDevOpsS3Infra deploy-production-us-east-1: stage: production-us-east-1 environment: production-us-east-1 rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - aws cloudformation deploy --region us-east-1 --template-file template.yml --stack-name OpenDevOpsS3Infra deploy-production-ca-central-1: stage: production-ca-central-1 environment: production-ca-central-1 rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - aws cloudformation deploy --region us-east-2 --template-file template.yml --stack-name OpenDevOpsS3Infra

Understanding a .gitlab-ci.yml file

Stages

Add a stages block to define the execution order of your GitLab pipeline. Stages are executed in the order they are defined in the stages block. Jobs associated with a stage are executed in parallel.

1 2 3 4 5 6 7 stages: - merge-request - test-us-west-1 - test-us-east-2 - production-us-west-2 - production-us-east-1 - production-ca-central-1

Jobs

Jobs are associated with a stage, and can be associated with an environment. Rules control whether, or not a particular job will execute. The rule in this example checks if the pipeline branch is not the default branch, and if the pipeline is not being run automatically as part of a merge request.

You can specify a different image for each job. You can setup images with tools necessary for your job scripts. The script section defines the set of steps that run when the job is executed.

1 2 3 4 5 6 7 8 9 10 deploy-test-us-west-1: stage: test-us-west-1 environment: test-us-west-1 rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - aws cloudformation deploy --region us-west-1 --template-file template.yml --stack-name OpenDevOpsS3Infra

Merge request pipeline

A pipeline is run automatically by GitLab when a merge request is approved . You can create a job for this pipeline by adding a rule. The job always succeeds in this example.

1 2 3 4 5 6 7 merge-request-pipeline-job: stage: merge-request rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" script: - echo "This pipeline always succeeds and enables merges during merge requests" - echo true

Pushing to a feature branch

Run the following from the command line to push your changes to the IM-5 branch of your s3_infra repository. Include the Jira issue ID in commit messages, and branch names to enable the Jira GitLab integration to keep track of what is happening in your project.

1 2 3 git add --all git commit -m "IM-5 add .gitlab-ci.yml to s3_infra" git push -u origin IM-5

Click CI/CD, then Pipelines to see the pipeline run.

screenshot of running pipeline in gitlab

Click the pipeline ID of the running pipeline.

screenshot of pipeline ID in gitlab

Click on a job to see more details.

screenshot of pipeline details in gitlab

Create a merge request

To create a merge request click Merge requests, then Create merge request.

screenshot of merge requests in gitlab

Choose your feature branch as the source branch then, click Compare branches and continue.

screenshot of creating new merge request in gitlab

Choose an Assignee, and a Reviewer.

screenshot of choosing assignees for merge request in gitlab

Click Create merge request.

screenshot of completed merge request form in gitlab

Review the code changes, then click Approve.

screenshot of reviewing merge request in gitlab

Click CI/CI, then click Pipelines to see the merge request pipeline run.

screenshot of pipelines run in gitlab

Click the pipeline ID. Notice that the merge-request-pipeline-job is the only job that ran.

screenshot of pipeline run in gitlab

Go back to the merge request by clicking Merge requests, then clicking the active merge request, and click Merge. This kicks off another pipeline.

screenshot of pull request in gitlab

Click CI/CD, then click Pipelines. Click the pipeline ID.

screenshot of pipeline ID in gitlab

Create a repository for SystemTests

Go to Jira and create a Jira issue for adding a SystemTests repository to GitLab. Take note of the Jira issue ID. In this example it is IM-7.

screenshot of adding a jira issue to create repo in gitlab

Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

screenshot of creating project "systems tests" in gitlab

In your terminal go to your SystemTests repository, and run the following to push your code to GitLab.

1 2 3 4 5 git add --all git commit -m "IM-7 add SystemTests repository to gitlab" git remote add origin git@gitlab.com:pmmquickstartguides/systemtests.git git branch -m mainline git push -u origin mainline

The SystemTests repository doesn’t need a .gitlab-ci.yml file. It has no pipeline of its own since it provides tests for other pipelines to run.

Add a Deploy token

You need to add a deploy token to clone this repository during the execution of other pipelines. Click Settings, then Repository. Scroll down, and expand Deploy tokens.

screenshot of deploy tokens page in gitlab

Enter a name, check read_repository, and click Create deploy token.

screenshot of create deploy token modal in gitlab

Create a repository for SubmitImage AWS Lambda

Go to Jira, and create a new issue for adding a SubmitImage AWS Lambda repository to GitLab. Make note of the issue ID. IM-8 in this example.

screenshot of creating jira issue to add repo "submitimage" in gitlab

Go to GitLab and click New project, then Create blank project. Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

screenshot of creating new project "submitimage" in gitlab

In your terminal go to your SubmitImage repository, and run the following to push your code to GitLab.

1 2 3 4 5 git add --all git commit -m "IM-8 add SubmitImage to gitlab" git remote add origin git@gitlab.com:pmmquickstartguides/submitimage.git git branch -m mainline git push -u origin mainline

You must add AWS access keys, configure protected branches, and setup deployment environments.

Then add the deploy keys from your SystemTests repository to enable the SubmitImage GitLab pipeline to download, and run the SystemTests.

Finally, add your AWS account ID as a CI/CD Variable.

screenshot of variables screen in gitlab

.gitlab-ci.yml for deploying to AWS

Go to your SubmitImage repository in your terminal, and create a branch named after your Jira issue ID.

1 git checkout -b IM-8

Create a .gitlab-ci.yml file with the following yaml. This defines a deployment workflow for your Test, Staging, and Production environments.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 stages: - merge-request - run-unit-tests #US-WEST-1 - deploy-us-west-1 - test-us-west-1 #US-EAST-2 - deploy-us-east-2 - test-us-east-2 #US-WEST-2 - deploy-us-west-2 - test-us-west-2 #US-EAST-1 - deploy-us-east-1 - test-us-east-1 #CA-CENTRAL-1 - deploy-ca-central-1 - test-ca-central-1 merge-request-pipeline-job: stage: merge-request rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" script: - echo "This pipeline always succeeds and enables merge" - echo true run-unit-tests: stage: run-unit-tests image: golang:buster rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" script: - cd submitImage - go test ./opendevopslambda/... #US-WEST-1 deploy-us-west-1: stage: deploy-us-west-1 environment: us-west-1 image: python:latest rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --region us-west-1 - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-1 --no-fail-on-empty-changeset #test-us-west-1: # stage: test-us-west-1 # environment: us-west-1 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-west-1 #US-EAST-2 deploy-us-east-2: stage: deploy-us-east-2 environment: us-east-2 image: python:latest rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --region us-east-2 - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-2 --no-fail-on-empty-changeset #test-us-east-2: # stage: test-us-east-2 # environment: us-east-2 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-east-2 #US-WEST-2 deploy-us-west-2: stage: deploy-us-west-2 environment: us-west-2 image: python:latest rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --region us-west-2 - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-2 --no-fail-on-empty-changeset #test-us-west-2: # stage: test-us-west-2 # environment: us-west-2 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-west-2 #US-EAST-1 deploy-us-east-1: stage: deploy-us-east-1 environment: us-east-1 image: python:latest rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --region us-east-1 - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-1 --no-fail-on-empty-changeset #test-us-east-1: # stage: test-us-east-1 # environment: us-east-1 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-east-1 #CA-CENTRAL-1 deploy-central-1: stage: deploy-ca-central-1 environment: ca-central-1 image: python:latest rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --region ca-central-1 - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region ca-central-1 --no-fail-on-empty-changeset #test-central-1: # stage: test-ca-central-1 # environment: ca-central-1 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=ca-central-1

The execution of the integration tests is commented out for now. The system tests will only pass when the entire application is deployed. Uncomment the integration test steps in your repository, and do another push to run the deployment pipeline after all components of ImageLabeller are deployed.

Understanding a .gitlab-ci.yml file

This step executes unit tests that are a part of the SubmitImage repository.

1 2 3 4 5 6 7 8 9 unit-test-us-west-1: stage: unit-test-us-west-1 environment: us-west-1 image: golang:buster rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" script: - cd submitImage - go test ./opendevopslambda/...

This step deploys the SubmitImage AWS Lambda using AWS SAM. Notice the before_script section. This step runs before the script section, and can be used to install dependencies, and setup various tools.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 deploy-us-west-1: stage: deploy-us-west-1 environment: us-west-1 image: python:latest rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz - export PATH=$PATH:/usr/local/go/bin - go version script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --region us-west-1 - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-1 --no-fail-on-empty-changeset

This step downloads, and runs the integration tests in the SystemTests repository.

1 2 3 4 5 6 7 8 9 10 11 12 test-us-west-1: stage: test-us-west-1 environment: us-west-1 image: golang:buster rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git - cd systemtests - go test -v ./... -aws_region=us-west-1

The deploy token created earlier is referenced in the git clone line.

1 - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git

Pushing to a feature branch

Run the following from the command line to push your changes to the IM-8 branch of your SubmitImage repository. Include the Jira issue ID in commit messages, and branch names to enable the Jira GitLab integration to keep track of what is happening in your project.

1 2 3 git add --all git commit -m "IM-8 add .gitlab-ci.yml to SubmitImage" git push -u origin IM-8

Click CI/CD, then Pipelines to see the pipeline run.

screenshot of pipeline run in gitlab

Create a merge request

Create a merge request to deploy to your production environments after GitLab deploys to your test environments. Choose the IM-8 branch.

screenshot of merge requests in gitlab

Click CI/CD, then Pipelines to see the running merge request pipeline.

screenshot of running merge request oin gitlab

Merge the changes to mainline after the merge request pipeline completes. Click CI/CD, then Pipelines to see the running Production pipeline.

screenshot of running production pipeline in gitlab

Create a repository for InvokeLabeller AWS Lambda

Go to Jira, and create a new issue for adding a InvokeLabeller AWS Lambda repository to GitLab. Make note of the issue ID. IM-10 in this example.

screenshot of jira issue create repo "invokelabeller" in gitlab

Go to GitLab and click New project, then Create blank project. Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

screenshot of creating new project "invokelabeller" in gitlab

In your terminal go to your InvokeLabeller repository, and run the following to push your code to GitLab.

1 2 3 4 5 git add --all git commit -m "IM-10 add InvokeLabeller to gitlab" git remote add origin git@gitlab.com:pmmquickstartguides/invokelabeller.git git branch -m mainline git push -u origin mainline

You must add AWS access keys, configure protected branches, and setup deployment environments.

Then add the deploy keys from your SystemTests repository to enable the InvokeLabeller GitLab pipeline to download, and run the SystemTests.

Finally, add your AWS account ID as a CI/CD Variable.

screenshot of variables page in gitlab

.gitlab-ci.yml for deploying to AWS

Go to your InvokeLabeller repository in your terminal, and create a branch named after your Jira issue ID.

1 git checkout -b IM-10

Create a .gitlab-ci.yml file with the following yaml. This defines a deployment workflow for your Test, Staging, and Production environments.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 stages: - merge-request - run-unit-tests #US-WEST-1 - deploy-us-west-1 - test-us-west-1 #US-EAST-2 - deploy-us-east-2 - test-us-east-2 #US-WEST-2 - deploy-us-west-2 - test-us-west-2 #US-EAST-1 - deploy-us-east-1 - test-us-east-1 #CA-CENTRAL-1 - deploy-ca-central-1 - test-ca-central-1 merge-request-pipeline-job: stage: merge-request rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" script: - echo "This pipeline always succeeds and enables merge" - echo true run-unit-tests: stage: run-unit-tests image: python:3.8-buster rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install pytest - pip3 install moto - pip3 install -r tst/requirements.txt --user script: - python3 -m pytest -v tst/unit --junitxml=test-reports/report.xml #US-WEST-1 deploy-us-west-1: stage: deploy-us-west-1 environment: us-west-1 image: python:3.8-buster rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --region us-west-1 - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-1 --no-fail-on-empty-changeset #test-us-west-1: # stage: test-us-west-1 # environment: us-west-1 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-west-1 #US-EAST-2 deploy-us-east-2: stage: deploy-us-east-2 environment: us-east-2 image: python:3.8-buster rules: - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --region us-east-2 - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-2 --no-fail-on-empty-changeset #test-us-east-2: # stage: test-us-east-2 # environment: us-east-2 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-east-2 #US-WEST-2 deploy-us-west-2: stage: deploy-us-west-2 environment: us-west-2 image: python:3.8-buster rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --region us-west-2 - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-2 --no-fail-on-empty-changeset #test-us-west-2: # stage: test-us-west-2 # environment: us-west-2 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-west-2 #US-EAST-1 deploy-us-east-1: stage: deploy-us-east-1 environment: us-east-1 image: python:3.8-buster rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --region us-east-1 - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-1 --no-fail-on-empty-changeset #test-us-east-1: # stage: test-us-east-1 # environment: us-east-1 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=us-east-1 #CA-CENTRAL-1 deploy-central-1: stage: deploy-ca-central-1 environment: ca-central-1 image: python:3.8-buster rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" before_script: - pip3 install awscli --upgrade - pip3 install aws-sam-cli --upgrade script: - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY - sam build - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --region ca-central-1 - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region ca-central-1 --no-fail-on-empty-changeset #test-central-1: # stage: test-ca-central-1 # environment: ca-central-1 # image: golang:buster # rules: # - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event" # script: # - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID # - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY # - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git # - cd systemtests # - go test -v ./... -aws_region=ca-central-1

The execution of the integration tests is commented out for now. The system tests will only pass when the entire application is deployed. Uncomment the integration test steps in your repository, and do another push to run the deployment pipeline after all components of ImageLabeller are deployed.

Pushing to a feature branch

Run the following from the command line to push your changes to the IM-10 branch of your InvokeLabeller repository. Include the Jira issue ID in commit messages, and branch names to enable the Jira GitLab integration to keep track of what is happening in your project.

1 2 3 git add --all git commit -m "IM-10 add .gitlab-ci.yml to InvokeLabeller" git push -u origin IM-10

Click CI/CD, then Pipelines to see the pipeline run.

screenshot of pipeline running in gitlab

Create a merge request

Create a merge request to deploy to your production environments after GitLab deploys to your test environments. Choose the IM-10 branch.

screenshot of creating merge request in gitlab

Merge the changes to mainline after the merge request pipeline completes. Click CI/CD, then Pipelines to see the running Production pipeline.

screenshot of running production pipeline in gitlab

If you’ve made this far, congratulations! You just deployed ImageLabeller. The next step is to set up monitoring ImageLabeller with Opsgenie.

 

Last modified on Sep 24, 2021
Cached at 1:42 AM on Oct 16, 2021 |

Additional Help