This is the full 1-page documentation for Octopilot v1.10.0.
Changelog
New Features
- improve the display of repositories pr url at the end (#307)
Introduction
Octopilot is a CLI tool designed to help you automate your Gitops workflow, by automatically creating and merging GitHub Pull Requests to update specific content in Git repositories.
If you are doing Gitops with GitHub-hosted repositories, Octopilot is your swiss army knife to propagate changes in your infrastructure.
Octopilot was initially developed at Dailymotion, and is a core component of our Gitops workflow - you can read our blog post Introducing Octopilot: a CLI to automate the creation of GitHub pull requests in your gitops workflow.
It works by:
- cloning one or more repositories, defined either:
- statically
- dynamically, using environment variables or GitHub search queries
- running one or more updaters on each cloned repository, using either:
- the YAML updater, to quickly update YAML files
- the YQ updater, based on mikefarah’s yq, to manipulate YAML or JSON files as you want
- the Helm updater, to easily update the dependencies of an Helm chart
- The sops updater, to manipulate files encrypted with mozilla’s sops
- The regex updater, to update any kind of text file using a regular expression
- The exec updater, to execute any command you want
- commit/push the changes
- create Pull Requests and optionally merge them
If you want to see what you can do with Octopilot for real, here is a set of real-world use-cases that we have at Dailymotion:
- Promoting a new application release with a gitops workflow
- Promoting a new library release to a dynamic list of application repositories
- Updating certificates with a gitops workflow
- Updating Go dependencies
- Previsualizing changes done by octopilot, without pushing to the remote GitHub repository
Usage
You can download the binary or see the docker pull
commands from the release page on GitHub.
There are no dependencies - not even on git
.
There are no configuration files - we use CLI flags for everything. You can run octopilot -h
to see all the flags. Some of them can be used multiple times, such as the --repo
or --update
flags.
You will need a GitHub token or app to authenticate with GitHub. See the GitHub Auth section for more details.
Here is an example of using Octopilot to update multiple repositories, using multiple updaters:
$ octopilot \
--github-token "my-github-token" \
--repo "my-org/some-repo" \
--repo "my-org/another-repo(merge=true)" \
--repo "discover-from(env=PROMOTE_TO_REPOSITORIES)" \
--repo "discover-from(query=org:my-org topic:my-topic)" \
--repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path in-file-text)" \
--repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path fork:true)" \
--update "yaml(file=config.yaml,path='version')=file(path=VERSION)" \
--update "yq(file=helmfile.yaml,expression='(.releases[] | select(.chart == \"repo/my-chart\") | .version ) = strenv(VERSION)')" \
--update "sops(file=secrets.yaml,key=path.to.base64encodedCertificateKey)=$(kubectl -n cert-manager get secrets tls-myapp -o template='{{index .data \"tls.key\"}}')" \
--pr-title "Updating some files" \
...
Continuous Delivery Pipelines
Octopilot has been designed to be used in a Continuous Delivery pipeline: no dependencies, no configuration file, only 1 command to update multiple repositories…
You can use it with Jenkins, Jenkins X, Tekton, GitHub Actions, …
At Dailymotion, we’re using it through Jenkins X/Tekton pipelines, and also a few Jenkins pipelines.
Repositories
Octopilot operates on GitHub repositories. A single execution of Octopilot can update one or more repositories. There are 2 ways to define the repositories to update:
- using a static list
- using a dynamic list
Static list
It is the easiest: just specify the repositories on the CLI using the --repo
flag, such as:
$ octopilot \
--repo "my-github-org/my-first-repo" \
--repo "my-github-org/my-second-repo(merge=true)" \
--repo "my-github-org/my-third-repo(draft=true,merge=false,branch=dev)"
You can add as much repositories as you want, each with different configuration.
It supports the following parameters:
merge
(boolean): iftrue
, then the PR created on this repository will be automatically merged - see the Pull Requests section for more details. It overrides the value of the--pr-merge
flag for this specific repository.mergeauto
(boolean): iftrue
, then the PR will merged by Github’s auto-merge PR feature. See the Pull Requests section for more details. It overrides the value of the--pr-merge-auto
flag for this specific repository.mergeautowait
(boolean): iftrue
, then wait until the PR is actually merged. See the Pull Requests section for more details. It overrides the value of the--pr-merge-auto-wait
flag for this specific repository.draft
(boolean): iftrue
, then the PR will be created as a draft PR on GitHub. You will need to manually mark it as “ready for review” before being able to merge it. It overrides the value of the--pr-draft
flag for this specific repository.branch
(string): the name of the base branch to use when cloning the repository. Default to theHEAD
branch - which means the default branch configured in GitHub: usuallymain
ormaster
.
Dynamic list
Octopilot can also be used with a “dynamic” list of repositories: repositories that are unknown when you write the command arguments.
It can retrieve the list of repositories to update from:
- one or more environment variables
- one or more GitHub Repositories Search Query
- one or more GitHub Code Search Query
Using environment variables
At runtime, Octopilot will read the list of repositories defined in one or more environment variables, such as:
$ export PROMOTE_TO_REPOSITORIES="my-github-org/my-first-repo my-github-org/my-second-repo(draft=true,merge=false)"
$ export ANOTHER_SET_OF_REPOSITORIES="some-org/some-repo;another-org/another-repo(draft=false)"
$ octopilot \
--repo "discover-from(env=PROMOTE_TO_REPOSITORIES)" \
--repo "discover-from(env=ANOTHER_SET_OF_REPOSITORIES,sep=;,draft=true)"
It supports the following parameters:
env
(string): the name of the environment variable to use, to retrieve the list of repositories.sep
(string): the separator between each repository, default to" "
(space).merge
(boolean): iftrue
, then the PRs created on the repositories from this env var will be automatically merged - see the Pull Requests section for more details. It overrides the value of the--pr-merge
flag for the repositories defined in this env var.mergeauto
(boolean): iftrue
, then the PR will merged by Github’s auto-merge PR feature. See the Pull Requests section for more details. It overrides the value of the--pr-merge-auto
flag for these specific repository.mergeautowait
(boolean): iftrue
, then wait until the PR is actually merged. See the Pull Requests section for more details. It overrides the value of the--pr-merge-auto-wait
flag for these specific repository.draft
(boolean): iftrue
, then the PRs will be created as draft PRs on GitHub. You will need to manually mark them as “ready for review” before being able to merge them. It overrides the value of the--pr-draft
flag for the repositories defined in this env var.branch
(string): the name of the base branch to use when cloning the repositories. Default to theHEAD
branch - which means the default branch configured in GitHub: usuallymain
ormaster
.
Note that each repository listed in an environment variable supports all the parameters defined in the static definition of a repository.
Using GitHub Search Query
A more powerful feature is the ability to load a list of repositories from a GitHub Repositories Search Query or a GitHub Code Search Query, such as:
$ octopilot \
--repo "discover-from(query=org:my-github-org topic:some-topic)" \
--repo "discover-from(query=org:my-github-org in:readme some-specific-content-in-the-readme,draft=true)" \
--repo "discover-from(query=org:my-github-org language:java is:private mirror:false archived:false,merge=true)" \
--repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path in-file-text)" \
--repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path fork:true)"
At runtime, Octopilot will use the GitHub API to retrieve the list of repositories matching a given query. This is useful when you have a common library used/imported by many repositories, and you want to create a PR to update the version when there is a new release of your lib. Instead of hardcoding the list of “dependant” repositories in your library repository, you can use a GitHub Search Query to find all repositories with a specific topic, or specific content in the description of the repo, or specific content in the README.md file of the repo, and so on. So “dependant” repositories can easily opt-in to get automatic PRs just by adding a topic for example.
It supports the following parameters:
searchtype
(string): represents the type of github search to be performed. Onlycode
andrepositories
are availables. Default torepositories
.query
(string): Specifies search criteria for listing repositories, including filters for file contents, location, language, topic, and more.merge
(boolean): iftrue
, then the PRs created on the repositories from this query will be automatically merged - see the Pull Requests section for more details. It overrides the value of the--pr-merge
flag for the repositories retrieved from this query.mergeauto
(boolean): iftrue
, then the PR will merged by Github’s auto-merge PR feature. See the Pull Requests section for more details. It overrides the value of the--pr-merge-auto
flag for these specific repository.mergeautowait
(boolean): iftrue
, then wait until the PR is actually merged. See the Pull Requests section for more details. It overrides the value of the--pr-merge-auto-wait
flag for these specific repository.draft
(boolean): iftrue
, then the PRs will be created as draft PRs on GitHub. You will need to manually mark them as “ready for review” before being able to merge them. It overrides the value of the--pr-draft
flag for the repositories retrieved from this query.branch
(string): the name of the base branch to use when cloning the repositories. Default to theHEAD
branch - which means the default branch configured in GitHub: usuallymain
ormaster
.
See the “promoting a new library release” use-case for a real-life example of what you can do with this feature.
Commits
After running all the updaters, octopilot will commit the changes on each repository. You can control which files will be committed, and how the commit will be created, using the Octopilot’s CLI flags.
Git clone
Flags related to the git clone
operation:
--git-clone-dir
(string): optional path to a directory used to clone the git repositories. Default to a newly created directory in the system temporary directory (defined by theTMPDIR
environment variable, defaulting to/tmp
). Note that by default all files created inside this directory will be deleted at the end of the process, unless the--keep-files
flag is set.
Git index/stage
By default, all files changed by the updaters will be added to the git “index” - so that they can be added to the git commit. This is configurable through the following flags:
--git-stage-all-changed
(boolean): if set totrue
(the default), then all changed files will be “staged” - or added to the git index. Set it tofalse
to control which files should be staged.--git-stage-pattern
(array of string): list of path patterns that will be “staged” - or added to the git index.
For example, to make sure you will only commit the changes to the helmfile.yaml
file:
$ octopilot \
--git-stage-all-changed=false \
--git-stage-pattern=helmfile.yaml \
...
Git commit
Although there are good default values, you can configure how Octopilot will create the git commit:
git-author-name
(string): the name of the author of the git commit. Default to the value of theGIT_AUTHOR_NAME
environment variable, or if unset, theuser.name
git config value.git-author-email
(string): the email of the author of the git commit. Default to the value of theGIT_AUTHOR_EMAIL
environment variable, or if unset, theuser.email
git config value.git-committer-name
(string): the name of the committer. Default to the value of theGIT_COMMITTER_NAME
environment variable, or if unset, theuser.name
git config value.git-committer-email
(string): the email of the committer. Default to the value of theGIT_COMMITTER_EMAIL
environment variable, or if unset, theuser.email
git config value.git-commit-title
(string): the title of the commit. Note that you can use the templating feature here.git-commit-body
(string): the body of the commit. Note that you can use the templating feature here.git-commit-footer
(string): the footer of the commit. Default to Octopilot’s signature, to indicate that this commit was generated by a tool.
The commit message is composed of the commit title, body and optional footer:
${git-commit-title}
${git-commit-body}
--
${git-commit-footer}
For example, to update the version of an application named my-app
, and write a nice commit message:
$ octopilot \
--repo "my-org/my-gitops-repo" \
--update "yaml(file=my-app/config.yaml,path='version')=${VERSION}" \
--git-commit-title 'chore(deps): update my-app to {{ env "VERSION" }}' \
--git-commit-body '{{ githubRelease (print "my-org/my-app-repo/" (env "VERSION")) | md2txt }}' \
...
It’s using the templating feature to retrieve the GitHub Release for your application’s version, and convert it to raw text. So in your commit message, you’ll see what changed in the new version.
Git push
git-branch-prefix
(string): when pushing the changes to the “origin” git repository, a new branch with a random name will be created. You can control the prefix of this random name, which default tooctopilot-
.
Note that Octopilot requires permissions to push on the GitHub repositories to update. For the moment, it doesn’t support forking the repository, and creating the Pull Request from the fork.
Git signing
git-signing-key-path
(string): when provided, Octopilot will use the GPG private key file to sign commits or tags, which will allow GitHub to verify the committer’s identity. See Add a GPG key.git-signing-key-passphrase
(string): Passphrase to decrypt the GPG private key used to sign commits. Leave empty if the private key is stored unencrypted.
Pull Requests
After running all the updaters and creating a git commit, octopilot will create a Pull Request for each repository.
Strategies
Octopilot has 3 strategies for creating a Pull Request:
- reset (the default): reset any existing Pull Request from the base branch
- append: append new commits to any existing Pull Request
- recreate: always create a new Pull Request
You can control which strategy to use using the --strategy
CLI flag.
Reset Strategy
This is the default strategy. With this strategy, Octopilot will reset any existing Pull Request from the base branch.
In detail, it will:
- clone the git repository
- find a “matching” Pull Request - based on the pre-configured labels. If there is a matching Pull Request, it will use the PR’s branch. Otherwise, it will just create a new branch
- reset the branch to the base branch - usually
main
ormaster
- run the updaters
- commit the changes and (force) push the commit
- update the existing Pull Request title/body/labels/comments/assignees or create a new one
Note that you can control how the existing Pull Request will be updated. For both the title and body, you can either:
- ignore the new changes. The existing PR won’t be changed - except for the labels, comments, assignees, and of course the commit.
- replace the title and/or body with the new ones. This is the default for the reset strategy.
- prepend the title and/or body with the new ones. This is mostly useful for the body.
- append the title and/or body with the new ones. This is mostly useful for the body.
Append Strategy
With this strategy, Octopilot will append new commits to any existing Pull Request.
In detail, it will:
- clone the git repository
- find a “matching” Pull Request - based on the pre-configured labels. If there is a matching Pull Request, it will switch to the PR’s branch. Otherwise it will just create a new branch from the base branch, and switch to it.
- run the updaters
- commit the changes and push the commit
- update the existing Pull Request title/body/labels/comments/asignees or create a new one
Note that you can control how the existing Pull Request will be updated. For both the title and body, you can either:
- ignore the new changes. The existing PR won’t be changed - except for the labels, comments, assignees, and of course the commit. This is the default for the append strategy.
- replace the title and/or body with the new ones.
- prepend the title and/or body with the new ones. This is mostly useful for the body.
- append the title and/or body with the new ones. This is mostly useful for the body.
Recreate Strategy
With this strategy, Octopilot will always create a new Pull Request.
In detail, it will:
- clone the git repository
- create a new branch from the base branch, and switch to it
- run the updaters
- commit the changes and push the commit
- create a new Pull Request
Creating / updating Pull Requests
You can control how the Pull Requests will be created or updated using the following CLI flags:
--strategy
(string): strategy to use when creating/updating the Pull Requests: eitherreset
(reset any existing PR from the current base branch),append
(append new commit to any existing PR) orrecreate
(always create a new PR). Default toreset
.--dry-run
(bool): if enabled, won’t perform any operation on the remote git repository or on GitHub: all operations will be done in the local cloned repository. So no Pull Request will be created/updated. Default tofalse
.--pr-title
(string): the title of the Pull Request. Default to the commit title. Note that you can use the templating feature here.--pr-title-update-operation
(string): the type of operation when updating a Pull Request’s title: eitherignore
(keep old value),replace
,prepend
orappend
. Default is:ignore
for “append” strategy,replace
for “reset” strategy, and not applicable for “recreate” strategy.--pr-body
(string): the body of the Pull Request. Default to the commit body and the commit footer. Note that you can use the templating feature here.--pr-body-update-operation
(string): the type of operation when updating a Pull Request’s body: eitherignore
(keep old value),replace
,prepend
orappend
. Default is:ignore
for “append” strategy,replace
for “reset” strategy, and not applicable for “recreate” strategy.--pr-comment
(array of string): optional list of comments to add to the Pull Request.--pr-assignees
(array of string): optional list of assignees (Github usernames) to add to the Pull Request.--pr-labels
(array of string): optional list of labels to set on the pull requests, and used to find existing pull requests to update. Default to["octopilot-update"]
.--pr-base-branch
(string): name of the branch used as a base when creating pull requests. Default tomaster
.--pr-draft
(bool): if enabled, the Pull Request will be created as a draft - instead of regular ones. It means that the PRs can’t be merged until marked as “ready for review”. Default tofalse
.
Merging Pull Requests
Optionally, Octopilot can also automatically merge the Pull Requests it creates. Before merging a Pull Request, Octopilot will wait for the PR to be in a “mergable” state, and for all required status checks to pass.
--pr-merge
(bool): if enabled, the Pull Requests will be automatically merged. It will wait until the PRs are “mergeable” before merging them. Default tofalse
.
All the following flags only apply if --pr-merge
is enabled.
--pr-merge-auto
(bool): merge PRs using Github’s auto-merge PR feature. By default, this will not wait until the PR is merged. Note, this must also be enabled at the repository level for it to work. This is a one-time task andocotopilot
will not do this for you. One way of doing this in bulk is using thegh
CLI:gh search repos --json url --jq '.[].url' ... | xargs -L1 gh repo edit --enable-auto-merge
.--pr-merge-auto-wait
(bool): when--pr-merge-auto
is enabled, wait for the PR to be merged by Github.--pr-merge-method
(string): the merge method to use. Eithermerge
,squash
, orrebase
. Default tomerge
.--pr-merge-commit-title
(string): optional title of the merge commit.--pr-merge-commit-message
(string): optional body of the merge commit.--pr-merge-sha
(string): optional SHA that pull request head must match to allow merge.--pr-merge-poll-timeout
(string/duration): maximum duration to wait for a Pull Request to be mergeable/merged, using the Golang syntax. Default to10m
(10 minutes).--pr-merge-poll-interval
(string/duration): duration to wait for between each GitHub API call to check if a PR is mergeable/merged, using the Golang syntax. Default to30s
(30 seconds).--pr-merge-retry-count
(int): number of times to retry the merge operation in case of merge failure. Default to3
.--pr-merge-branch-protection
(string): One ofstatusChecks
,all
,bypass
. Wait for the specified kind of branch protection rules to be satisfied before attempting to merge.statusChecks
waits only for status checks to be passing. This is the default.all
waits for every rule (approvals, commit signature, etc).bypass
will bypass branch protection rules when possible (i.e. the authenticated user/app have permissions to do so).
Updaters
The core feature of Octopilot is to update git repositories, and to do it you can use one or more of the available “updaters”:
- the YAML updater, to quickly update YAML files
- the YQ updater, based on mikefarah’s yq, to manipulate YAML or JSON files as you want
- the Helm updater, to easily update the dependencies of an Helm chart
- The sops updater, to manipulate files encrypted with mozilla’s sops
- The regex updater, to update any kind of text file using a regular expression
- The exec updater, to execute any command you want
Each updater can be used once or more, such as:
$ octopilot \
--update "yaml(file=config.yaml,path='version')=file(path=VERSION)" \
--update "regex(file=some-file.txt,pattern='version: \"(.*)\"')=${VERSION}" \
--update "yaml(file=another-config.yaml,path='path.to.version')=$(cat VERSION)" \
...
YAML
The YAML updater is great when you want to quickly set a value for a specific path in one or more files. Such as if you want to update a version used in a YAML file:
$ octopilot \
--update "yaml(file=config.yaml,path='app.version')=file(path=VERSION)" \
...
Given the following config.yaml
file:
app:
name: foo
version: 1.0.0
Octopilot will set the value of the app.version
key to the content of the VERSION
file.
The syntax is: yaml(params)=value
- you can read more about the value in the “value” section.
It supports the following parameters:
file
(string): mandatory path to the file to update. Can be a file pattern - such asconfig/*.yaml
to match files in the same directory, orconfig/**/*.yaml
using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.path
(string): mandatory path to the key to update in the YAML file(s). We support yq v3 path expressions or yq v4 syntax.indent
(int): optional number of spaces used for indentation when writing the YAML file(s) after update. Default to2
.trim
(boolean): iftrue
, the content will be “trimmed” before being written to disk - to avoid extra line break at the end of the file for example.create
(boolean): iftrue
, then thepath
will always be set to the given value, even if no such key existed before. The default behaviour (false
) is to NOT create any new path/key.style
(string): an optional style to apply to the new value:double
(add double quotes),single
(add single quotes),literal
,folded
orflow
- see yq style reference.
Note that Octopilot will keep the comments in the YAML files - because we’re using the great go-yaml v3 lib. Just that it might rewrite a bit your indentation.
See the “updating certificates” use-case for a real-life example of what you can do with this updater.
YQ
The YQ updater is based on the excellent yq application - and Go lib. It is much more powerful than the basic YAML updater - because it supports all the yq operators, and because you are not limited to setting a value for a specific key. You can do very powerful things, such as manipulating YAML comments, use variables, output to json (note that it can also read JSON input), …
The syntax is: yq(params)
, such as:
$ octopilot \
--update "yq(file=config.yaml,expression='.path.to.version = strenv(VERSION)')" \
...
It supports the following parameters:
file
(string): mandatory path to the file to update. Can be a file pattern - such asconfig/*.yaml
to match files in the same directory, orconfig/**/*.yaml
using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.expression
(string): mandatory yq v4 expression that will be evaluated against each file.output
(string): optional output of the result. By default the result is written to the source file - in-place editing. But you can send the result tostdout
,stderr
or a specific file.json
(boolean): iftrue
, then the output will be written in JSON format instead of YAML format.indent
(int): optional number of spaces used for indentation when writing the YAML file(s) after update. See yq doc on indent. Default to2
.trim
(boolean): iftrue
, the content will be “trimmed” before being written to disk - to avoid extra line break at the end of the file for example.unwrapscalar
(boolean): iftrue
(the default), only the value will be printed - not the comments. See yq doc on unwrap scalars.
Note that Octopilot will keep the comments in the YAML files - because we’re using the great go-yaml v3 lib. Just that it might rewrite a bit your indentation.
See the “promoting a new application release” use-case for a real-life example of what you can do with this updater.
Helm
The Helm updater is made to easily update the dependencies of one or more Helm charts. It supports both:
- Helm v3, with the dependencies declared in the
Chart.yaml
file - Helm v2, with the dependencies declared in the
requirements.yaml
file
If you run the following command:
$ octopilot \
--update "helm(dependency=chart-name)=1.2.3" \
...
Octopilot will discover all charts stored in the cloned repository, and for each, try to change the version of the chart-name
dependency to 1.2.3
.
The syntax is: helm(params)=value
- you can read more about the value in the “value” section.
It supports the following parameters:
dependency
(string): mandatory name of the dependency to update. Must exist in the dependencies list - it won’t be added.indent
(int): optional number of spaces used for indentation when writing the YAML file(s) after update. Default to2
.
Note that Octopilot will keep the comments in the YAML files - because we’re using the great go-yaml v3 lib. Just that it might rewrite a bit your indentation.
See the “promoting a new library release” use-case for a real-life example of what you can do with this updater.
Sops
The sops updater can manipulate files encrypted with mozilla’s sops natively, without the need to install sops. This is great if you want to store sensitive data in your git repositories: you can encrypt them with sops
, and use Octopilot to update them automatically.
For example if you want to store your TLS certificate as a base64 encoded string in a sops-encrypted file:
$ octopilot \
--update `sops(file=secrets.yaml,key=app.tls.base64encodedCertificateKey)=$(kubectl -n cert-manager get secrets tls-myapp -o template='{{index .data "tls.key"}}')` \
...
Given the following (decrypted) secrets.yaml
file:
app:
tls:
base64encodedCertificateKey: LS0tLS1CRUdJTiBSU0EgU...
Octopilot will decrypt the secrets.yaml
file, set the value of the app.tls.base64encodedCertificateKey
key to the given value, and re-encrypt the secrets.yaml
file before writing it to disk.
The syntax is: sops(params)=value
- you can read more about the value in the “value” section.
It supports the following parameters:
file
(string): mandatory path to the file to update. Can be a file pattern - such asconfig/secrets.*
to match files in the same directory, orconfig/**/secrets.*
using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.key
(string): mandatory key to update in the file(s).
Note that depending on the sops backend you use (KMS, age, vault, …) you might need to set some environment variables, such as:
- for GCP KMS, the
GOOGLE_APPLICATION_CREDENTIALS
env var - for age, the
SOPS_AGE_KEY_FILE
env var - …
See the “updating certificates” use-case for a real-life example of what you can do with this updater.
Regex
The regex updater can be used to update any kind of text file using a regular expression - with the Golang syntax, such as:
$ octopilot \
--update "regex(file=some-file.txt,pattern='version: \"(.*)\"')=${VERSION}" \
...
Given the following some-file.txt
file:
version: "1.0.0"
...
Octopilot will replace the first line with version: "1.2.3"
if the $VERSION
env var is set to 1.2.3
for example.
The syntax is: regex(params)=value
- you can read more about the value in the “value” section.
It supports the following parameters:
file
(string): mandatory path to the file to update. Can be a file pattern - such asfiles/*.txt
to match files in the same directory, orfiles/**/*.txt
using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.pattern
(string): mandatory regex pattern to find and replace something in the file(s). The pattern must be in the Golang syntax. If this pattern includes a capturing group, then it will be replaced by the provided value.
A few things you can do with the regex updater:
- replace the whole content of a file
$ octopilot \ --update "regex(file=my-file,pattern='(?ms)(.*)')=new content"
Exec
The exec updater can execute any command you want, so you can change files in the cloned git repository with any tool you have available.
For example to update all your Go dependencies to the latest version:
$ octopilot \
--update "exec(cmd=go,args=get -d -t -u)" \
--update "exec(cmd=go,args=mod tidy)" \
--update "exec(cmd=go,args=mod vendor)" \
--git-stage-pattern "vendor" \
...
This will execute 3 commands, that will update the go.mod
& go.sum
files, and the vendor
dir. Octopilot will then add/commit all the changes, including the new files in the vendor
directory.
The syntax is: exec(params)
.
It supports the following parameters:
cmd
(string): mandatory command to execute.path
(string): optional path to execute the command in.args
(string): optional arguments for the command. The arguments are space-separated. If you have a space in an argument, you can quote it, such as:-c 'some arg' -x another
.stdout
(string): optional path to a file where the std output of the command will be written. If it’s a relative path, it will be relative to the root of the cloned git repository.stderr
(string): optional path to a file where the std error output of the command will be written. If it’s a relative path, it will be relative to the root of the cloned git repository.timeout
(string/duration): optional maximum duration to wait for the command to finish, using the Golang syntax.
A few things you can do with the regex updater:
use a bash command to enable shell expansion (disabled by default when invoking commands through Octopilot)
$ octopilot \ --update "exec(cmd=sh,args=-c 'cat files/*.txt',stdout=output.txt)"
use the an external tool (here kustomize) and change the current working directory before executing the command
$ octopilot \ --update "exec(cmd=kustomize, path=k8s/overlays/dev, args=edit set image containername=registry.tld/repo/image:newtag)"
See the “updating go dependencies” use-case for a real-life example of what you can do with this updater.
Value
Some updaters accept a value in their syntax, such as updater(params)=value
.
This value can be either:
- a raw value
- the content of a file
Raw value
This is the easiest way to set a value: just use a raw value, such as:
$ octopilot \
--update "yaml(file=config.yaml,path='version')=v1.2.3" \
...
Note that you can also use an environment variable:
$ export VERSION=v1.2.3
$ octopilot \
--update "yaml(file=config.yaml,path='version')=${VERSION}" \
...
or any command you want:
$ echo v1.2.3 > /tmp/VERSION
$ octopilot \
--update "yaml(file=config.yaml,path='version')=$(cat /tmp/VERSION)" \
...
File content
If you want to use the content of a file, you can use the file valuer:
$ octopilot \
--update "yaml(file=config.yaml,path='version')=file(path=VERSION)" \
...
It will read the VERSION
file located at the root of the cloned git repository, and use its content as the value.
The syntax is: file(params)
.
It supports the following parameters:
path
(string): mandatory path to the file to read. If it’s a relative path, it will be relative to the root of the cloned git repository.
GitHub Auth
Octopilot needs a way to authenticate against the GitHub API, to:
- clone the repositories
- push new changes
- and create/update/merge Pull Requests
It supports 2 ways to authenticate:
- using a personal access token, which is the default
- using a GitHub app
You can define which method to use, using the --github-auth-method
CLI flag.
Personal Access Token
By default, the --github-auth-token
flag is set to token
, so Octopilot will use a Personal Access Token - or PAT
. This token can be defined either by the GITHUB_TOKEN
environment variable, or by setting the --github-token
CLI flag.
You can read GitHub’s documentation on creating a personal access token. You’ll need at least the repo
permissions.
GitHub App
An alternative to the “simple” token is to use a GitHub App.
First, you’ll need to set the --github-auth-token
flag value to app
, and then configure the following settings:
--github-app-id
(int): the GitHub App ID. Default to the value of theGITHUB_APP_ID
environment variable.--github-installation-id
(int): the GitHub App installation ID. Default to the value of theGITHUB_INSTALLATION_ID
environment variable.--github-privatekey
(string): the app’s private key - used to sign access token requests - in PEM format. Default to the value of theGITHUB_PRIVATEKEY
environment variable. You can either set this, or the--github-privatekey-path
.--github-privatekey-path
(string): the path to the app’s private key - used to sign access token requests - in PEM format. Default to the value of theGITHUB_PRIVATEKEY_PATH
environment variable. Will be used if the--github-privatekey
flag is not set.
See GitHub’s documentation for more details on:
Using octopilot with Enterprise GitHub
By default, octopilot will operate on repositories hosted on https://github.com. Octopilot can work on repositories hosted on an Enterprise Github servers by adding the --github-url
flag.
This flag must be set to the URL of the Enterprise GitHub server (the URL used to browse to the GitHub server main page). Authentication is usually required using any of the authentication method described above.
Example of use:
$ octopilot \
--github-url https://mygitserver.acme.com \
--repo "${ORG_NAME}/test-repo" \
...
Advanced
Octopilot advanced features:
Templating
For some of the CLI flags - such as commit title/body or Pull Requests title/body - you can use our “templating” feature. This will allow you to write nice commits and Pull Requests.
Octopilot uses the Go template syntax, and supports the following functions:
- all the Go template functions
- all the sprig functions
- Octopilot’s own custom functions
Octopilot’s own custom functions
Octopilot comes with the following custom functions:
readFile
to read a file from the cloned git repository, and return its contentgithubRelease
to retrieve the release notes for a specific GitHub releaseexpandGithubLinks
to transform GitHub short links - such as #123 - to absolute URLsextractMarkdownURLs
to transform markdown links to plain URLsmd2txt
to strip all the markdown syntax from a string
readFile
The readFile
function will read a file from the cloned git repository, and return its content as a string. If the path of the file is relative, it will be relative to the root of the git repository.
Definition: readFile(filePath string) string
.
Example: {{readFile "README.md"}}
to print the content of the README.md
file.
githubRelease
The githubRelease
function will retrieve the release notes for a specific GitHub release. The release is identified by the owner, repository and release version.
Definition: githubRelease(releaseID string) string
.
Example: {{ githubRelease "owner/repo/v1.2.3" }}
to print the release notes for the release v1.2.3
of the owner/repo
github repository.
expandGithubLinks
The expandGithubLinks
function will transform GitHub short links - such as #123 - to absolute URLs. It is most useful when combined with the githubRelease
function, to ensure that the links in the release notes are always absolute.
Definition: expandGithubLinks(fullRepositoryName string, markdownInput string) string
. The fullRepositoryName
parameter is the full name of the repository, such as owner/repo
. It is used to build the absolute URLs.
Example: {{ githubRelease (print "owner/repo/" (env "VERSION")) | expandGithubLinks "owner/repo" }}
.
extractMarkdownURLs
The extractMarkdownURLs
function will transform markdown links to plain URLs. It is most useful when combined with the githubRelease
function, to ensure that the links in the release notes are always plain URLs - when you want to print plain text, and not markdown, such as in a commit message.
Definition: extractMarkdownURLs(markdownInput string) string
.
Example: {{ githubRelease (print "owner/repo/" (env "VERSION")) | expandGithubLinks "owner/repo" | extractMarkdownURLs }}
.
md2txt
The md2txt
function will strip all the markdown syntax from a string. It is most useful when combined with the githubRelease
function, to ensure that the release notes are always plain text, and not markdown. Use it for your commit messages.
Definition: md2txt(markdownInput string) string
.
Example: {{ githubRelease (print "owner/repo/" (env "VERSION")) | expandGithubLinks "owner/repo" | extractMarkdownURLs | md2txt }}
.
Use cases
If you want to see what you can do with Octopilot for real, here is a set of real-world use-cases that we have at Dailymotion:
- Promoting a new application release with a gitops workflow
- Promoting a new library release to a dynamic list of application repositories
- Updating certificates with a gitops workflow
- Updating Go dependencies
- Previsualizing changes done by octopilot, without pushing to the remote GitHub repository
Promoting a new application release
If you’re doing gitops, you’ll most likely have:
- at least one “application” git repository, for your(s) application(s).
- at least one “environment” git repository, for your(s) environment(s).
When you release a new version of your application, you’ll need to “promote” it to (at least) one of your environments. In the gitops world, we’re doing that by creating a new Pull Request against the target environment git repository, to update the version of the application in a configuration file.
And this is where Octopilot shines. You can use it in your application’s Continuous Delivery pipeline to create the Pull Request against your environment’s git repository.
Let’s see an example, where we want to:
- automatically deploy new releases in the staging environment
- manually deploy new releases in the production environment
To do that, we’ll:
- create a Pull Request on the
staging-env
git repository, and ask Octopilot to automatically merge it - and thus deploying it - create a Pull Request on the
prod-env
git repository, but not merge it. We’ll even create the Pull Request as a “draft”, making it explicit that the release is not ready yet.
Oh, and we’d like to get nice commit messages and Pull Request title/description, so that people can understand what really is changing with this Pull Request. If all you have is “Update my-app to version 1.2.3”, and a diff that shows a version change in a config file, it won’t help you understand if this release is just fixing a typo in a README, fixing a critical bug, or introducing a new feature. We’re already generating release notes with a changelog - if not, you should, using git-chglog for example - so let’s re-use it and include it in our commit messages and Pull Request.
Here is an example of an Octopilot invocation you can use to achieve our goal - we’ll go over it in details later:
$ export GITHUB_TOKEN=<your_github_token>
$ export ORG_NAME=my-org
$ export APP_NAME=my-app
$ export VERSION=1.2.3
$ octopilot \
--repo "${ORG_NAME}/staging-env(merge=true)" \
--repo "${ORG_NAME}/prod-env(draft=true)" \
--update "yq(file=helmfile.yaml,expression='.releases[] | select(.name == strenv(APP_NAME)) | .version',output=.git/previous-version.txt)" \
--update "yq(file=helmfile.yaml,expression='(.releases[] | select(.name == strenv(APP_NAME)) | .version) = strenv(VERSION)')" \
--git-commit-title 'chore(deps): update {{ env "APP_NAME" }} from {{ readFile ".git/previous-version.txt" | trim }} to {{ env "VERSION" }}' \
--git-commit-body '{{ githubRelease (printf "%s/%s/v%s" (env "ORG_NAME") (env "APP_NAME") (env "VERSION")) | expandGithubLinks (printf "%s/%s" (env "ORG_NAME") (env "APP_NAME")) | extractMarkdownURLs | md2txt }}' \
--git-branch-prefix "octopilot-update-${APP_NAME}-" \
--pr-labels "update-${APP_NAME}" \
--pr-title "Update ${APP_NAME} to ${VERSION}" \
--pr-title-update-operation "replace" \
--pr-body '{{ githubRelease (printf "%s/%s/v%s" (env "ORG_NAME") (env "APP_NAME") (env "VERSION")) | expandGithubLinks (printf "%s/%s" (env "ORG_NAME") (env "APP_NAME")) }}' \
--pr-body-update-operation "prepend" \
--strategy "append"
As you can see, we’re making the same set of changes to 2 different repositories, with different configurations:
- the PR on the
staging-env
repository will be automatically merged - and thus our release deployed - the PR on the
prod-env
repository will be created as “draft”, and won’t be automatically merged - thus requiring a human intervention to merge it
We’re running the YQ updater twice, on the same helmfile.yaml
file, which would look like the following:
releases:
- name: my-app
version: 1.0.0
- name: another-app
version: 1.5.0
This is a very simplified configuration file for helmfile, an application used to describe Helm releases. You can use whatever you want, it’s just to base our example on a real-life use-case.
So we’re running the YQ updater twice:
- the first time, with the
.releases[] | select(.name == strenv(APP_NAME)) | .version
expression, to extract the current version for our application namedmy-app
. And we’re sending the output to the.git/previous-version.txt
file - which will contain just1.0.0
. We’re doing that to store the “previous” version before changing it without new version. You’ll notice that the output file is located in the.git
directory, this is to avoid committing it by default. - the second time, we’re replacing the version with the new one - stored in the
VERSION
environment variable - with the following expression:(.releases[] | select(.name == strenv(APP_NAME)) | .version) = strenv(VERSION)
.
Now, we have:
- a locally modified
helmfile.yaml
- a new
.git/previous-version.txt
file
So it’s time to commit!
- in the commit title, we want to include both the previous version and the new version. We’ll use the templating feature to include both the content of a file and an environment variable:
chore(deps): update {{ env "APP_NAME" }} from {{ readFile ".git/previous-version.txt" | trim }} to {{ env "VERSION" }}
- in the commit body, we want to include the release notes for the new version. We’ll use the templating feature to retrieve the release notes, convert all the GitHub short links to absolute URLs, and transform the markdown into raw text:
{{ githubRelease (printf "%s/%s/v%s" (env "ORG_NAME") (env "APP_NAME") (env "VERSION")) | expandGithubLinks (printf "%s/%s" (env "ORG_NAME") (env "APP_NAME")) | extractMarkdownURLs | md2txt }}
We’ll also use a custom prefix for the branch name, to make it easier to find which branch belongs to which app, if you have multiple promotion PRs for different applications opened at the same time.
Next, the Pull Request. We’ll use the append
strategy, which means that if we need to promote a release in prod, and the previous one hasn’t been merged/deployed yet, we’ll just append a new commit on the existing branch/PR. So that your production PR for your application will stay the same, accumulating releases until it is merged. We’re using a specific label for our Pull Request: update-${APP_NAME}
- to make sure we’ll find any existing PR for our application, and that each application will get its own PR.
Same as for the commit, we’ll use the templating feature to write a nice title and description for our Pull Request - just that this time we won’t need to convert from markdown to raw text. And we’ll use specific “update operations”:
- we’ll always replace the PR title, because we want a short title, with only the app name and the latest release’s version - using the
--pr-title-update-operation "replace"
flag - we’ll “prepend” the new PR body, before the existing one, using the
--pr-body-update-operation "prepend"
flag. So that the release notes for the latest release will be first - just as when you read a changelog. And of course we don’t want to remove the previous release notes.
Result
Staging environment
This is a screenshot of a Pull Request on the staging environment git repository, which has been automatically merged. You can see the release notes, and you’ll notice the “signature” at the bottom: we’re making it easy for people to know:
- that this PR has been generated by an application - and not created manually by a human
- which version of Octopilot has been used
- from where it has been executed. This will most likely be the application’s GitHub repository, because it is where you (should) define your application’s Continuous Delivery pipeline, which contains a step to execute Octopilot.
Production environment
This is a screenshot of a Pull Request on the production environment git repository:
- the PR has been created to promote
v3.14.1
of the application - later,
v3.15.0
has been released - and promoted. Thus adding a new commit to the PR - and then,
some-user
merged the Pull Request, to deploy in prod
You’ll notice that we have 2 release notes in the PR: 1 for each release. So you can see the full changelog for every release that will be deployed when you’ll merge this PR.
In this screenshot you can see the 2 commits:
Feedback
The benefit of adding the application’s release notes in the promotion pull request body, is that not only will you know exactly what you’ll deploy, but you’ll also get links between the application pull request and the promotion pull requests. So that if you go back to the application’s PR, you’ll see something like:
You can see at the bottom the links to our 2 promotion pull requests, with their statuses - both have been merged already in this case.
Going further
As you can see, it’s easy to adapt this example for your own use-case. For example, you might want:
- to create PRs on a different set of repositories: QA, staging and production
- to update different kinds of files, using different updaters
- to customize the commit or the Pull Request
Promoting a new library release
Promoting a new release of a library is similar to promoting a new release of an application, except that instead of promoting to a small number of git repositories - representing the different environments - you will promote to a much larger number of git repositories: one for each application using the library.
The main change is that instead of using a static list of repositories, you can use a dynamic list of repositories - based on a GitHub Repositories Search Query or a GitHub Code Search Query for example.
GitHub Repositories Search Query
Here is an example of an Octopilot invocation using a dynamic list of repos and GitHub Repositories Search Query - we’ll go over it in details later:
$ export GITHUB_TOKEN=<your_github_token>
$ export VERSION=1.0.15
$ octopilot \
--repo "discover-from(query=org:my-org topic:octopilot-my-base-chart)" \
--update "helm(dependency=my-base-chart)=${VERSION}" \
--git-commit-title 'chore(deps): update my-base-chart to {{ env "VERSION" }}' \
--git-commit-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" | extractMarkdownURLs | md2txt }}' \
--git-branch-prefix "octopilot-update-my-base-chart-" \
--pr-labels "update-my-base-chart" \
--pr-title "Update Helm Base Chart to ${VERSION}" \
--pr-title-update-operation "replace" \
--pr-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" }}' \
--pr-body-update-operation "prepend" \
--strategy "append"
We’re using the following GitHub Repositories search query to discover the repositories that are using the my-base-chart
Helm chart:
query=org:my-org topic:octopilot-my-base-chart
Which means that developers just need to add the octopilot-my-base-chart
topic to their repositories - in the GitHub UI - to enable automatic update of the Helm base chart. It’s easy to add, and easy to turn off if you don’t want it. Note that you are not limited to searching by topic - you can also search for specific content in the repository’s name, description, README.md
file, and so on.
We’re using the Helm updater to update all the Helm charts, and set the version of their dependency on my-base-chart
to the new release version.
Everything else is similar to the promotion of a new application release.
GitHub Code Search Query
This search type is useful in case you experience some limitations with the default Repositories
one
- 50 characters max
- 20 topics max per repository
Here is an example of an Octopilot invocation using a dynamic list of repos and GitHub Code Search Query - we’ll go over it in details later:
$ export GITHUB_TOKEN=<your_github_token>
$ export VERSION=1.0.15
$ octopilot \
--repo "discover-from(searchtype=code,query=org:my-org filename:Chart.yaml path:charts my-base-chart)" \
--update "yaml(file=charts/*/Chart.yaml,path=dependencies.(name==my-base-chart).version,style=folded)=${VERSION}" \
--git-commit-title 'chore(deps): update my-base-chart to {{ env "VERSION" }}' \
--git-commit-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" | extractMarkdownURLs | md2txt }}' \
--git-branch-prefix "octopilot-update-my-base-chart-" \
--pr-labels "update-my-base-chart" \
--pr-title "Update Helm Base Chart to ${VERSION}" \
--pr-title-update-operation "replace" \
--pr-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" }}' \
--pr-body-update-operation "prepend" \
--strategy "append"
We can use the following GitHub Code search type and query to discover the repositories that are using the my-base-chart
Helm chart:
searchtype=code,query=org:my-org filename:Chart.yaml path:charts my-base-chart
By default searchtype
is equal to repositories
, because here we need to use the Github Code Search query, we have to put it to code.
With Github Code Search query, developers don’t need to do anything to enable automatic update of their Helm base chart.
We’re using the Helm updater to update all the Helm charts, and set the version of their dependency on my-base-chart
to the new release version.
Everything else is similar to the promotion of a new application release.
Result
This is the screenshot of the bottom of the library PR which is at the origin of the promotion - and which is referenced in the library’s release notes.
You can see that multiple promotion pull requests have been created, one per repository matching the GitHub search query. This is an easy way to see which application repositories have upgraded to the new version of the library.
Updating certificates
One of the questions you will have to answer when doing gitops is: “where do I stop?”. What do you store in your environment git repository? For example, should you store your certificates there too, or do you consider them “outside” of the gitops-scope, and so managed by something else, such as cert-manager?
Let’s say we want to manage them with our gitops process. It means that we’ll need to:
- store them in a git repository. It’s not a problem for the certificate itself, but we’ll need to ensure that the private key won’t be stored in cleartext.
- update the git repository every time a new certificate is issued - ideally by automatically creating a Pull Request.
In a Kubernetes environment where you have multiple clusters - for example in different regions of the world, different environments, etc - we can setup and use cert-manager to manage the certificates from a “central” cluster. No need to setup cert-manager in all your clusters. And we’ll use Octopilot to “propagate” the certificates to all the clusters, through a gitops workflow, by storing and updating the certificates in one or more git repositories.
You can setup a nightly CronJob, a scheduled pipeline, or anything else you prefer to regularly call Octopilot, to make sure that all the certificates stored in the git repositories are up-to-date - or to create Pull Requests to update them.
In fact, we won’t call Octopilot directly, we’ll call a script that will perform a few operations before executing Octopilot:
- first, we’ll need to retrieve all the certificates from the Kubernetes API - using something like
kubectl -n cert-manager get certificates.cert-manager.io
- then, for each certificate, we’ll need to retrieve its associated Kubernetes Secret, which contains the actual certificate and its private key - both base64-encoded - using something like
kubectl -n cert-manager get secret $secretName -o go-template='{{index .data "tls.crt"}}' > tls.crt.base64
- optionally, we can extract some data from the certificates, such as the DNS names, validity dates, and so on - to generate nice commit messages and Pull requests. Use the
openssl
tool to extract thestartdate
andenddate
fields for example. - and then we can execute Octopilot with something like the following:
$ octopilot \
--repo "owner/prod-env" \
--update "yaml(file=${APP_NAME}-values.yaml,path=tls.certificate)=file(path=tls.crt.base64)" \
--update "sops(file=${APP_NAME}-secrets.yaml,key=tls.certificateKey)=file(path=tls.key.base64)" \
--git-commit-title "Update ${APP_NAME} certificate" \
--git-commit-body "..." \
--git-branch-prefix "octopilot-cert-${APP_NAME}-" \
--pr-labels "update-cert-${APP_NAME}"
Here, we are running 2 updaters:
- the YAML updater, to update the Helm values file of our application, and set the
tls.certificate
value to the content of atls.cert.base64
file - the SOPS updater, to update the Helm secrets file of our application - which is in fact a sops-encrypted YAML file - and set the
tls.certificateKey
value to the content of atls.key.base64
file
We’re using the default reset
strategy, which means that if we don’t merge the PR right away, and your CronJob runs a second time, it will find the existing PR and just reset it from the base branch, and then force push the commit. So you’ll always only see 1 commit, rebased every day, with the latest certificate. We’re using a specific label for our Pull Request: octopilot-cert-${APP_NAME}
- to make sure we’ll find any existing PR for our application/certificate, and that each application/certificate will get its own PR.
Result
This is a screenshot of a Pull Request on an environment git repository, which updates a specific certificate.
You can see that we extracted a few information, to make it easy for reviewers - because the diff is just a base64 blob replaced by another one:
- the creation date of the new certificate - in fact this is the creation timestamp of the latest cert-manager order, which means the latest time the certificate has been renewed
- the new validity dates, extracted using
openssl x509 -in tls.crt -noout -startdate
for example - the validity dates of the currently deployed certificate, retrieved using
openssl s_client -servername $domain -connect $domain:443 > previous.crt
for example, and then extracted usingopenssl
once again - the list of domains - or DNS names - for which this certificate is valid, extracted using
kubectl -n cert-manager get $cert -o go-template={{.spec.dnsNames}}
for example
Merging this Pull Request would result in a re-deployment of our application, with an updated certificate. So with this approach we can control when we want to deploy new certificates.
Updating Go dependencies
One of the downside of using a micro-services architecture is that it requires a lot of maintenance on the different git repositories. Even more when you have more repositories than teams or developers. For example, you’ll need to ensure that:
- the dependencies are up to date
- the “config files” for your Continuous Delivery pipeline(s) are up to date
- and so on…
Automating the creation of Pull Requests to keep your git repositories “in sync” is a good way to reduce the maintenance effort. Dependabot is one way to do it, Octopilot is another. The benefits of using Octopilot are:
- you control where it is executed - in your own infrastructure
- you control exactly what it does - including running your own custom scripts/binaries
One use-case we have is to update the Go dependencies, by running the following commands:
go get -d -t -u
go mod tidy
go mod vendor
go mod verify
For example you can do it on octopilot’s own repository, by running:
$ export GITHUB_TOKEN=<your_github_token>
$ octopilot \
--repo "dailymotion-oss/octopilot" \
--update "exec(cmd=go,args=get -d -t -u)" \
--update "exec(cmd=go,args=mod tidy)" \
--update "exec(cmd=go,args=mod vendor)" \
--update "exec(cmd=go,args=mod verify)" \
--git-stage-pattern "vendor" \
--git-commit-title "chore(deps): update Go dependencies"
You can then run it on a regular basis, such as every week, to update all your dependencies at once. And you can use the dynamic repositories feature to do it on all your repositories with a specific GitHub topic.
Result
This is a screenshot of a Pull Request on a git repository, which updates all the Go modules.
Using the exec updater we can run any command we want, capture its stdout and/or stderr, and use it in the commit message and/or Pull Request description.
Previsualizing changes
If you’re working on your workflow with Octopilot, at some point you might want to “preview” the changes Octopilot will do, without actually creating the Pull Request(s).
There is an easy way to do that. You’ll need to:
- use the
--dry-run
CLI flag, to ensure that no operation will be performed on the remote git repository: not pushing branches/commits, not creating/updating Pull Requests, … - use the
--keep-files
CLI flag, to ensure that the changes to the local cloned git repositories won’t be lost at the end of the process - because by default Octopilot removes all temporary files - use a verbose log level - such as
debug
ortrace
- with the--log-level
CLI flag, to retrieve the path of the temporary files created by Octopilot
For example, if you run:
$ export GITHUB_TOKEN=<your_github_token>
$ octopilot \
--repo "dailymotion-oss/octopilot" \
--update "yq(file=.goreleaser.yml,expression='(.dockers[] | select(.dockerfile == \"Dockerfile.goreleaser\") | .dockerfile) = \"a.new.Dockerfile\"')" \
--update "yaml(file=.golangci.yml,path=run.timeout)=42m" \
--update "regex(file=README.md,pattern='(?ms)(.*)')=replacing the content of the README.md file with this new content" \
--dry-run --keep-files --log-level=debug
Then you should see something like:
DEBU[0000] Updaters ready updaters="[YQ[file=.goreleaser.yml,expression=(.dockers[] | select(.dockerfile == \"Dockerfile.goreleaser\") | .dockerfile) = \"a.new.Dockerfile\",output=,indent=2] YAML[path=run.timeout,file=.golangci.yml,style=,create=false,trim=false,indent=2] Regex[pattern=(?ms)(.*),file=README.md]]"
DEBU[0000] Repositories ready repositories="[{dailymotion-oss octopilot map[]}]"
DEBU[0000] Using 'reset' strategy repository=dailymotion-oss/octopilot
DEBU[0003] Git repository cloned git-reference=HEAD git-url="https://github.com/dailymotion-oss/octopilot.git" local-path=/var/folders/v0/fx5l3skn17785d8f4l883m6w0000gp/T/octopilot092369223/dailymotion-oss/octopilot
DEBU[0003] No existing Pull Request found labels="[octopilot-update]" repository=dailymotion-oss/octopilot
DEBU[0004] Switched Git branch branch=octopilot-c3qif432dnc2961tuuh0 repository-name=octopilot
DEBU[0004] Updater finished changes=true repository=dailymotion-oss/octopilot updater="YQ[file=.goreleaser.yml,expression=(.dockers[] | select(.dockerfile == \"Dockerfile.goreleaser\") | .dockerfile) = \"a.new.Dockerfile\",output=,indent=2]"
DEBU[0004] Updater finished changes=true repository=dailymotion-oss/octopilot updater="YAML[path=run.timeout,file=.golangci.yml,style=,create=false,trim=false,indent=2]"
DEBU[0004] Updater finished changes=true repository=dailymotion-oss/octopilot updater="Regex[pattern=(?ms)(.*),file=README.md]"
DEBU[0004] All updaters finished repository=dailymotion-oss/octopilot
DEBU[0005] Git status repository-name=octopilot status=" M README.md\n M .golangci.yml\n M .goreleaser.yml\n"
DEBU[0006] Git commit commit=d263de874faf26a6ccc8bb2325bc4eb47e0a7029 repository-name=octopilot
WARN[0006] Running in dry-run mode, not pushing changes repository=dailymotion-oss/octopilot
WARN[0006] Repository update has no changes repository=dailymotion-oss/octopilot
INFO[0006] Updates finished repositories-count=1
The 2 interesting lines are:
Git repository cloned local-path=/var/folders/v0/fx5l3skn17785d8f4l883m6w0000gp/T/octopilot092369223/dailymotion-oss/octopilot
Git commit commit=d263de874faf26a6ccc8bb2325bc4eb47e0a7029
If you go to the directory identified by the local-path
value, you can inspect the git repository - for example:
$ cd /var/folders/v0/fx5l3skn17785d8f4l883m6w0000gp/T/octopilot092369223/dailymotion-oss/octopilot
$ git show d263de874faf26a6ccc8bb2325bc4eb47e0a7029
and you should see:
commit d263de874faf26a6ccc8bb2325bc4eb47e0a7029 (HEAD -> octopilot-c3qif432dnc2961tuuh0)
Author: author <author@example.com>
Date: Mon Jul 19 09:19:46 2021 +0200
Octopilot update
Updates:
### YQ[file=.goreleaser.yml,expression=(.dockers[] | select(.dockerfile == "Dockerfile.goreleaser") | .dockerfile) = "a.new.Dockerfile",output=,indent=2]
Update .goreleaser.yml
Updating file(s) `.goreleaser.yml` using yq expression `(.dockers[] | select(.dockerfile == "Dockerfile.goreleaser") | .dockerfile) = "a.new.Dockerfile"`
### YAML[path=run.timeout,file=.golangci.yml,style=,create=false,trim=false,indent=2]
Update .golangci.yml
Updating path `run.timeout` in file(s) `.golangci.yml`
### Regex[pattern=(?ms)(.*),file=README.md]
Update README.md
Updating file(s) `README.md` using pattern `(?ms)(.*)`
--
Generated by [Octopilot](https://github.com/dailymotion-oss/octopilot) [v0.2.16](https://github.com/dailymotion-oss/octopilot/releases/tag/v0.2.16) from https://github.com/dailymotion-oss/octopilot
diff --git a/.golangci.yml b/.golangci.yml
index 9a84d2c..b8a04d9 100644
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -1,4 +1,4 @@
# See https://golangci-lint.run/usage/configuration/#config-file
run:
- timeout: 3m
+ timeout: 42m
diff --git a/.goreleaser.yml b/.goreleaser.yml
index 3d8734f..1b96958 100644
--- a/.goreleaser.yml
+++ b/.goreleaser.yml
@@ -12,12 +12,10 @@ builds:
- -X main.buildDate={{.Date}}
env:
- CGO_ENABLED=0
-
archives:
- format: binary
-
dockers:
- - dockerfile: Dockerfile.goreleaser
+ - dockerfile: a.new.Dockerfile
image_templates:
- "ghcr.io/dailymotion-oss/{{.ProjectName}}:{{ .Version }}"
- "ghcr.io/dailymotion-oss/{{.ProjectName}}:{{ .Tag }}"
@@ -31,6 +29,5 @@ dockers:
- "--label=org.opencontainers.image.revision={{.FullCommit}}"
- "--label=org.opencontainers.image.version={{.Version}}"
- "--label=org.opencontainers.image.source={{.GitURL}}"
-
changelog:
sort: asc
diff --git a/README.md b/README.md
index a1dbefa..b247a46 100644
--- a/README.md
+++ b/README.md
@@ -1,115 +1 @@
-# Octo Pilot
-...
+replacing the content of the README.md file with this new content
This is a good way to ensure that you have the right syntax for your updater(s), and/or that your git commit is what you want.
Detecting errors in scripts
By default Octopilot does not exit with error even when updates fail. Add the --fail-on-error
flag so that Octopilot exits with error when updates fail.
Handle maximum number of repositories in parallel
By default Octopilot handles all repositories in parallel - creating as many goroutines as there are repositories.
Add the --max-concurrent-repos
flag so that Octopilot handles them in a batch way to avoid issues such as github rate limiting, or high load on your CI platform.