v1.12.12 released on Sep 4, 2024

Documentation for Octopilot v1.12.12

Released on Sep 4, 2024

This is the full 1-page documentation for Octopilot v1.12.12.

Changelog

Documentation

  • github auth method flag instead of token (#351)

Introduction

Octopilot is a CLI tool designed to help you automate your Gitops workflow, by automatically creating and merging GitHub Pull Requests to update specific content in Git repositories.

If you are doing Gitops with GitHub-hosted repositories, Octopilot is your swiss army knife to propagate changes in your infrastructure.

Octopilot was initially developed at Dailymotion, and is a core component of our Gitops workflow - you can read our blog post Introducing Octopilot: a CLI to automate the creation of GitHub pull requests in your gitops workflow.

It works by:

If you want to see what you can do with Octopilot for real, here is a set of real-world use-cases that we have at Dailymotion:

Usage

You can download the binary or see the docker pull commands from the release page on GitHub.

There are no dependencies - not even on git.

There are no configuration files - we use CLI flags for everything. You can run octopilot -h to see all the flags. Some of them can be used multiple times, such as the --repo or --update flags.

You will need a GitHub token or app to authenticate with GitHub. See the GitHub Auth section for more details.

Here is an example of using Octopilot to update multiple repositories, using multiple updaters:

$ octopilot \
    --github-token "my-github-token" \
    --repo "my-org/some-repo" \
    --repo "my-org/another-repo(merge=true)" \
    --repo "discover-from(env=PROMOTE_TO_REPOSITORIES)" \
    --repo "discover-from(query=org:my-org topic:my-topic)" \
    --repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path in-file-text)" \
    --repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path fork:true)" \
    --update "yaml(file=config.yaml,path='version')=file(path=VERSION)" \
    --update "yq(file=helmfile.yaml,expression='(.releases[] | select(.chart == \"repo/my-chart\") | .version ) = strenv(VERSION)')" \
    --update "sops(file=secrets.yaml,key=path.to.base64encodedCertificateKey)=$(kubectl -n cert-manager get secrets tls-myapp -o template='{{index .data \"tls.key\"}}')" \
    --pr-title "Updating some files" \
    ...

Continuous Delivery Pipelines

Octopilot has been designed to be used in a Continuous Delivery pipeline: no dependencies, no configuration file, only 1 command to update multiple repositories…

You can use it with Jenkins, Jenkins X, Tekton, GitHub Actions, …

At Dailymotion, we’re using it through Jenkins X/Tekton pipelines, and also a few Jenkins pipelines.

Repositories

Octopilot operates on GitHub repositories. A single execution of Octopilot can update one or more repositories. There are 2 ways to define the repositories to update:

Static list

It is the easiest: just specify the repositories on the CLI using the --repo flag, such as:

$ octopilot \
    --repo "my-github-org/my-first-repo" \
    --repo "my-github-org/my-second-repo(merge=true)" \
    --repo "my-github-org/my-third-repo(draft=true,merge=false,branch=dev)"

You can add as much repositories as you want, each with different configuration.

It supports the following parameters:

  • merge (boolean): if true, then the PR created on this repository will be automatically merged - see the Pull Requests section for more details. It overrides the value of the --pr-merge flag for this specific repository.
  • mergeauto (boolean): if true, then the PR will merged by Github’s auto-merge PR feature. See the Pull Requests section for more details. It overrides the value of the --pr-merge-auto flag for this specific repository.
  • mergeautowait (boolean): if true, then wait until the PR is actually merged. See the Pull Requests section for more details. It overrides the value of the --pr-merge-auto-wait flag for this specific repository.
  • draft (boolean): if true, then the PR will be created as a draft PR on GitHub. You will need to manually mark it as “ready for review” before being able to merge it. It overrides the value of the --pr-draft flag for this specific repository.
  • branch (string): the name of the base branch to use when cloning the repository. Default to the HEAD branch - which means the default branch configured in GitHub: usually main or master.

Dynamic list

Octopilot can also be used with a “dynamic” list of repositories: repositories that are unknown when you write the command arguments.

It can retrieve the list of repositories to update from:

Using environment variables

At runtime, Octopilot will read the list of repositories defined in one or more environment variables, such as:

$ export PROMOTE_TO_REPOSITORIES="my-github-org/my-first-repo my-github-org/my-second-repo(draft=true,merge=false)"
$ export ANOTHER_SET_OF_REPOSITORIES="some-org/some-repo;another-org/another-repo(draft=false)"
$ octopilot \
    --repo "discover-from(env=PROMOTE_TO_REPOSITORIES)" \
    --repo "discover-from(env=ANOTHER_SET_OF_REPOSITORIES,sep=;,draft=true)"

It supports the following parameters:

  • env (string): the name of the environment variable to use, to retrieve the list of repositories.
  • sep (string): the separator between each repository, default to " " (space).
  • merge (boolean): if true, then the PRs created on the repositories from this env var will be automatically merged - see the Pull Requests section for more details. It overrides the value of the --pr-merge flag for the repositories defined in this env var.
  • mergeauto (boolean): if true, then the PR will merged by Github’s auto-merge PR feature. See the Pull Requests section for more details. It overrides the value of the --pr-merge-auto flag for these specific repository.
  • mergeautowait (boolean): if true, then wait until the PR is actually merged. See the Pull Requests section for more details. It overrides the value of the --pr-merge-auto-wait flag for these specific repository.
  • draft (boolean): if true, then the PRs will be created as draft PRs on GitHub. You will need to manually mark them as “ready for review” before being able to merge them. It overrides the value of the --pr-draft flag for the repositories defined in this env var.
  • branch (string): the name of the base branch to use when cloning the repositories. Default to the HEAD branch - which means the default branch configured in GitHub: usually main or master.

Note that each repository listed in an environment variable supports all the parameters defined in the static definition of a repository.

Using GitHub Search Query

A more powerful feature is the ability to load a list of repositories from a GitHub Repositories Search Query or a GitHub Code Search Query, such as:

$ octopilot \
    --repo "discover-from(query=org:my-github-org topic:some-topic)" \
    --repo "discover-from(query=org:my-github-org in:readme some-specific-content-in-the-readme,draft=true)" \
    --repo "discover-from(query=org:my-github-org language:java is:private mirror:false archived:false,merge=true)" \
    --repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path in-file-text)" \
    --repo "discover-from(searchtype=code,query=org:my-org filename:my-file path:dir-path fork:true)"

At runtime, Octopilot will use the GitHub API to retrieve the list of repositories matching a given query. This is useful when you have a common library used/imported by many repositories, and you want to create a PR to update the version when there is a new release of your lib. Instead of hardcoding the list of “dependant” repositories in your library repository, you can use a GitHub Search Query to find all repositories with a specific topic, or specific content in the description of the repo, or specific content in the README.md file of the repo, and so on. So “dependant” repositories can easily opt-in to get automatic PRs just by adding a topic for example.

It supports the following parameters:

  • searchtype (string): represents the type of github search to be performed. Only code and repositories are availables. Default to repositories.
  • query (string): Specifies search criteria for listing repositories, including filters for file contents, location, language, topic, and more.
  • merge (boolean): if true, then the PRs created on the repositories from this query will be automatically merged - see the Pull Requests section for more details. It overrides the value of the --pr-merge flag for the repositories retrieved from this query.
  • mergeauto (boolean): if true, then the PR will merged by Github’s auto-merge PR feature. See the Pull Requests section for more details. It overrides the value of the --pr-merge-auto flag for these specific repository.
  • mergeautowait (boolean): if true, then wait until the PR is actually merged. See the Pull Requests section for more details. It overrides the value of the --pr-merge-auto-wait flag for these specific repository.
  • draft (boolean): if true, then the PRs will be created as draft PRs on GitHub. You will need to manually mark them as “ready for review” before being able to merge them. It overrides the value of the --pr-draft flag for the repositories retrieved from this query.
  • branch (string): the name of the base branch to use when cloning the repositories. Default to the HEAD branch - which means the default branch configured in GitHub: usually main or master.

See the “promoting a new library release” use-case for a real-life example of what you can do with this feature.

Commits

After running all the updaters, octopilot will commit the changes on each repository. You can control which files will be committed, and how the commit will be created, using the Octopilot’s CLI flags.

Git clone

Flags related to the git clone operation:

  • --git-clone-dir (string): optional path to a directory used to clone the git repositories. Default to a newly created directory in the system temporary directory (defined by the TMPDIR environment variable, defaulting to /tmp). Note that by default all files created inside this directory will be deleted at the end of the process, unless the --keep-files flag is set.

  • --git-recurse-submodules (bool): recursively initialize all submodules. Disabled by default.

Git index/stage

By default, all files changed by the updaters will be added to the git “index” - so that they can be added to the git commit. This is configurable through the following flags:

  • --git-stage-all-changed (boolean): if set to true (the default), then all changed files will be “staged” - or added to the git index. Set it to false to control which files should be staged.
  • --git-stage-pattern (array of string): list of path patterns that will be “staged” - or added to the git index.

For example, to make sure you will only commit the changes to the helmfile.yaml file:

$ octopilot \
    --git-stage-all-changed=false \
    --git-stage-pattern=helmfile.yaml \
    ...

Git commit

Although there are good default values, you can configure how Octopilot will create the git commit:

  • git-author-name (string): the name of the author of the git commit. Default to the value of the GIT_AUTHOR_NAME environment variable, or if unset, the user.name git config value.
  • git-author-email (string): the email of the author of the git commit. Default to the value of the GIT_AUTHOR_EMAIL environment variable, or if unset, the user.email git config value.
  • git-committer-name (string): the name of the committer. Default to the value of the GIT_COMMITTER_NAME environment variable, or if unset, the user.name git config value.
  • git-committer-email (string): the email of the committer. Default to the value of the GIT_COMMITTER_EMAIL environment variable, or if unset, the user.email git config value.
  • git-commit-title (string): the title of the commit. Note that you can use the templating feature here.
  • git-commit-body (string): the body of the commit. Note that you can use the templating feature here.
  • git-commit-footer (string): the footer of the commit. Default to Octopilot’s signature, to indicate that this commit was generated by a tool.

The commit message is composed of the commit title, body and optional footer:

${git-commit-title}

${git-commit-body}

-- 
${git-commit-footer}

For example, to update the version of an application named my-app, and write a nice commit message:

$ octopilot \
    --repo "my-org/my-gitops-repo" \
    --update "yaml(file=my-app/config.yaml,path='version')=${VERSION}" \
    --git-commit-title 'chore(deps): update my-app to {{ env "VERSION" }}' \
    --git-commit-body '{{ githubRelease (print "my-org/my-app-repo/" (env "VERSION")) | md2txt }}' \
    ...

It’s using the templating feature to retrieve the GitHub Release for your application’s version, and convert it to raw text. So in your commit message, you’ll see what changed in the new version.

Git push

  • git-branch-prefix (string): when pushing the changes to the “origin” git repository, a new branch with a random name will be created. You can control the prefix of this random name, which default to octopilot-.

Note that Octopilot requires permissions to push on the GitHub repositories to update. For the moment, it doesn’t support forking the repository, and creating the Pull Request from the fork.

Git signing

  • git-signing-key-path (string): when provided, Octopilot will use the GPG private key file to sign commits or tags, which will allow GitHub to verify the committer’s identity. See Add a GPG key.
  • git-signing-key-passphrase (string): Passphrase to decrypt the GPG private key used to sign commits. Leave empty if the private key is stored unencrypted.

Pull Requests

After running all the updaters and creating a git commit, octopilot will create a Pull Request for each repository.

Strategies

Octopilot has 3 strategies for creating a Pull Request:

  • reset (the default): reset any existing Pull Request from the base branch
  • append: append new commits to any existing Pull Request
  • recreate: always create a new Pull Request

You can control which strategy to use using the --strategy CLI flag.

Reset Strategy

This is the default strategy. With this strategy, Octopilot will reset any existing Pull Request from the base branch.

In detail, it will:

  • clone the git repository
  • find a “matching” Pull Request - based on the pre-configured labels. If there is a matching Pull Request, it will use the PR’s branch. Otherwise, it will just create a new branch
  • reset the branch to the base branch - usually main or master
  • run the updaters
  • commit the changes and (force) push the commit
  • update the existing Pull Request title/body/labels/comments/assignees or create a new one

Note that you can control how the existing Pull Request will be updated. For both the title and body, you can either:

  • ignore the new changes. The existing PR won’t be changed - except for the labels, comments, assignees, and of course the commit.
  • replace the title and/or body with the new ones. This is the default for the reset strategy.
  • prepend the title and/or body with the new ones. This is mostly useful for the body.
  • append the title and/or body with the new ones. This is mostly useful for the body.

Append Strategy

With this strategy, Octopilot will append new commits to any existing Pull Request.

In detail, it will:

  • clone the git repository
  • find a “matching” Pull Request - based on the pre-configured labels. If there is a matching Pull Request, it will switch to the PR’s branch. Otherwise it will just create a new branch from the base branch, and switch to it.
  • run the updaters
  • commit the changes and push the commit
  • update the existing Pull Request title/body/labels/comments/asignees or create a new one

Note that you can control how the existing Pull Request will be updated. For both the title and body, you can either:

  • ignore the new changes. The existing PR won’t be changed - except for the labels, comments, assignees, and of course the commit. This is the default for the append strategy.
  • replace the title and/or body with the new ones.
  • prepend the title and/or body with the new ones. This is mostly useful for the body.
  • append the title and/or body with the new ones. This is mostly useful for the body.

Recreate Strategy

With this strategy, Octopilot will always create a new Pull Request.

In detail, it will:

  • clone the git repository
  • create a new branch from the base branch, and switch to it
  • run the updaters
  • commit the changes and push the commit
  • create a new Pull Request

Creating / updating Pull Requests

You can control how the Pull Requests will be created or updated using the following CLI flags:

  • --strategy (string): strategy to use when creating/updating the Pull Requests: either reset (reset any existing PR from the current base branch), append (append new commit to any existing PR) or recreate (always create a new PR). Default to reset.
  • --dry-run (bool): if enabled, won’t perform any operation on the remote git repository or on GitHub: all operations will be done in the local cloned repository. So no Pull Request will be created/updated. Default to false.
  • --pr-title (string): the title of the Pull Request. Default to the commit title. Note that you can use the templating feature here.
  • --pr-title-update-operation (string): the type of operation when updating a Pull Request’s title: either ignore (keep old value), replace, prepend or append. Default is: ignore for “append” strategy, replace for “reset” strategy, and not applicable for “recreate” strategy.
  • --pr-body (string): the body of the Pull Request. Default to the commit body and the commit footer. Note that you can use the templating feature here.
  • --pr-body-update-operation (string): the type of operation when updating a Pull Request’s body: either ignore (keep old value), replace, prepend or append. Default is: ignore for “append” strategy, replace for “reset” strategy, and not applicable for “recreate” strategy.
  • --pr-comment (array of string): optional list of comments to add to the Pull Request.
  • --pr-assignees (array of string): optional list of assignees (Github usernames) to add to the Pull Request.
  • --pr-reviewers (array of string): optional list of reviewers (Github usernames) for the Pull Request.
  • --pr-team-reviewers (array of string): optional list of team reviewers (Github team names) for the Pull Request.
  • --pr-labels (array of string): optional list of labels to set on the pull requests, and used to find existing pull requests to update. Default to ["octopilot-update"].
  • --pr-base-branch (string): name of the branch used as a base when creating pull requests. Default to master.
  • --pr-draft (bool): if enabled, the Pull Request will be created as a draft - instead of regular ones. It means that the PRs can’t be merged until marked as “ready for review”. Default to false.

Merging Pull Requests

Optionally, Octopilot can also automatically merge the Pull Requests it creates. Before merging a Pull Request, Octopilot will wait for the PR to be in a “mergable” state, and for all required status checks to pass.

  • --pr-merge (bool): if enabled, the Pull Requests will be automatically merged. It will wait until the PRs are “mergeable” before merging them. Default to false.

All the following flags only apply if --pr-merge is enabled.

  • --pr-merge-auto (bool): merge PRs using Github’s auto-merge PR feature. By default, this will not wait until the PR is merged. Note, this must also be enabled at the repository level for it to work. This is a one-time task and ocotopilot will not do this for you. One way of doing this in bulk is using the gh CLI: gh search repos --json url --jq '.[].url' ... | xargs -L1 gh repo edit --enable-auto-merge.
  • --pr-merge-auto-wait (bool): when --pr-merge-auto is enabled, wait for the PR to be merged by Github.
  • --pr-merge-method (string): the merge method to use. Either merge, squash, or rebase. Default to merge.
  • --pr-merge-commit-title (string): optional title of the merge commit.
  • --pr-merge-commit-message (string): optional body of the merge commit.
  • --pr-merge-sha (string): optional SHA that pull request head must match to allow merge.
  • --pr-merge-poll-timeout (string/duration): maximum duration to wait for a Pull Request to be mergeable/merged, using the Golang syntax. Default to 10m (10 minutes).
  • --pr-merge-poll-interval (string/duration): duration to wait for between each GitHub API call to check if a PR is mergeable/merged, using the Golang syntax. Default to 30s (30 seconds).
  • --pr-merge-retry-count (int): number of times to retry the merge operation in case of merge failure. Default to 3.
  • --pr-merge-branch-protection (string): One of statusChecks, all, bypass. Wait for the specified kind of branch protection rules to be satisfied before attempting to merge.
    • statusChecks waits only for status checks to be passing. This is the default.
    • all waits for every rule (approvals, commit signature, etc).
    • bypass will bypass branch protection rules when possible (i.e. the authenticated user/app have permissions to do so).

Updaters

The core feature of Octopilot is to update git repositories, and to do it you can use one or more of the available “updaters”:

Each updater can be used once or more, such as:

$ octopilot \
    --update "yaml(file=config.yaml,path='version')=file(path=VERSION)" \
    --update "regex(file=some-file.txt,pattern='version: \"(.*)\"')=${VERSION}" \
    --update "yaml(file=another-config.yaml,path='path.to.version')=$(cat VERSION)" \
    ...

YAML

The YAML updater is great when you want to quickly set a value for a specific path in one or more files. Such as if you want to update a version used in a YAML file:

$ octopilot \
    --update "yaml(file=config.yaml,path='app.version')=file(path=VERSION)" \
    ...

Given the following config.yaml file:

app:
  name: foo
  version: 1.0.0

Octopilot will set the value of the app.version key to the content of the VERSION file.

The syntax is: yaml(params)=value - you can read more about the value in the “value” section.

It supports the following parameters:

  • file (string): mandatory path to the file to update. Can be a file pattern - such as config/*.yaml to match files in the same directory, or config/**/*.yaml using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.
  • path (string): mandatory path to the key to update in the YAML file(s). We support yq v3 path expressions or yq v4 syntax.
  • indent (int): optional number of spaces used for indentation when writing the YAML file(s) after update. Default to 2.
  • trim (boolean): if true, the content will be “trimmed” before being written to disk - to avoid extra line break at the end of the file for example.
  • create (boolean): if true, then the path will always be set to the given value, even if no such key existed before. The default behaviour (false) is to NOT create any new path/key.
  • style (string): an optional style to apply to the new value: double (add double quotes), single (add single quotes), literal, folded or flow - see yq style reference.

Note that Octopilot will keep the comments in the YAML files - because we’re using the great go-yaml v3 lib. Just that it might rewrite a bit your indentation.

See the “updating certificates” use-case for a real-life example of what you can do with this updater.

YQ

The YQ updater is based on the excellent yq application - and Go lib. It is much more powerful than the basic YAML updater - because it supports all the yq operators, and because you are not limited to setting a value for a specific key. You can do very powerful things, such as manipulating YAML comments, use variables, output to json (note that it can also read JSON input), …

The syntax is: yq(params), such as:

$ octopilot \
    --update "yq(file=config.yaml,expression='.path.to.version = strenv(VERSION)')" \
    ...

It supports the following parameters:

  • file (string): mandatory path to the file to update. Can be a file pattern - such as config/*.yaml to match files in the same directory, or config/**/*.yaml using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.
  • expression (string): mandatory yq v4 expression that will be evaluated against each file.
  • output (string): optional output of the result. By default the result is written to the source file - in-place editing. But you can send the result to stdout, stderr or a specific file.
  • json (boolean): if true, then the output will be written in JSON format instead of YAML format.
  • indent (int): optional number of spaces used for indentation when writing the YAML file(s) after update. See yq doc on indent. Default to 2.
  • trim (boolean): if true, the content will be “trimmed” before being written to disk - to avoid extra line break at the end of the file for example.
  • unwrapscalar (boolean): if true (the default), only the value will be printed - not the comments. See yq doc on unwrap scalars.

Note that Octopilot will keep the comments in the YAML files - because we’re using the great go-yaml v3 lib. Just that it might rewrite a bit your indentation.

See the “promoting a new application release” use-case for a real-life example of what you can do with this updater.

Helm

The Helm updater is made to easily update the dependencies of one or more Helm charts. It supports both:

  • Helm v3, with the dependencies declared in the Chart.yaml file
  • Helm v2, with the dependencies declared in the requirements.yaml file

If you run the following command:

$ octopilot \
    --update "helm(dependency=chart-name)=1.2.3" \
    ...

Octopilot will discover all charts stored in the cloned repository, and for each, try to change the version of the chart-name dependency to 1.2.3.

The syntax is: helm(params)=value - you can read more about the value in the “value” section.

It supports the following parameters:

  • dependency (string): mandatory name of the dependency to update. Must exist in the dependencies list - it won’t be added.
  • indent (int): optional number of spaces used for indentation when writing the YAML file(s) after update. Default to 2.

Note that Octopilot will keep the comments in the YAML files - because we’re using the great go-yaml v3 lib. Just that it might rewrite a bit your indentation.

See the “promoting a new library release” use-case for a real-life example of what you can do with this updater.

Sops

The sops updater can manipulate files encrypted with mozilla’s sops natively, without the need to install sops. This is great if you want to store sensitive data in your git repositories: you can encrypt them with sops, and use Octopilot to update them automatically.

For example if you want to store your TLS certificate as a base64 encoded string in a sops-encrypted file:

$ octopilot \
    --update `sops(file=secrets.yaml,key=app.tls.base64encodedCertificateKey)=$(kubectl -n cert-manager get secrets tls-myapp -o template='{{index .data "tls.key"}}')` \
    ...

Given the following (decrypted) secrets.yaml file:

app:
  tls:
    base64encodedCertificateKey: LS0tLS1CRUdJTiBSU0EgU...

Octopilot will decrypt the secrets.yaml file, set the value of the app.tls.base64encodedCertificateKey key to the given value, and re-encrypt the secrets.yaml file before writing it to disk.

The syntax is: sops(params)=value - you can read more about the value in the “value” section.

It supports the following parameters:

  • file (string): mandatory path to the file to update. Can be a file pattern - such as config/secrets.* to match files in the same directory, or config/**/secrets.* using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.
  • key (string): mandatory key to update in the file(s).

Note that depending on the sops backend you use (KMS, age, vault, …) you might need to set some environment variables, such as:

  • for GCP KMS, the GOOGLE_APPLICATION_CREDENTIALS env var
  • for age, the SOPS_AGE_KEY_FILE env var
  • …

See the “updating certificates” use-case for a real-life example of what you can do with this updater.

Regex

The regex updater can be used to update any kind of text file using a regular expression - with the Golang syntax, such as:

$ octopilot \
    --update "regex(file=some-file.txt,pattern='version: \"(.*)\"')=${VERSION}" \
    ...

Given the following some-file.txt file:

version: "1.0.0"
...

Octopilot will replace the first line with version: "1.2.3" if the $VERSION env var is set to 1.2.3 for example.

The syntax is: regex(params)=value - you can read more about the value in the “value” section.

It supports the following parameters:

  • file (string): mandatory path to the file to update. Can be a file pattern - such as files/*.txt to match files in the same directory, or files/**/*.txt using double asterisks (**) to match files in subdirectories. If it’s a relative path, it will be relative to the root of the cloned git repository. For more information on using file patterns, you can refer to the go-zglob documentation.
  • pattern (string): mandatory regex pattern to find and replace something in the file(s). The pattern must be in the Golang syntax. If this pattern includes a capturing group, then it will be replaced by the provided value.

A few things you can do with the regex updater:

  • replace the whole content of a file
    $ octopilot \
    --update "regex(file=my-file,pattern='(?ms)(.*)')=new content"

Exec

The exec updater can execute any command you want, so you can change files in the cloned git repository with any tool you have available.

For example to update all your Go dependencies to the latest version:

$ octopilot \
    --update "exec(cmd=go,args=get -d -t -u)" \
    --update "exec(cmd=go,args=mod tidy)" \
    --update "exec(cmd=go,args=mod vendor)" \
    --git-stage-pattern "vendor" \
    ...

This will execute 3 commands, that will update the go.mod & go.sum files, and the vendor dir. Octopilot will then add/commit all the changes, including the new files in the vendor directory.

The syntax is: exec(params).

It supports the following parameters:

  • cmd (string): mandatory command to execute.
  • path (string): optional path to execute the command in.
  • args (string): optional arguments for the command. The arguments are space-separated. If you have a space in an argument, you can quote it, such as: -c 'some arg' -x another.
  • stdout (string): optional path to a file where the std output of the command will be written. If it’s a relative path, it will be relative to the root of the cloned git repository.
  • stderr (string): optional path to a file where the std error output of the command will be written. If it’s a relative path, it will be relative to the root of the cloned git repository.
  • timeout (string/duration): optional maximum duration to wait for the command to finish, using the Golang syntax.

A few things you can do with the regex updater:

  • use a bash command to enable shell expansion (disabled by default when invoking commands through Octopilot)

    $ octopilot \
    --update "exec(cmd=sh,args=-c 'cat files/*.txt',stdout=output.txt)"

  • use the an external tool (here kustomize) and change the current working directory before executing the command

    $ octopilot \
    --update "exec(cmd=kustomize, path=k8s/overlays/dev, args=edit set image containername=registry.tld/repo/image:newtag)"

See the “updating go dependencies” use-case for a real-life example of what you can do with this updater.

Value

Some updaters accept a value in their syntax, such as updater(params)=value.

This value can be either:

  • a raw value
  • the content of a file

Raw value

This is the easiest way to set a value: just use a raw value, such as:

$ octopilot \
    --update "yaml(file=config.yaml,path='version')=v1.2.3" \
    ...

Note that you can also use an environment variable:

$ export VERSION=v1.2.3
$ octopilot \
    --update "yaml(file=config.yaml,path='version')=${VERSION}" \
    ...

or any command you want:

$ echo v1.2.3 > /tmp/VERSION
$ octopilot \
    --update "yaml(file=config.yaml,path='version')=$(cat /tmp/VERSION)" \
    ...

File content

If you want to use the content of a file, you can use the file valuer:

$ octopilot \
    --update "yaml(file=config.yaml,path='version')=file(path=VERSION)" \
    ...

It will read the VERSION file located at the root of the cloned git repository, and use its content as the value.

The syntax is: file(params).

It supports the following parameters:

  • path (string): mandatory path to the file to read. If it’s a relative path, it will be relative to the root of the cloned git repository.

GitHub Auth

Octopilot needs a way to authenticate against the GitHub API, to:

  • clone the repositories
  • push new changes
  • and create/update/merge Pull Requests

It supports 2 ways to authenticate:

  • using a personal access token, which is the default
  • using a GitHub app

You can define which method to use, using the --github-auth-method CLI flag.

Personal Access Token

By default, the --github-auth-method flag is set to token, so Octopilot will use a Personal Access Token - or PAT. This token can be defined either by the GITHUB_TOKEN environment variable, or by setting the --github-token CLI flag.

You can read GitHub’s documentation on creating a personal access token. You’ll need at least the repo permissions.

GitHub App

An alternative to the “simple” token is to use a GitHub App.

First, you’ll need to set the --github-auth-method flag value to app, and then configure the following settings:

  • --github-app-id (int): the GitHub App ID. Default to the value of the GITHUB_APP_ID environment variable.
  • --github-installation-id (int): the GitHub App installation ID. Default to the value of the GITHUB_INSTALLATION_ID environment variable.
  • --github-privatekey (string): the app’s private key - used to sign access token requests - in PEM format. Default to the value of the GITHUB_PRIVATEKEY environment variable. You can either set this, or the --github-privatekey-path.
  • --github-privatekey-path (string): the path to the app’s private key - used to sign access token requests - in PEM format. Default to the value of the GITHUB_PRIVATEKEY_PATH environment variable. Will be used if the --github-privatekey flag is not set.

See GitHub’s documentation for more details on:

Using octopilot with Enterprise GitHub

By default, octopilot will operate on repositories hosted on https://github.com. Octopilot can work on repositories hosted on an Enterprise Github servers by adding the --github-url flag. This flag must be set to the URL of the Enterprise GitHub server (the URL used to browse to the GitHub server main page). Authentication is usually required using any of the authentication method described above.

Example of use:


$ octopilot \
    --github-url https://mygitserver.acme.com \
    --repo "${ORG_NAME}/test-repo" \
    ...

Advanced

Octopilot advanced features:

Templating

For some of the CLI flags - such as commit title/body or Pull Requests title/body - you can use our “templating” feature. This will allow you to write nice commits and Pull Requests.

Octopilot uses the Go template syntax, and supports the following functions:

Octopilot’s own custom functions

Octopilot comes with the following custom functions:

  • readFile to read a file from the cloned git repository, and return its content
  • githubRelease to retrieve the release notes for a specific GitHub release
  • expandGithubLinks to transform GitHub short links - such as #123 - to absolute URLs
  • extractMarkdownURLs to transform markdown links to plain URLs
  • md2txt to strip all the markdown syntax from a string

readFile

The readFile function will read a file from the cloned git repository, and return its content as a string. If the path of the file is relative, it will be relative to the root of the git repository.

Definition: readFile(filePath string) string.

Example: {{readFile "README.md"}} to print the content of the README.md file.

githubRelease

The githubRelease function will retrieve the release notes for a specific GitHub release. The release is identified by the owner, repository and release version.

Definition: githubRelease(releaseID string) string.

Example: {{ githubRelease "owner/repo/v1.2.3" }} to print the release notes for the release v1.2.3 of the owner/repo github repository.

The expandGithubLinks function will transform GitHub short links - such as #123 - to absolute URLs. It is most useful when combined with the githubRelease function, to ensure that the links in the release notes are always absolute.

Definition: expandGithubLinks(fullRepositoryName string, markdownInput string) string. The fullRepositoryName parameter is the full name of the repository, such as owner/repo. It is used to build the absolute URLs.

Example: {{ githubRelease (print "owner/repo/" (env "VERSION")) | expandGithubLinks "owner/repo" }}.

extractMarkdownURLs

The extractMarkdownURLs function will transform markdown links to plain URLs. It is most useful when combined with the githubRelease function, to ensure that the links in the release notes are always plain URLs - when you want to print plain text, and not markdown, such as in a commit message.

Definition: extractMarkdownURLs(markdownInput string) string.

Example: {{ githubRelease (print "owner/repo/" (env "VERSION")) | expandGithubLinks "owner/repo" | extractMarkdownURLs }}.

md2txt

The md2txt function will strip all the markdown syntax from a string. It is most useful when combined with the githubRelease function, to ensure that the release notes are always plain text, and not markdown. Use it for your commit messages.

Definition: md2txt(markdownInput string) string.

Example: {{ githubRelease (print "owner/repo/" (env "VERSION")) | expandGithubLinks "owner/repo" | extractMarkdownURLs | md2txt }}.

Use cases

If you want to see what you can do with Octopilot for real, here is a set of real-world use-cases that we have at Dailymotion:

Promoting a new application release

If you’re doing gitops, you’ll most likely have:

  • at least one “application” git repository, for your(s) application(s).
  • at least one “environment” git repository, for your(s) environment(s).

When you release a new version of your application, you’ll need to “promote” it to (at least) one of your environments. In the gitops world, we’re doing that by creating a new Pull Request against the target environment git repository, to update the version of the application in a configuration file.

And this is where Octopilot shines. You can use it in your application’s Continuous Delivery pipeline to create the Pull Request against your environment’s git repository.

Let’s see an example, where we want to:

  • automatically deploy new releases in the staging environment
  • manually deploy new releases in the production environment

To do that, we’ll:

  • create a Pull Request on the staging-env git repository, and ask Octopilot to automatically merge it - and thus deploying it
  • create a Pull Request on the prod-env git repository, but not merge it. We’ll even create the Pull Request as a “draft”, making it explicit that the release is not ready yet.

Oh, and we’d like to get nice commit messages and Pull Request title/description, so that people can understand what really is changing with this Pull Request. If all you have is “Update my-app to version 1.2.3”, and a diff that shows a version change in a config file, it won’t help you understand if this release is just fixing a typo in a README, fixing a critical bug, or introducing a new feature. We’re already generating release notes with a changelog - if not, you should, using git-chglog for example - so let’s re-use it and include it in our commit messages and Pull Request.

Here is an example of an Octopilot invocation you can use to achieve our goal - we’ll go over it in details later:

$ export GITHUB_TOKEN=<your_github_token>
$ export ORG_NAME=my-org
$ export APP_NAME=my-app
$ export VERSION=1.2.3
$ octopilot \
    --repo "${ORG_NAME}/staging-env(merge=true)" \
    --repo "${ORG_NAME}/prod-env(draft=true)" \
    --update "yq(file=helmfile.yaml,expression='.releases[] | select(.name == strenv(APP_NAME)) | .version',output=.git/previous-version.txt)" \
    --update "yq(file=helmfile.yaml,expression='(.releases[] | select(.name == strenv(APP_NAME)) | .version) = strenv(VERSION)')" \
    --git-commit-title 'chore(deps): update {{ env "APP_NAME" }} from {{ readFile ".git/previous-version.txt" | trim }} to {{ env "VERSION" }}' \
    --git-commit-body '{{ githubRelease (printf "%s/%s/v%s" (env "ORG_NAME") (env "APP_NAME") (env "VERSION")) | expandGithubLinks (printf "%s/%s" (env "ORG_NAME") (env "APP_NAME")) | extractMarkdownURLs | md2txt }}' \
    --git-branch-prefix "octopilot-update-${APP_NAME}-" \
    --pr-labels "update-${APP_NAME}" \
    --pr-title "Update ${APP_NAME} to ${VERSION}" \
    --pr-title-update-operation "replace" \
    --pr-body '{{ githubRelease (printf "%s/%s/v%s" (env "ORG_NAME") (env "APP_NAME") (env "VERSION")) | expandGithubLinks (printf "%s/%s" (env "ORG_NAME") (env "APP_NAME")) }}' \
    --pr-body-update-operation "prepend" \
    --strategy "append"

As you can see, we’re making the same set of changes to 2 different repositories, with different configurations:

  • the PR on the staging-env repository will be automatically merged - and thus our release deployed
  • the PR on the prod-env repository will be created as “draft”, and won’t be automatically merged - thus requiring a human intervention to merge it

We’re running the YQ updater twice, on the same helmfile.yaml file, which would look like the following:

releases:
  - name: my-app
    version: 1.0.0
  - name: another-app
    version: 1.5.0

This is a very simplified configuration file for helmfile, an application used to describe Helm releases. You can use whatever you want, it’s just to base our example on a real-life use-case.

So we’re running the YQ updater twice:

  • the first time, with the .releases[] | select(.name == strenv(APP_NAME)) | .version expression, to extract the current version for our application named my-app. And we’re sending the output to the .git/previous-version.txt file - which will contain just 1.0.0. We’re doing that to store the “previous” version before changing it without new version. You’ll notice that the output file is located in the .git directory, this is to avoid committing it by default.
  • the second time, we’re replacing the version with the new one - stored in the VERSION environment variable - with the following expression: (.releases[] | select(.name == strenv(APP_NAME)) | .version) = strenv(VERSION).

Now, we have:

  • a locally modified helmfile.yaml
  • a new .git/previous-version.txt file

So it’s time to commit!

  • in the commit title, we want to include both the previous version and the new version. We’ll use the templating feature to include both the content of a file and an environment variable: chore(deps): update {{ env "APP_NAME" }} from {{ readFile ".git/previous-version.txt" | trim }} to {{ env "VERSION" }}
  • in the commit body, we want to include the release notes for the new version. We’ll use the templating feature to retrieve the release notes, convert all the GitHub short links to absolute URLs, and transform the markdown into raw text: {{ githubRelease (printf "%s/%s/v%s" (env "ORG_NAME") (env "APP_NAME") (env "VERSION")) | expandGithubLinks (printf "%s/%s" (env "ORG_NAME") (env "APP_NAME")) | extractMarkdownURLs | md2txt }}

We’ll also use a custom prefix for the branch name, to make it easier to find which branch belongs to which app, if you have multiple promotion PRs for different applications opened at the same time.

Next, the Pull Request. We’ll use the append strategy, which means that if we need to promote a release in prod, and the previous one hasn’t been merged/deployed yet, we’ll just append a new commit on the existing branch/PR. So that your production PR for your application will stay the same, accumulating releases until it is merged. We’re using a specific label for our Pull Request: update-${APP_NAME} - to make sure we’ll find any existing PR for our application, and that each application will get its own PR.

Same as for the commit, we’ll use the templating feature to write a nice title and description for our Pull Request - just that this time we won’t need to convert from markdown to raw text. And we’ll use specific “update operations”:

  • we’ll always replace the PR title, because we want a short title, with only the app name and the latest release’s version - using the --pr-title-update-operation "replace" flag
  • we’ll “prepend” the new PR body, before the existing one, using the --pr-body-update-operation "prepend" flag. So that the release notes for the latest release will be first - just as when you read a changelog. And of course we don’t want to remove the previous release notes.

Result

Staging environment

This is a screenshot of a Pull Request on the staging environment git repository, which has been automatically merged. You can see the release notes, and you’ll notice the “signature” at the bottom: we’re making it easy for people to know:

  • that this PR has been generated by an application - and not created manually by a human
  • which version of Octopilot has been used
  • from where it has been executed. This will most likely be the application’s GitHub repository, because it is where you (should) define your application’s Continuous Delivery pipeline, which contains a step to execute Octopilot.

Production environment

This is a screenshot of a Pull Request on the production environment git repository:

  • the PR has been created to promote v3.14.1 of the application
  • later, v3.15.0 has been released - and promoted. Thus adding a new commit to the PR
  • and then, some-user merged the Pull Request, to deploy in prod

You’ll notice that we have 2 release notes in the PR: 1 for each release. So you can see the full changelog for every release that will be deployed when you’ll merge this PR.

In this screenshot you can see the 2 commits:

Feedback

The benefit of adding the application’s release notes in the promotion pull request body, is that not only will you know exactly what you’ll deploy, but you’ll also get links between the application pull request and the promotion pull requests. So that if you go back to the application’s PR, you’ll see something like:

You can see at the bottom the links to our 2 promotion pull requests, with their statuses - both have been merged already in this case.

Going further

As you can see, it’s easy to adapt this example for your own use-case. For example, you might want:

Promoting a new library release

Promoting a new release of a library is similar to promoting a new release of an application, except that instead of promoting to a small number of git repositories - representing the different environments - you will promote to a much larger number of git repositories: one for each application using the library.

The main change is that instead of using a static list of repositories, you can use a dynamic list of repositories - based on a GitHub Repositories Search Query or a GitHub Code Search Query for example.

GitHub Repositories Search Query

Here is an example of an Octopilot invocation using a dynamic list of repos and GitHub Repositories Search Query - we’ll go over it in details later:

$ export GITHUB_TOKEN=<your_github_token>
$ export VERSION=1.0.15
$ octopilot \
    --repo "discover-from(query=org:my-org topic:octopilot-my-base-chart)" \
    --update "helm(dependency=my-base-chart)=${VERSION}" \
    --git-commit-title 'chore(deps): update my-base-chart to {{ env "VERSION" }}' \
    --git-commit-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" | extractMarkdownURLs | md2txt }}' \
    --git-branch-prefix "octopilot-update-my-base-chart-" \
    --pr-labels "update-my-base-chart" \
    --pr-title "Update Helm Base Chart to ${VERSION}" \
    --pr-title-update-operation "replace" \
    --pr-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" }}' \
    --pr-body-update-operation "prepend" \
    --strategy "append"

We’re using the following GitHub Repositories search query to discover the repositories that are using the my-base-chart Helm chart:

query=org:my-org topic:octopilot-my-base-chart

Which means that developers just need to add the octopilot-my-base-chart topic to their repositories - in the GitHub UI - to enable automatic update of the Helm base chart. It’s easy to add, and easy to turn off if you don’t want it. Note that you are not limited to searching by topic - you can also search for specific content in the repository’s name, description, README.md file, and so on.

We’re using the Helm updater to update all the Helm charts, and set the version of their dependency on my-base-chart to the new release version.

Everything else is similar to the promotion of a new application release.

GitHub Code Search Query

This search type is useful in case you experience some limitations with the default Repositories one

  • 50 characters max
  • 20 topics max per repository

Here is an example of an Octopilot invocation using a dynamic list of repos and GitHub Code Search Query - we’ll go over it in details later:

$ export GITHUB_TOKEN=<your_github_token>
$ export VERSION=1.0.15
$ octopilot \
    --repo "discover-from(searchtype=code,query=org:my-org filename:Chart.yaml path:charts my-base-chart)" \
    --update "yaml(file=charts/*/Chart.yaml,path=dependencies.(name==my-base-chart).version,style=folded)=${VERSION}" \
    --git-commit-title 'chore(deps): update my-base-chart to {{ env "VERSION" }}' \
    --git-commit-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" | extractMarkdownURLs | md2txt }}' \
    --git-branch-prefix "octopilot-update-my-base-chart-" \
    --pr-labels "update-my-base-chart" \
    --pr-title "Update Helm Base Chart to ${VERSION}" \
    --pr-title-update-operation "replace" \
    --pr-body '{{ githubRelease (printf "my-org/my-base-chart/v%s" (env "VERSION")) | expandGithubLinks "my-org/my-base-chart" }}' \
    --pr-body-update-operation "prepend" \
    --strategy "append"

We can use the following GitHub Code search type and query to discover the repositories that are using the my-base-chart Helm chart:

searchtype=code,query=org:my-org filename:Chart.yaml path:charts my-base-chart

By default searchtype is equal to repositories, because here we need to use the Github Code Search query, we have to put it to code. With Github Code Search query, developers don’t need to do anything to enable automatic update of their Helm base chart.

We’re using the Helm updater to update all the Helm charts, and set the version of their dependency on my-base-chart to the new release version.

Everything else is similar to the promotion of a new application release.

Result

This is the screenshot of the bottom of the library PR which is at the origin of the promotion - and which is referenced in the library’s release notes.

You can see that multiple promotion pull requests have been created, one per repository matching the GitHub search query. This is an easy way to see which application repositories have upgraded to the new version of the library.

Updating certificates

One of the questions you will have to answer when doing gitops is: “where do I stop?”. What do you store in your environment git repository? For example, should you store your certificates there too, or do you consider them “outside” of the gitops-scope, and so managed by something else, such as cert-manager?

Let’s say we want to manage them with our gitops process. It means that we’ll need to:

  • store them in a git repository. It’s not a problem for the certificate itself, but we’ll need to ensure that the private key won’t be stored in cleartext.
  • update the git repository every time a new certificate is issued - ideally by automatically creating a Pull Request.

In a Kubernetes environment where you have multiple clusters - for example in different regions of the world, different environments, etc - we can setup and use cert-manager to manage the certificates from a “central” cluster. No need to setup cert-manager in all your clusters. And we’ll use Octopilot to “propagate” the certificates to all the clusters, through a gitops workflow, by storing and updating the certificates in one or more git repositories.

You can setup a nightly CronJob, a scheduled pipeline, or anything else you prefer to regularly call Octopilot, to make sure that all the certificates stored in the git repositories are up-to-date - or to create Pull Requests to update them.

In fact, we won’t call Octopilot directly, we’ll call a script that will perform a few operations before executing Octopilot:

  • first, we’ll need to retrieve all the certificates from the Kubernetes API - using something like kubectl -n cert-manager get certificates.cert-manager.io
  • then, for each certificate, we’ll need to retrieve its associated Kubernetes Secret, which contains the actual certificate and its private key - both base64-encoded - using something like kubectl -n cert-manager get secret $secretName -o go-template='{{index .data "tls.crt"}}' > tls.crt.base64
  • optionally, we can extract some data from the certificates, such as the DNS names, validity dates, and so on - to generate nice commit messages and Pull requests. Use the openssl tool to extract the startdate and enddate fields for example.
  • and then we can execute Octopilot with something like the following:
$ octopilot \
    --repo "owner/prod-env" \
    --update "yaml(file=${APP_NAME}-values.yaml,path=tls.certificate)=file(path=tls.crt.base64)" \
    --update "sops(file=${APP_NAME}-secrets.yaml,key=tls.certificateKey)=file(path=tls.key.base64)" \
    --git-commit-title "Update ${APP_NAME} certificate" \
    --git-commit-body "..." \
    --git-branch-prefix "octopilot-cert-${APP_NAME}-" \
    --pr-labels "update-cert-${APP_NAME}"

Here, we are running 2 updaters:

  • the YAML updater, to update the Helm values file of our application, and set the tls.certificate value to the content of a tls.cert.base64 file
  • the SOPS updater, to update the Helm secrets file of our application - which is in fact a sops-encrypted YAML file - and set the tls.certificateKey value to the content of a tls.key.base64 file

We’re using the default reset strategy, which means that if we don’t merge the PR right away, and your CronJob runs a second time, it will find the existing PR and just reset it from the base branch, and then force push the commit. So you’ll always only see 1 commit, rebased every day, with the latest certificate. We’re using a specific label for our Pull Request: octopilot-cert-${APP_NAME} - to make sure we’ll find any existing PR for our application/certificate, and that each application/certificate will get its own PR.

Result

This is a screenshot of a Pull Request on an environment git repository, which updates a specific certificate.

You can see that we extracted a few information, to make it easy for reviewers - because the diff is just a base64 blob replaced by another one:

  • the creation date of the new certificate - in fact this is the creation timestamp of the latest cert-manager order, which means the latest time the certificate has been renewed
  • the new validity dates, extracted using openssl x509 -in tls.crt -noout -startdate for example
  • the validity dates of the currently deployed certificate, retrieved using openssl s_client -servername $domain -connect $domain:443 > previous.crt for example, and then extracted using openssl once again
  • the list of domains - or DNS names - for which this certificate is valid, extracted using kubectl -n cert-manager get $cert -o go-template={{.spec.dnsNames}} for example

Merging this Pull Request would result in a re-deployment of our application, with an updated certificate. So with this approach we can control when we want to deploy new certificates.

Updating Go dependencies

One of the downside of using a micro-services architecture is that it requires a lot of maintenance on the different git repositories. Even more when you have more repositories than teams or developers. For example, you’ll need to ensure that:

  • the dependencies are up to date
  • the “config files” for your Continuous Delivery pipeline(s) are up to date
  • and so on…

Automating the creation of Pull Requests to keep your git repositories “in sync” is a good way to reduce the maintenance effort. Dependabot is one way to do it, Octopilot is another. The benefits of using Octopilot are:

  • you control where it is executed - in your own infrastructure
  • you control exactly what it does - including running your own custom scripts/binaries

One use-case we have is to update the Go dependencies, by running the following commands:

  • go get -d -t -u
  • go mod tidy
  • go mod vendor
  • go mod verify

For example you can do it on octopilot’s own repository, by running:

$ export GITHUB_TOKEN=<your_github_token>
$ octopilot \
    --repo "dailymotion-oss/octopilot" \
    --update "exec(cmd=go,args=get -d -t -u)" \
    --update "exec(cmd=go,args=mod tidy)" \
    --update "exec(cmd=go,args=mod vendor)" \
    --update "exec(cmd=go,args=mod verify)" \
    --git-stage-pattern "vendor" \
    --git-commit-title "chore(deps): update Go dependencies"

You can then run it on a regular basis, such as every week, to update all your dependencies at once. And you can use the dynamic repositories feature to do it on all your repositories with a specific GitHub topic.

Result

This is a screenshot of a Pull Request on a git repository, which updates all the Go modules.

Using the exec updater we can run any command we want, capture its stdout and/or stderr, and use it in the commit message and/or Pull Request description.

Previsualizing changes

If you’re working on your workflow with Octopilot, at some point you might want to “preview” the changes Octopilot will do, without actually creating the Pull Request(s).

There is an easy way to do that. You’ll need to:

  • use the --dry-run CLI flag, to ensure that no operation will be performed on the remote git repository: not pushing branches/commits, not creating/updating Pull Requests, …
  • use the --keep-files CLI flag, to ensure that the changes to the local cloned git repositories won’t be lost at the end of the process - because by default Octopilot removes all temporary files
  • use a verbose log level - such as debug or trace - with the --log-level CLI flag, to retrieve the path of the temporary files created by Octopilot

For example, if you run:

$ export GITHUB_TOKEN=<your_github_token>
$ octopilot \
    --repo "dailymotion-oss/octopilot" \
    --update "yq(file=.goreleaser.yml,expression='(.dockers[] | select(.dockerfile == \"Dockerfile.goreleaser\") | .dockerfile) = \"a.new.Dockerfile\"')" \
    --update "yaml(file=.golangci.yml,path=run.timeout)=42m" \
    --update "regex(file=README.md,pattern='(?ms)(.*)')=replacing the content of the README.md file with this new content" \
    --dry-run --keep-files --log-level=debug

Then you should see something like:

DEBU[0000] Updaters ready                                updaters="[YQ[file=.goreleaser.yml,expression=(.dockers[] | select(.dockerfile == \"Dockerfile.goreleaser\") | .dockerfile) = \"a.new.Dockerfile\",output=,indent=2] YAML[path=run.timeout,file=.golangci.yml,style=,create=false,trim=false,indent=2] Regex[pattern=(?ms)(.*),file=README.md]]"
DEBU[0000] Repositories ready                            repositories="[{dailymotion-oss octopilot map[]}]"
DEBU[0000] Using 'reset' strategy                        repository=dailymotion-oss/octopilot
DEBU[0003] Git repository cloned                         git-reference=HEAD git-url="https://github.com/dailymotion-oss/octopilot.git" local-path=/var/folders/v0/fx5l3skn17785d8f4l883m6w0000gp/T/octopilot092369223/dailymotion-oss/octopilot
DEBU[0003] No existing Pull Request found                labels="[octopilot-update]" repository=dailymotion-oss/octopilot
DEBU[0004] Switched Git branch                           branch=octopilot-c3qif432dnc2961tuuh0 repository-name=octopilot
DEBU[0004] Updater finished                              changes=true repository=dailymotion-oss/octopilot updater="YQ[file=.goreleaser.yml,expression=(.dockers[] | select(.dockerfile == \"Dockerfile.goreleaser\") | .dockerfile) = \"a.new.Dockerfile\",output=,indent=2]"
DEBU[0004] Updater finished                              changes=true repository=dailymotion-oss/octopilot updater="YAML[path=run.timeout,file=.golangci.yml,style=,create=false,trim=false,indent=2]"
DEBU[0004] Updater finished                              changes=true repository=dailymotion-oss/octopilot updater="Regex[pattern=(?ms)(.*),file=README.md]"
DEBU[0004] All updaters finished                         repository=dailymotion-oss/octopilot
DEBU[0005] Git status                                    repository-name=octopilot status=" M README.md\n M .golangci.yml\n M .goreleaser.yml\n"
DEBU[0006] Git commit                                    commit=d263de874faf26a6ccc8bb2325bc4eb47e0a7029 repository-name=octopilot
WARN[0006] Running in dry-run mode, not pushing changes  repository=dailymotion-oss/octopilot
WARN[0006] Repository update has no changes              repository=dailymotion-oss/octopilot
INFO[0006] Updates finished                              repositories-count=1

The 2 interesting lines are:

  • Git repository cloned local-path=/var/folders/v0/fx5l3skn17785d8f4l883m6w0000gp/T/octopilot092369223/dailymotion-oss/octopilot
  • Git commit commit=d263de874faf26a6ccc8bb2325bc4eb47e0a7029

If you go to the directory identified by the local-path value, you can inspect the git repository - for example:

$ cd /var/folders/v0/fx5l3skn17785d8f4l883m6w0000gp/T/octopilot092369223/dailymotion-oss/octopilot
$ git show d263de874faf26a6ccc8bb2325bc4eb47e0a7029

and you should see:

commit d263de874faf26a6ccc8bb2325bc4eb47e0a7029 (HEAD -> octopilot-c3qif432dnc2961tuuh0)
Author: author <author@example.com>
Date:   Mon Jul 19 09:19:46 2021 +0200

    Octopilot update

    Updates:

    ### YQ[file=.goreleaser.yml,expression=(.dockers[] | select(.dockerfile == "Dockerfile.goreleaser") | .dockerfile) = "a.new.Dockerfile",output=,indent=2]
    Update .goreleaser.yml
    Updating file(s) `.goreleaser.yml` using yq expression `(.dockers[] | select(.dockerfile == "Dockerfile.goreleaser") | .dockerfile) = "a.new.Dockerfile"`

    ### YAML[path=run.timeout,file=.golangci.yml,style=,create=false,trim=false,indent=2]
    Update .golangci.yml
    Updating path `run.timeout` in file(s) `.golangci.yml`

    ### Regex[pattern=(?ms)(.*),file=README.md]
    Update README.md
    Updating file(s) `README.md` using pattern `(?ms)(.*)`

    --
    Generated by [Octopilot](https://github.com/dailymotion-oss/octopilot) [v0.2.16](https://github.com/dailymotion-oss/octopilot/releases/tag/v0.2.16) from https://github.com/dailymotion-oss/octopilot

diff --git a/.golangci.yml b/.golangci.yml
index 9a84d2c..b8a04d9 100644
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -1,4 +1,4 @@
 # See https://golangci-lint.run/usage/configuration/#config-file

 run:
-  timeout: 3m
+  timeout: 42m
diff --git a/.goreleaser.yml b/.goreleaser.yml
index 3d8734f..1b96958 100644
--- a/.goreleaser.yml
+++ b/.goreleaser.yml
@@ -12,12 +12,10 @@ builds:
       - -X main.buildDate={{.Date}}
     env:
       - CGO_ENABLED=0
-
 archives:
   - format: binary
-
 dockers:
-  - dockerfile: Dockerfile.goreleaser
+  - dockerfile: a.new.Dockerfile
     image_templates:
       - "ghcr.io/dailymotion-oss/{{.ProjectName}}:{{ .Version }}"
       - "ghcr.io/dailymotion-oss/{{.ProjectName}}:{{ .Tag }}"
@@ -31,6 +29,5 @@ dockers:
       - "--label=org.opencontainers.image.revision={{.FullCommit}}"
       - "--label=org.opencontainers.image.version={{.Version}}"
       - "--label=org.opencontainers.image.source={{.GitURL}}"
-
 changelog:
   sort: asc
diff --git a/README.md b/README.md
index a1dbefa..b247a46 100644
--- a/README.md
+++ b/README.md
@@ -1,115 +1 @@
-# Octo Pilot
-...
+replacing the content of the README.md file with this new content

This is a good way to ensure that you have the right syntax for your updater(s), and/or that your git commit is what you want.

Detecting errors in scripts

By default Octopilot does not exit with error even when updates fail. Add the --fail-on-error flag so that Octopilot exits with error when updates fail.

Handle maximum number of repositories in parallel

By default Octopilot handles all repositories in parallel - creating as many goroutines as there are repositories. Add the --max-concurrent-repos flag so that Octopilot handles them in a batch way to avoid issues such as github rate limiting, or high load on your CI platform.