Proposal: adoption of pipelines-as-code

[vdemeester]

TL;DR: adoption of pipelines-as-code in the tektoncd org (https://pipelinesascode.com/).

The pipelines-as-code propose an opinionated CI based on OpenShift Pipelines / Tekton. Its goal is to let you define your Tekton templates inside your source code repository and have the pipeline run and report the status of the execution when triggered by a Pull Request or a Push.

Some notably features of pipelines-as-code:

• Pull-request status support: When iterating over a Pull Request. Statuses and Control is done on GitHub.
• GitHub Checks API support to set the status of a PipelineRun including rechecks
• GitHub Pull Request and Push event support
• Pull-request “GitOps” actions through comments with /retest, /test and so on.
• Automatic Task resolution in Pipelines (local Tasks, Tekton Hub, and remote URLs) — Note: This predates tektoncd/pipelines resolvers, and will slowly but surely be replaced by built-in resolvers as users will adopt those resolvers.
• Efficient use of GitHub blobs and objects API for retrieving configurations
• Git events Filtering and support for separate pipelines for each event
• Gitlab, Bitbucket Server, Bitbucket Cloud and GitHub Webhook support.
• tkn-pac plug-in for Tekton CLI for managing pipelines-as-code repositories and bootstrapping.

It is currently shipped with OpenShift Pipelines but it is released independently. It is also working and tested on vanilla kubernetes and has nothing OpenShift specific in the code.

The proposal is to have pipelines-as-code hosted and maintained by the community, and help shape the next set of Tektoncd components.

Who will own it

• @chmouel
• @sm43
• @savitaashture
• @vdemeester
• the existing maintainers
• [any other takers]

[vdemeester]

/kind feature
/cc @afrittoli @bobcatfish @lbernick @abayer @jerop @dibyom @priyawadhwa

[abayer]

I'm all in favor!

[lbernick]

I'm very excited about this as well! I created a doc comparing our goals for workflows with pipelines as code (join tekton-dev google group to view + comment).

[bendory]

I like this approach too; as a general statement, it looks like the projects are more convergent than divergent, and the areas of divergence appear surmountable.

[lbernick]

I've opened #936 proposing adopting Pipelines as Code. Please take a look if you're interested!

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vdemeester]

@afrittoli @abayer @JeromeJu @jerop @dibyom @wlynch I would love to re-open this to discussion.
The main requirement we have today is, it's heavily use in production already, meaning, we consider the API as relatively stable, and any huge refactoring would need to happen targeting a v2 or something.

cc @chmouel, wdyt ?

One thought we had was, to setup pipelines-as-code on a cluster (maybe dogfooding or somewhere else), hook it up to a repository under tektoncd organiziton (for example tektoncd/cli) and showcase its possible usage within the org. If the experiments turns out to be a success, we could define what would be next.

[chmouel]

yes the refactoring could be quite a lot of work (ie: ~ 6 months) if we were planning a rewrite,

i'll be happy to help setup pac as demo to show how this could works on one of the tektoncd/ repos

[vdemeester]

One thought we had was, to setup pipelines-as-code on a cluster (maybe dogfooding or somewhere else), hook it up to a repository under tektoncd organiziton (for example tektoncd/cli) and showcase its possible usage within the org. If the experiments turns out to be a success, we could define what would be next.

@afrittoli @chitrangpatel what do you think of the above proposal ? We would start with tektoncd/cli as it's probably the simpler setup.

[chitrangpatel]

@vdemeester I really like it! Did you mean tektoncd/ci, not cli?

[vdemeester]

I mean tektoncd/cli, there is no such project as tektoncd/ci. My proposal here is to deploy openshift-pipelines/pipelines-as-code in dogfooding, hooking it to tektoncd/cli to showcase its usage. Then, if we agree we want to move forward with adoption, discussion where to move it 👼🏼

[chitrangpatel]

I mean tektoncd/cli, there is no such project as tektoncd/ci. My proposal here is to deploy openshift-pipelines/pipelines-as-code in dogfooding, hooking it to tektoncd/cli to showcase its usage. Then, if we agree we want to move forward with adoption, discussion where to move it 👼🏼

Understood. Makes sense to me.

[afrittoli]

In terms of Tekton adopting the project, it's ok for me, as long as the current project community is happy continue contributing to the project under the new home, abide to the Tekton CoC, CLA, etc (which I expect it won't be an issue as they are all Tekton contributors already).

Tekton has long needed a quick-path option to adoption, an optional component for people to quickly get up and running with Tekton, and PAC may provide that functionality. By joining Tekton, the project will gain access to the hosting foundation outreach channels which may help further grow its community and adopters.

The only concern I have is in term of infrastructure. Tekton has been working to reduce infrastructure spending, and before PAC is accepted I would like to make sure that this will not bring a significant increase in infra costs.

The project should publish releases to ghcr.io, like other Tekton projects. As you suggested PAC itself could be used for CI, which would require additionally running the PAC control plane in our dogfooding cluster. If that would enable us to sunset our usage of prow it would be a net win. We could switch to GitHub merging queues to sunset tide and use GitHub action based chatopts.

[chmouel]

@afrittoli Thanks for rekicking this!

Pipelines-as-Code (PAC) generally adheres to the Tekton project's best practices. For instance, we already use the repositories.tekton.dev namespace, and most of the linting tools and libraries (Knative) align with Tekton components.

While we don’t currently have a built-in merge queue, integrating with GitHub's native merge queue (or similar solutions) shouldn't present any major challenges. If issues arise, we’re committed to addressing them as needed.

About the Infrastructure:

The current Pipelines-as-Code dogfooding setup is running on:

1. A three-node OpenShift cluster.
2. One VM for GitHub Enterprise (we are getting the license from GitHub).
3. One VM for Bitbucket Server (we are paying for the license yearly).
4. The tiniest VM for running gosmee on https://webhook.pipelinesascode.com.
5. A bootstrap VM from which we reinstall the cluster (reinstalled daily).
6. The pipelinesascode.com domain, which runs on Cloudflare, and the documentation (Hugo-based with a few custom tags) is generated from Cloudflare.
7. Releases are autogenerated by Pipelines-as-Code using a release.yaml file. They are pushed to branches, and container images are built with Docker and pushed to ghcr.io.

CI Details:

The CI is entirely based on Kind and runs on GitHub Actions (GHA). It executes for every PR and includes a nightly job. We had some rate limiting due of tokens (which is not automated i mostly renew them every 3 months manually) but since we moved to GHA we don't do as much running tests on public github.

Most of our "functional" tests that is not provider specifics has been moved to Gitea/Forgejo as well, so we have quite a lot of testcases that are "self-contained" on the gitea instance deployed on CI.

Things to Consider:

1. Cluster (OpenShift → Kubernetes):

• The OpenShift cluster can be easily converted to a Kubernetes cluster since there is nothing OpenShift-specific in Pipelines-as-Code (PAC).
• Scaling will depend on the number of projects and PRs managed via PAC. The requirements should be similar to what we currently have on the Prow cluster since the PAC controller/reconciler adds minimal overhead.

2/3. Migrating VMs:

• The VMs for GitHub Enterprise and Bitbucket Server will need migration, the requirement for running them is quite large and they are mostly petted VM

4. Gosmee Service:

• The gosmee service is not strictly necessary; it’s provided as a convenience for users to quickly try PAC.
• Users could use smee.io instead.
• However, I created gosmee and webhook.pipelinesascode.com because the Node.js client had issues, and smee.io experienced multiple, sometimes long, outages.
• I suggest we keep this service for reliability.

5. Bootstrap VM:

• I assume you already have a VM for bootstrapping?

6. Documentation:

• We need to consider migrating the docs to https://tekton.dev.
• This could involve porting some custom tags to upstream Tekton and making minor adjustments to the theme. We currently use the hugo-book theme.

7. Release Process:

• I’m unsure about the current process for releasing Tekton components, but our process is straightforward:
  • A Git push of a tag triggers a PipelineRun through PAC to handle everything.
• It would be ideal for PAC to manage its releases with PAC itself, as this would provide an additional dogfooding use case.

Let me know if I missed anything!