Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

devfile integration tests should run against Kubernetes cluster #2752

Closed
kadel opened this issue Mar 24, 2020 · 18 comments · Fixed by #2884
Closed

devfile integration tests should run against Kubernetes cluster #2752

kadel opened this issue Mar 24, 2020 · 18 comments · Fixed by #2884
Assignees
@prietyc123
Labels
area/devfile-spec Issues or PRs related to the Devfile specification and how odo handles and interprets it. area/testing Issues or PRs related to testing, Quality Assurance or Quality Engineering estimated-size/XL (40-60) Rough sizing for Epics. About 3 sprints of work for a person priority/High Important issue; should be worked on before any other issues (except priority/Critical issue(s)).

Comments

@kadel
Copy link
Member

kadel commented Mar 24, 2020

Everything related to devfiles should work with Kubernetes as well as OpenShift clusters.

Integration tests need to be executed against Kubernetes as well as OpenShift cluster

/area devfile
/kind test
/area testing
/priority high
@openshift-ci-robot openshift-ci-robot added area/devfile-spec Issues or PRs related to the Devfile specification and how odo handles and interprets it. kind/test area/testing Issues or PRs related to testing, Quality Assurance or Quality Engineering priority/High Important issue; should be worked on before any other issues (except priority/Critical issue(s)). labels Mar 24, 2020
@kadel kadel mentioned this issue Mar 24, 2020
Merged
@johnmcollier
Copy link
Member

johnmcollier commented Mar 24, 2020 •
edited
Loading

@kadel I've been thinking about the same. As both a stopgap and the most free/easiest solution, I was thinking about bringing up Minikube in Travis, and running the devfile tests against it.

@kadel kadel mentioned this issue Mar 27, 2020
Merged
@kadel
Copy link
Member Author

kadel commented Mar 27, 2020

@kadel I've been thinking about the same. As both a stopgap and the most free/easiest solution, I was thinking about bringing up Minikube in Travis, and running the devfile tests against it.

+1
minikube start --vm-driver=none should work well on Travis.

@kadel
Copy link
Member Author

kadel commented Mar 27, 2020 •
edited
Loading

The tricky part will be nicely structuring our tests as we want to keep testing both Kubernetes and OpenShift.

We have 2 groups of tests:

  • tests that need to be executed with both Kubernetes and OpenShift (everything around devfile, except when Route is involved)
  • tests that can be executed only with OpenShift cluster (legacy s2i components, and devfile component when Route is created)

@girishramnani girishramnani added triage/needs-information Indicates an issue needs more information in order to work on it. and removed triage/ready labels Apr 1, 2020
@prietyc123
Copy link
Contributor

We have raised a concern in test-platform whether we can get a kubernetes cluster using openshift CI/prow. However I have gone through some of the documents to setup the kubernetes cluster on travis CI.
There are certain configuration w.r.t Kubernetes which needs to be taken care like writing some bash script to start minikube for example : https://github.com/bitnami/kubernetes-travis/tree/minimal-minikube-branch/scripts .
As part of running test script is concerned, tests may need some restructure w.r.t to oc and kubectl commands, it can be handled smartly through env var IMO. @mohammedzee1000 @amitkrout

@girishramnani
Copy link
Contributor

I would suggest to start on travis and not consider having a kubernetes cluster on openshift.

@girishramnani girishramnani added the estimated-size/XL (40-60) Rough sizing for Epics. About 3 sprints of work for a person label Apr 2, 2020
@amitkrout
Copy link
Contributor

amitkrout commented Apr 2, 2020 •
edited
Loading

@prietyc123 refer https://minikube.sigs.k8s.io/docs/start/linux/ for how to install minikube on travis (Ubuntu VM). While installing make sure that you are using --driver=none . if travis does not support configuration for --driver=none (However there is an open issue kubernetes/minikube#2704) then use ``--driver=docker``` in your travis configuration script.

@prietyc123
Copy link
Contributor

I would suggest to start on travis and not consider having a kubernetes cluster on openshift.

Yes, As OpenShift CI does not support such functionality (Got confirmation form test platform team), so as per the above discussion the only way left behind is travis CI.

@prietyc123 prietyc123 mentioned this issue Apr 3, 2020
Merged
@prietyc123
Copy link
Contributor

prietyc123 commented Apr 14, 2020 •
edited
Loading

This was referenced Apr 15, 2020
Merged
Merged
Merged
Merged
@prietyc123
Copy link
Contributor

#2913 is blocked and issue has been covered in #2928

Issue has been shortly explained in #2913 (comment)

@prietyc123 prietyc123 added the status/blocked Denotes an issue or PR that is blocked on something (e.g., issue/PR in different repo) label Apr 20, 2020
@prietyc123
Copy link
Contributor

#2884 is also blocked. Refer comment : #2884 (comment)

@prietyc123 prietyc123 removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Apr 23, 2020
@prietyc123 prietyc123 mentioned this issue Apr 29, 2020
Merged
@prietyc123
Copy link
Contributor

prietyc123 commented Apr 29, 2020 •
edited
Loading

pr #3041 and #2913 is blocked due to #2877 (comment)

@prietyc123
Copy link
Contributor

Blocked due to issue #3055

@prietyc123 prietyc123 removed the status/blocked Denotes an issue or PR that is blocked on something (e.g., issue/PR in different repo) label May 13, 2020
@prietyc123
Copy link
Contributor

Running devfile delete integration test on kubernetes cluster using travis CI #2913 and Running devfile push integration test on kubernetes cluster using travis CI #3041 is in review. Need to update some requested change for the proposed pr.

Running devfile create integration test on kubernetes cluster using travis CI #2884 is ready to get reviewed but on hold due to #3124. Once PR gets merged I will open it for review.

@prietyc123
Copy link
Contributor

prietyc123 commented May 13, 2020 •
edited
Loading

As per the comment I am going to open a separate pr to address the comment.

@prietyc123
Copy link
Contributor

#2913 is blocked and issue has been covered in #2928

Fixes the issue #2928 with pr #3109 and unblocked #2913

@prietyc123
Copy link
Contributor

#2884 is also blocked. Refer comment : #2884 (comment)

#2884 was failing due to #2888 and gets unblocked with pr #3124

@prietyc123
Copy link
Contributor

prietyc123 commented May 18, 2020 •
edited
Loading

This issue is in queue for a long time. And I have already proposed the prs related to this issue. It took time because I was getting lots of unknown issue while executing it and tried to resolve those one by one. While execution I tried to resolve racing condition with the namespace by copying the kubeconfig file to local temporary config file. Tried the same with openshift cluster and successfully executing those on travis CI.

But here comes the twist... When I am trying to copy the kubeconfig to the temp config on prow I am facing some other challenges Like image streams are not found in that temporary config. I am not sure how much unknown challenges will come up. And at the same time I don't want team to get blocked on running their tests on kubernetes cluster, as it is equally Important. More importantly all the tests are running on kubernetes cluster so its not the failure from test side. Its particularly failing on certain platform like prow.

IMO a possible solution is Lets handle the kubeconfig thing on travis CI for now and I will open a new pr to handle the current issue with kubeconfig on prow. I am suspecting it could be an infra issue. Anyway I will be working on that issue in this sprint itself. And in this way I guess going forward every individual will be able to run their pr on minikube.

@girishramnani
Copy link
Contributor

girishramnani commented Jun 1, 2020 •
edited by prietyc123
Loading

here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prietyc123 prietyc123

Labels
area/devfile-spec Issues or PRs related to the Devfile specification and how odo handles and interprets it. area/testing Issues or PRs related to testing, Quality Assurance or Quality Engineering estimated-size/XL (40-60) Rough sizing for Epics. About 3 sprints of work for a person priority/High Important issue; should be worked on before any other issues (except priority/Critical issue(s)).
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

devfile create Interation test on kubernetes cluster using travis CI prietyc123/odo
6 participants
@kadel @girishramnani @johnmcollier @amitkrout @openshift-ci-robot @prietyc123