🐛 Fixing tests configuration #306

xoscar · 2022-08-16T16:09:32Z

Fixes #305.

Changes

There is an issue with some of the test configurations that weren't caught before merging the prev PR.

xoscar · 2022-08-16T16:10:49Z

@mviitane Do you mind taking a look?

mviitane · 2022-08-17T05:43:21Z

@mviitane Do you mind taking a look?

@xoscar I can still see the same problem when running with your changes.

Just to be clear, I haven't installed cypress or ava on this machine beforehand, and I think I shouldn't have to, but is that correct?

$ docker compose run frontendTests
[+] Running 13/0
 ⠿ Container redis-cart               Running                                                                                                                                                                             0.0s
 ⠿ Container jaeger                   Running                                                                                                                                                                             0.0s
 ⠿ Container otel-col                 Running                                                                                                                                                                             0.0s
 ⠿ Container product-catalog-service  Running                                                                                                                                                                             0.0s
 ⠿ Container recommendation-service   Running                                                                                                                                                                             0.0s
 ⠿ Container cart-service             Running                                                                                                                                                                             0.0s
 ⠿ Container email-service            Running                                                                                                                                                                             0.0s
 ⠿ Container shipping-service         Running                                                                                                                                                                             0.0s
 ⠿ Container ad-service               Running                                                                                                                                                                             0.0s
 ⠿ Container payment-service          Running                                                                                                                                                                             0.0s
 ⠿ Container currency-service         Running                                                                                                                                                                             0.0s
 ⠿ Container checkout-service         Running                                                                                                                                                                             0.0s
 ⠿ Container frontend                 Running                                                                                                                                                                             0.0s
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[228:0817/053441.971654:ERROR:sandbox_linux.cc(377)] InitializeSandbox() called with multiple threads in process gpu-process.
[228:0817/053442.010610:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
Your configFile is invalid: /cypress/cypress.config.ts

It threw an error when required, check the stack trace below:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /cypress/cypress.config.ts
    at new NodeError (node:internal/errors:371:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:87:11)
    at defaultGetFormat (node:internal/modules/esm/get_format:102:38)
    at defaultLoad (node:internal/modules/esm/load:21:14)
    at ESMLoader.load (node:internal/modules/esm/loader:359:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:280:58)
    at new ModuleJob (node:internal/modules/esm/module_job:66:26)
    at ESMLoader.#createModuleJob (node:internal/modules/esm/loader:297:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:261:34)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:337:24)
    at async importModuleDynamicallyWrapper (node:internal/vm/module:437:15)
    at async loadFile (/root/.cache/Cypress/10.3.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_require_async_child.js:106:14)
    at async EventEmitter.<anonymous> (/root/.cache/Cypress/10.3.1/Cypress/resources/app/node_modules/@packages/server/lib/plugins/child/run_require_async_child.js:116:32)
$

$ docker compose run integrationTests
[+] Running 12/0
 ⠿ Container redis-cart               Running                                                                                                                                                                             0.0s
 ⠿ Container jaeger                   Running                                                                                                                                                                             0.0s
 ⠿ Container otel-col                 Running                                                                                                                                                                             0.0s
 ⠿ Container cart-service             Running                                                                                                                                                                             0.0s
 ⠿ Container email-service            Running                                                                                                                                                                             0.0s
 ⠿ Container currency-service         Running                                                                                                                                                                             0.0s
 ⠿ Container shipping-service         Running                                                                                                                                                                             0.0s
 ⠿ Container ad-service               Running                                                                                                                                                                             0.0s
 ⠿ Container payment-service          Running                                                                                                                                                                             0.0s
 ⠿ Container product-catalog-service  Running                                                                                                                                                                             0.0s
 ⠿ Container recommendation-service   Running                                                                                                                                                                             0.0s
 ⠿ Container checkout-service         Running                                                                                                                                                                             0.0s

> [email protected] test
> ava

sh: ava: not found

xoscar · 2022-08-17T13:47:44Z

@mviitane qq, did you run the build command for both the frontend and the integration tests from the latest commit? I'm not getting those errors 🤔

mviitane · 2022-08-17T13:53:40Z

@mviitane qq, did you run the build command for both the frontend and the integration tests from the latest commit? I'm not getting those errors 🤔

I ran docker compose build --no-cache for the fix-tests branch. It should also build the integrationTests?

mviitane · 2022-08-17T14:03:35Z

@xoscar After re-building with the latest, I got the integrationTests working! However, the frontendTests still have the same problem.

docker compose build frontend --no-cache
docker compose build integrationTests --no-cache

xoscar · 2022-08-17T14:49:23Z

Hey @mviitane thanks for the feedback, I figured out what the issue was, do you mind rebuilding the frontendTests image and rerunning it again? thanks!

mviitane · 2022-08-17T16:34:10Z

@xoscar Great, I got the tests now running! The frontend tests seem to have 2 tests failing, but I'm not sure if the problem is in the tests or the frontend.

In the original issue, I also proposed to add some simple documentation to test/README.md about how to start the tests. What do you think about that?

xoscar · 2022-08-18T15:52:24Z

@mviitane do you mind taking another look, please rebuild the image and rerun it from the latest commit. I ran it multiple times and got all green

mviitane · 2022-08-18T18:27:45Z

@xoscar I got now all the frontend tests passing few times, but still at least half of the times some tests were failing. So some flakiness at least in my env.

Integration tests are passing all the time.

austinlparker

Several things...

1 - Still getting some flakiness in frontend tests.

frontend-tests  |
frontend-tests  |   Running:  Checkout.cy.ts                                                                  (1 of 3)
frontend-tests  |
frontend-tests  |
frontend-tests  |   Checkout Flow
frontend-tests  |     1) should create an order with two items
frontend-tests  |
frontend-tests  |
frontend-tests  |   0 passing (7s)
frontend-tests  |   1 failing
frontend-tests  |
frontend-tests  |   1) Checkout Flow
frontend-tests  |        should create an order with two items:
frontend-tests  |      AssertionError: Timed out retrying after 4000ms: Expected to find element: `[data-cy="cart-item-count"]`, but never found it.
frontend-tests  |       at Context.eval (http://frontend:8080/__cypress/tests?p=cypress/e2e/Checkout.cy.ts:111:158)
frontend-tests  |
frontend-tests  |
frontend-tests  |
frontend-tests  |
frontend-tests  |   (Results)
frontend-tests  |
frontend-tests  |   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
frontend-tests  |   │ Tests:        1                                                                                │
frontend-tests  |   │ Passing:      0                                                                                │
frontend-tests  |   │ Failing:      1                                                                                │
frontend-tests  |   │ Pending:      0                                                                                │
frontend-tests  |   │ Skipped:      0                                                                                │
frontend-tests  |   │ Screenshots:  1                                                                                │
frontend-tests  |   │ Video:        true                                                                             │
frontend-tests  |   │ Duration:     6 seconds                                                                        │
frontend-tests  |   │ Spec Ran:     Checkout.cy.ts                                                                   │
frontend-tests  |   └────────────────────────────────────────────────────────────────────────────────────────────────┘
frontend-tests  |
frontend-tests  |
frontend-tests  |   (Screenshots)
frontend-tests  |
frontend-tests  |   -  /app/cypress/screenshots/Checkout.cy.ts/Checkout Flow -- should create an order      (1280x720)
frontend-tests  |      with two items (failed).png
frontend-tests  |
frontend-tests  |
frontend-tests  |   (Video)
frontend-tests  |
frontend-tests  |   -  Started processing:  Compressing to 32 CRF
frontend-tests  |   -  Finished processing: /app/cypress/videos/Checkout.cy.ts.mp4                          (1 second)

Didn't see any errors in integration tests, however.

2 - Not clear how these tests should be run, there's no associated make target for them or compose target that I could find? I ran them by doing a clean build + docker compose up, then running docker compose up frontendTests and docker compose up integrationTests.

…to fix-tests

xoscar · 2022-08-25T17:43:55Z

Hey @austinlparker & @mviitane sorry that it took me a while to come back to this PR. With the latest commit, the frontend E2E tests issues should be fixed and I also added an entry to the main Readme file around the test execution and such. Please take a look at let me know what you think. Thanks!

julianocosta89 · 2022-08-25T20:10:08Z

Hello @xoscar,

I'm using Ubuntu 22.04 and getting 2 failing tests. Both because there is no item in the cart.

Here are the screenshots generated by cypress:

When running the make run-tests, I'm also getting those errors:

libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[196:0825/200230.341818:ERROR:sandbox_linux.cc(377)] InitializeSandbox() called with multiple threads in process gpu-process.
[196:0825/200230.348329:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
Missing baseUrl in compilerOptions. tsconfig-paths will be skipped

Maybe those are responsible for the failure.

austinlparker · 2022-08-29T15:47:34Z

frontend tests passing, integration tests failing, ubuntu 22.04

Attaching to integrationTests
integrationTests  |
integrationTests  | > [email protected] test
integrationTests  | > ava
integrationTests  |
integrationTests  |
integrationTests  |   ✔ shipping: order (102ms)
integrationTests  |   ✔ shipping: quote (145ms)
integrationTests  |   ✔ shipping: empty quote (144ms)
integrationTests  |   ✔ ad: get (222ms)
integrationTests  |   ✔ currency: supported (216ms)
integrationTests  |   ✔ currency: convert (216ms)
integrationTests  |   ✔ payment: invalid credit card (192ms)
integrationTests  |   ✔ payment: amex credit card not allowed (192ms)
integrationTests  |   ✔ payment: expired credit card (193ms)
integrationTests  |   ✔ payment: valid credit card (200ms)
integrationTests  |   ✖ product: search Rejected promise returned by test
integrationTests  |   ✔ product: list (232ms)
integrationTests  |   ✖ product: get
integrationTests  |   ✔ recommendation: list products (248ms)
integrationTests  |   ✔ email: confirmation (284ms)
integrationTests  |   ✔ checkout: place order (311ms)
integrationTests  |   ✔ cart: all (348ms)
integrationTests  |   ─
integrationTests  |
integrationTests  |   product: search
integrationTests  |
integrationTests  |   test.js:267
integrationTests  |
integrationTests  |    266:   const res = await productSearch({ query: "hold" });
integrationTests  |    267:   t.is(res.results.length, 2);
integrationTests  |    268:   t.is(res.results[0].name, "Candle Holder");
integrationTests  |
integrationTests  |   Rejected promise returned by test. Reason:
integrationTests  |
integrationTests  |   TypeError {
integrationTests  |     message: 'Cannot read properties of undefined (reading \'length\')',
integrationTests  |   }
integrationTests  |
integrationTests  |   › test.js:267:20
integrationTests  |
integrationTests  |
integrationTests  |
integrationTests  |   product: get
integrationTests  |
integrationTests  |   test.js:258
integrationTests  |
integrationTests  |    257:   const res = await productGet({ id: "OLJCESPC7Z" });
integrationTests  |    258:   t.is(res.name, "Sunglasses");
integrationTests  |    259:   t.truthy(res.description);
integrationTests  |
integrationTests  |   Difference:
integrationTests  |
integrationTests  |   - 'Natiolal Park Foundation Explorascope'
integrationTests  |   + 'Sunglasses'
integrationTests  |
integrationTests  |   › test.js:258:5
integrationTests  |
integrationTests  |   ─
integrationTests  |
integrationTests  |   2 tests failed
integrationTests  | npm notice
integrationTests  | npm notice New minor version of npm available! 8.15.0 -> 8.18.0
integrationTests  | npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.18.0>
integrationTests  | npm notice Run `npm install -g [email protected]` to update!
integrationTests  | npm notice
integrationTests exited with code 1

make run-tests giving an error as well -

libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[195:0829/154620.876677:ERROR:sandbox_linux.cc(377)] InitializeSandbox() called with multiple threads in process gpu-process.
[195:0829/154620.883062:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
Missing baseUrl in compilerOptions. tsconfig-paths will be skipped

xoscar · 2022-08-30T21:02:26Z

Ok everyone, thank you for your patience. I think I finally fixed the tests. The warning at the beginning shouldn't be a problem, tests should pass either way.

For the integration tests I fixed the ones related to the catalog changes. But there is one outstanding issue regarding the C++ service not returning the right amount of supported currencies.

CC: @austinlparker @mviitane @julianocosta89

xoscar · 2022-09-28T10:09:57Z

@cartersocha sorry for the late response, I took one week off 😅 I just updated the PR skipping the flaky tests. Let me know what you think. @austinlparker

mviitane · 2022-09-29T07:03:51Z

The frontend tests seem stable now

      Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  Checkout.cy.ts                            20ms        1        -        -        1        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  Home.cy.ts                               00:03        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  ProductDetail.cy.ts                       20ms        2        -        -        2        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:03        5        2        -        3        -

The integration tests run successfully when first starting the app with
docker compose up

And then using another terminal
docker compose run integrationTests

cartersocha · 2022-09-29T16:25:27Z

@julianocosta89, @austinlparker could we get another pass of reviews here?

julianocosta89

@xoscar with the changes on the shipping service, which now gets the quote from the quoteservice, in order to get the tests to succeed we need to do what @mviitane mentioned.
1st docker compose up

And then using another terminal
docker compose run integrationTests

We need to either update the README or update the the test to start the quoteservicetogether with the test.

xoscar · 2022-10-04T20:41:09Z

@julianocosta89 @austinlparker @cartersocha done 😄 I added the quote service as a dependency of the integrationTests

Re review the update

julianocosta89 · 2022-10-07T06:47:29Z

@cartersocha I think we are good to go with this one

austinlparker · 2022-10-07T19:59:29Z

Needs to be rebased! @xoscar

reyang · 2022-10-07T20:02:09Z

Needs to be rebased! @xoscar

Curious - are folks trying to achieve "zero open PRs" today? 😆

xoscar · 2022-10-07T20:04:52Z

@reyang done :)

reyang · 2022-10-07T20:07:29Z

@reyang done :)

Wow!! I guess we might see for the 1st time that an OpenTelemetry repository reaching zero opening PR 🍾🍾🍾🥇🥇🥇

cartersocha · 2022-10-07T20:18:11Z

@xoscar one more time please? Im waiting on the merge button 😉

cartersocha · 2022-10-07T20:18:29Z

@reyang I can close my language update PR if needed lolol

cartersocha · 2022-10-07T20:18:36Z

For the day

xoscar · 2022-10-07T20:26:53Z

@cartersocha I think it should be visible now

* fixing tests configuration * adding custom docker file for FE tests * improving tests * multiple fixes and readme entry * multiple fixes and readme entry * fixing E2E tests * skipping flaky tests * adding the quote service as a dependency for the integration tests

fixing tests configuration

52b0de6

xoscar marked this pull request as ready for review August 16, 2022 16:10

xoscar requested a review from a team August 16, 2022 16:10

mic-max approved these changes Aug 16, 2022

View reviewed changes

Merge branch 'main' into fix-tests

a28c543

adding custom docker file for FE tests

45fe2b7

xoscar added 2 commits August 17, 2022 09:49

Merge branch 'main' into fix-tests

0116a8a

Merge branch 'main' into fix-tests

5e3c563

xoscar added 2 commits August 18, 2022 08:27

Merge branch 'main' into fix-tests

1151acb

improving tests

8003eab

austinlparker previously requested changes Aug 24, 2022

View reviewed changes

xoscar added 4 commits August 25, 2022 10:42

Merge branch 'main' into fix-tests

d52f8ea

Merge branch 'main' into fix-tests

d9efb35

Merge branch 'fix-tests' of github.com:kubeshop/opentelemetry-demo in…

4452a12

…to fix-tests

multiple fixes and readme entry

5352c40

multiple fixes and readme entry

c4cbc79

fixing E2E tests

473d055

xoscar closed this Sep 28, 2022

xoscar reopened this Sep 28, 2022

github-actions bot removed the Stale label Sep 29, 2022

mviitane approved these changes Sep 29, 2022

View reviewed changes

Merge branch 'main' into fix-tests

0745f82

julianocosta89 requested changes Oct 1, 2022

View reviewed changes

xoscar added 2 commits October 4, 2022 15:31

Merge branch 'main' into fix-tests

d9c2189

adding the quote service as a dependency for the integration tests

4be1e70

julianocosta89 approved these changes Oct 5, 2022

View reviewed changes

Merge branch 'main' into fix-tests

1baa6f3

cartersocha requested a review from austinlparker October 6, 2022 15:58

Merge branch 'main' into fix-tests

06d98de

Merge branch 'main' into fix-tests

b2e62a8

cartersocha merged commit 32d1ee6 into open-telemetry:main Oct 7, 2022

xoscar deleted the fix-tests branch October 8, 2022 14:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Fixing tests configuration #306

🐛 Fixing tests configuration #306

xoscar commented Aug 16, 2022

xoscar commented Aug 16, 2022

mviitane commented Aug 17, 2022

xoscar commented Aug 17, 2022

mviitane commented Aug 17, 2022

mviitane commented Aug 17, 2022

xoscar commented Aug 17, 2022

mviitane commented Aug 17, 2022

xoscar commented Aug 18, 2022

mviitane commented Aug 18, 2022

austinlparker left a comment •

edited

Loading

xoscar commented Aug 25, 2022

julianocosta89 commented Aug 25, 2022

austinlparker commented Aug 29, 2022

xoscar commented Aug 30, 2022

xoscar commented Sep 28, 2022

mviitane commented Sep 29, 2022

cartersocha commented Sep 29, 2022

julianocosta89 left a comment

xoscar commented Oct 4, 2022

julianocosta89 commented Oct 7, 2022

austinlparker commented Oct 7, 2022

reyang commented Oct 7, 2022

xoscar commented Oct 7, 2022

reyang commented Oct 7, 2022

cartersocha commented Oct 7, 2022

cartersocha commented Oct 7, 2022

cartersocha commented Oct 7, 2022

xoscar commented Oct 7, 2022

🐛 Fixing tests configuration #306

🐛 Fixing tests configuration #306

Conversation

xoscar commented Aug 16, 2022

Changes

xoscar commented Aug 16, 2022

mviitane commented Aug 17, 2022

xoscar commented Aug 17, 2022

mviitane commented Aug 17, 2022

mviitane commented Aug 17, 2022

xoscar commented Aug 17, 2022

mviitane commented Aug 17, 2022

xoscar commented Aug 18, 2022

mviitane commented Aug 18, 2022

austinlparker left a comment • edited Loading

Choose a reason for hiding this comment

xoscar commented Aug 25, 2022

julianocosta89 commented Aug 25, 2022

austinlparker commented Aug 29, 2022

xoscar commented Aug 30, 2022

xoscar commented Sep 28, 2022

mviitane commented Sep 29, 2022

cartersocha commented Sep 29, 2022

julianocosta89 left a comment

Choose a reason for hiding this comment

xoscar commented Oct 4, 2022

julianocosta89 commented Oct 7, 2022

austinlparker commented Oct 7, 2022

reyang commented Oct 7, 2022

xoscar commented Oct 7, 2022

reyang commented Oct 7, 2022

cartersocha commented Oct 7, 2022

cartersocha commented Oct 7, 2022

cartersocha commented Oct 7, 2022

xoscar commented Oct 7, 2022

austinlparker left a comment •

edited

Loading