[Core] Add opt-in integration resource provision service #1297

erikzaadi · 2025-01-05T16:35:05Z

Description

What - Add a flow that verifies the USE_PROVISIONED_DEFAULTS and uses the provisioning resources accordingly

Logic:

Integration marked use_provisioned_defaults as true but org feature toggle USE_PROVISIONED_DEFAULTS isn't available

=> Regular flow

Org feature toggle USE_PROVISIONED_DEFAULTS enabled but integration wasn't marked with use_provisioned_defaults

=> Regular flow

Integration marked use_provisioned_defaults as true and org feature toggle USE_PROVISIONED_DEFAULTS is set

=> Create integration with the `provisionEnabled` property set to true
=> Skip creating resources
=> Wait until config is populated before continuing flow

Type of change

New feature (non-breaking change which adds functionality)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Tankilevitch

Nice work
left a few comments, overall seems like you've added some core functionality to the integration initialization which I think is really worth adding testing for to make sure everything is behaving as expected

.github/workflows/integrations-test.yml

port_ocean/clients/port/mixins/organization.py

port_ocean/core/defaults/initialize.py

Tankilevitch · 2025-01-06T17:26:17Z

port_ocean/clients/port/mixins/integrations.py

+    async def _poll_integration_until_default_provisioning_is_complete(
+        self,
+        interval=15,


lets not use magic numbers

can we do some kind of backoff or eventually mechanism? 15 seconds is huge time to wait for integration to finish set up

good idea, will add

port_ocean/clients/port/mixins/integrations.py

Tankilevitch · 2025-01-06T17:30:02Z

port_ocean/config/settings.py

@@ -68,6 +68,7 @@ class IntegrationConfiguration(BaseOceanSettings, extra=Extra.allow):
    initialize_port_resources: bool = True
    scheduled_resync_interval: int | None = None
    client_timeout: int = 60
+    use_provisioned_defaults: bool = False


are we sure we want to use only two options False and True ? maybe worth changing the variable to provisioning_mode

port

local
which will allow us more flexability in the future?

WDYM? Why would that be different?

port_ocean/core/defaults/initialize.py

port_ocean/clients/port/mixins/integrations.py

Tankilevitch · 2025-01-07T11:15:54Z

port_ocean/clients/port/mixins/integrations.py

+                f"Fetching created integration and validating config, attempt {attempts+1}/{INTEGRATION_POLLING_RETRY_LIMIT}"
+            )
+            response = await self._get_current_integration()
+            response_json = response.json()


Suggested change

response_json = response.json()

integration_response = response.json()

Tankilevitch · 2025-01-07T11:17:17Z

port_ocean/clients/port/mixins/integrations.py

+            if config != {}:
+                return response_json
+


Suggested change

if config != {}:

return response_json

if config:

return response_json

Tankilevitch · 2025-01-07T11:27:29Z

port_ocean/clients/port/mixins/integrations.py

+        if use_provisioned_defaults:
+            json["provisionEnabled"] = use_provisioned_defaults
+


this shouldn't be something on top of the integration, but rather as a query param
see the image below for reference

Tankilevitch · 2025-01-07T11:28:32Z

port_ocean/exceptions/port_defaults.py

+        super().__init__(
+            f"Failed to retrieve integration config after {retries} attempts"


does it mean that the integration won't continue to run after this issue happens

Yes, and it'll be in the logs

Co-authored-by: Tom Tankilevitch <[email protected]>

erikzaadi requested a review from a team as a code owner January 5, 2025 16:35

github-actions bot added the size/M label Jan 5, 2025

erikzaadi force-pushed the PORT-12242-onboarding-provision-feature-flag branch 9 times, most recently from a5c6b85 to 0886913 Compare January 6, 2025 11:48

github-actions bot added size/L and removed size/M labels Jan 6, 2025

erikzaadi force-pushed the PORT-12242-onboarding-provision-feature-flag branch from 802d38d to b00e679 Compare January 6, 2025 16:23

erikzaadi changed the title ~~Add opt-in integration resource provision service~~ [CORE] Add opt-in integration resource provision service Jan 6, 2025

erikzaadi changed the title ~~[CORE] Add opt-in integration resource provision service~~ [Core] Add opt-in integration resource provision service Jan 6, 2025

Tankilevitch requested changes Jan 6, 2025

View reviewed changes

erikzaadi force-pushed the PORT-12242-onboarding-provision-feature-flag branch 2 times, most recently from e3b9410 to 78cf234 Compare January 7, 2025 07:13

Tankilevitch reviewed Jan 7, 2025

View reviewed changes

erikzaadi force-pushed the PORT-12242-onboarding-provision-feature-flag branch from f3d696d to 1a40313 Compare January 8, 2025 08:41

github-actions bot added size/M and removed size/L labels Jan 8, 2025

erikzaadi force-pushed the PORT-12242-onboarding-provision-feature-flag branch 3 times, most recently from 79e3d2a to 152f355 Compare January 11, 2025 14:36

erikzaadi and others added 4 commits January 12, 2025 13:37

[CORE] Opt-in integration resource provision

0f38c65

[PR-FIX] Update port_ocean/core/defaults/initialize.py

c1dda3a

Co-authored-by: Tom Tankilevitch <[email protected]>

[PR-FIX] Unify naming and add backoff retry mechanism

99e3a2f

[PR-FIXES]

4cf9204

erikzaadi added 2 commits January 12, 2025 13:37

[LINT-FIX]

cf65613

[SQUASH] Fix test

bf41328

erikzaadi force-pushed the PORT-12242-onboarding-provision-feature-flag branch from 11f99ed to bf41328 Compare January 12, 2025 11:38

erikzaadi added 2 commits January 12, 2025 15:11

[SQUASH] Fix flow

9a065ef

[SQUASH] MOAR tests

bd841bb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Add opt-in integration resource provision service #1297

[Core] Add opt-in integration resource provision service #1297

erikzaadi commented Jan 5, 2025

Tankilevitch left a comment

Tankilevitch Jan 6, 2025

Tankilevitch Jan 6, 2025

Tankilevitch Jan 6, 2025

erikzaadi Jan 6, 2025

erikzaadi Jan 7, 2025

Tankilevitch Jan 6, 2025

erikzaadi Jan 6, 2025

Tankilevitch Jan 7, 2025

erikzaadi Jan 12, 2025

Tankilevitch Jan 7, 2025

erikzaadi Jan 12, 2025

Tankilevitch Jan 7, 2025

erikzaadi Jan 12, 2025

Tankilevitch Jan 7, 2025

erikzaadi Jan 8, 2025

	response_json = response.json()
	integration_response = response.json()

		if use_provisioned_defaults:
		json["provisionEnabled"] = use_provisioned_defaults

		super().__init__(
		f"Failed to retrieve integration config after {retries} attempts"

[Core] Add opt-in integration resource provision service #1297

Are you sure you want to change the base?

[Core] Add opt-in integration resource provision service #1297

Conversation

erikzaadi commented Jan 5, 2025

Description

Type of change

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Tankilevitch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment