feat: Make arrow primary interchange for online ODFV execution #4143
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: tokoko <[email protected]>
|
What if we store arrow format in the online/offline store? Is it going to be better than the current proto serialized value? |
Signed-off-by: tokoko <[email protected]>
|
@HaoXuAI that did occur to me, I compared the sizes for some sample results (very non-scientific). We would lose considerably on size (and retrieval latency as a result, I'm assuming), arrow has significantly more overhead, which is a big problem when you start using it on a per-row basis. Additionally it is intended to have O(1) retrieval for particular row index, which means that proto encoding tricks (like varints) go out the window. As for the offline store, most of the same reasons apply there as well, retrieval latency is not as important, but formats like parquet take significantly less storage size still. but i guess we could support arrow in |
That's good to know. thanks, but I guess for retrieval (get_online_feature) is just a |
|
https://github.com/adriangb/pgpq Something like this. but looks like it still needs to encode into postgresl field which doesn't give benefits anymore. |
lokeshrangineni
pushed a commit
to lokeshrangineni/feast
that referenced
this pull request
Apr 25, 2024
…-dev#4143) * rewrite online flow to use transform_arrow Signed-off-by: tokoko <[email protected]> * fix transformation server Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]>
jeremyary
added a commit
that referenced
this pull request
Apr 25, 2024
* chore: Move feast install to docker build in java it tests (#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <[email protected]> * remove commented out lines in compose file Signed-off-by: tokoko <[email protected]> * make local compose mode default Signed-off-by: tokoko <[email protected]> * limit COPY contents Signed-off-by: tokoko <[email protected]> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <[email protected]> * include pyproject.toml in dockerfile Signed-off-by: tokoko <[email protected]> * change links to depends_on Signed-off-by: tokoko <[email protected]> * try updating setup-python to v5 Signed-off-by: tokoko <[email protected]> * pin macos image to macos-12 Signed-off-by: tokoko <[email protected]> * force rerun Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * chore: Incorporate release 0.37.1 to master (#4127) * chore(release): release 0.37.1 ## [0.37.1](v0.37.0...v0.37.1) (2024-04-17) ### Bug Fixes * Pgvector patch ([#4108](#4108)) ([1a1f0b1](1a1f0b1)) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([#4112](#4112)) ([d5ded69](d5ded69)), closes [#3953](#3953) * chore: Move feast install to docker build in java it tests (#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <[email protected]> * remove commented out lines in compose file Signed-off-by: tokoko <[email protected]> * make local compose mode default Signed-off-by: tokoko <[email protected]> * limit COPY contents Signed-off-by: tokoko <[email protected]> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <[email protected]> * include pyproject.toml in dockerfile Signed-off-by: tokoko <[email protected]> * change links to depends_on Signed-off-by: tokoko <[email protected]> * try updating setup-python to v5 Signed-off-by: tokoko <[email protected]> * pin macos image to macos-12 Signed-off-by: tokoko <[email protected]> * force rerun Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: Tornike Gurgenidze <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * chore: Install python dependencies with uv in workflows (#4086) * install dependencies in unit-tests with uv Signed-off-by: tokoko <[email protected]> * install dependencies in unit-tests with uv Signed-off-by: tokoko <[email protected]> * enable caching, change linter job Signed-off-by: tokoko <[email protected]> * change local integration tests to uv Signed-off-by: tokoko <[email protected]> * change all installs to uv Signed-off-by: tokoko <[email protected]> * try adding uv cache Signed-off-by: tokoko <[email protected]> * fix lambda cache step name Signed-off-by: tokoko <[email protected]> * reenable caches for uv Signed-off-by: tokoko <[email protected]> * remove dangling cache step Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * feat: Make arrow primary interchange for online ODFV execution (#4143) * rewrite online flow to use transform_arrow Signed-off-by: tokoko <[email protected]> * fix transformation server Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * feat: Feast/IKV online store documentation (#4146) * feat: Feast/IKV online store documentation Signed-off-by: Pushkar Gupta <[email protected]> * functionality matric Signed-off-by: Pushkar Gupta <[email protected]> * more changes Signed-off-by: Pushkar Gupta <[email protected]> * mount dir Signed-off-by: Pushkar Gupta <[email protected]> --------- Signed-off-by: Pushkar Gupta <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Default value is not set in Redis connection string using environment variable (#4136) Removed documentation of Redis connection string supporting default values when using environment variables as it isn't supported Fixes #3669 Signed-off-by: Theodor Mihalache <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource (#4131) * Fix get_table_query_string method for Snowflake datasource Signed-off-by: TomSteenbergen <[email protected]> * Add quotes to table string Signed-off-by: TomSteenbergen <[email protected]> --------- Signed-off-by: TomSteenbergen <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Change checkout action back to v3 from v5 which isn't released yet (#4147) Signed-off-by: Jeremy Ary <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * changed the code the way mysql container is initialized. Trying to fix the issue - #4128 Also going to check if this change will be resolved in the github actions as well. Signed-off-by: Lokesh Rangineni <[email protected]> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <[email protected]> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Pushkar Gupta <[email protected]> Signed-off-by: Theodor Mihalache <[email protected]> Signed-off-by: TomSteenbergen <[email protected]> Signed-off-by: Jeremy Ary <[email protected]> Co-authored-by: Tornike Gurgenidze <[email protected]> Co-authored-by: Jeremy Ary <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: Pushkar Gupta <[email protected]> Co-authored-by: Theodor Mihalache <[email protected]> Co-authored-by: Tom Steenbergen <[email protected]>
lokeshrangineni
added a commit
to lokeshrangineni/feast
that referenced
this pull request
Apr 25, 2024
…dev#4140) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <[email protected]> * remove commented out lines in compose file Signed-off-by: tokoko <[email protected]> * make local compose mode default Signed-off-by: tokoko <[email protected]> * limit COPY contents Signed-off-by: tokoko <[email protected]> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <[email protected]> * include pyproject.toml in dockerfile Signed-off-by: tokoko <[email protected]> * change links to depends_on Signed-off-by: tokoko <[email protected]> * try updating setup-python to v5 Signed-off-by: tokoko <[email protected]> * pin macos image to macos-12 Signed-off-by: tokoko <[email protected]> * force rerun Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * chore: Incorporate release 0.37.1 to master (feast-dev#4127) * chore(release): release 0.37.1 ## [0.37.1](feast-dev/feast@v0.37.0...v0.37.1) (2024-04-17) ### Bug Fixes * Pgvector patch ([feast-dev#4108](feast-dev#4108)) ([1a1f0b1](feast-dev@1a1f0b1)) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([feast-dev#4112](feast-dev#4112)) ([d5ded69](feast-dev@d5ded69)), closes [feast-dev#3953](feast-dev#3953) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <[email protected]> * remove commented out lines in compose file Signed-off-by: tokoko <[email protected]> * make local compose mode default Signed-off-by: tokoko <[email protected]> * limit COPY contents Signed-off-by: tokoko <[email protected]> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <[email protected]> * include pyproject.toml in dockerfile Signed-off-by: tokoko <[email protected]> * change links to depends_on Signed-off-by: tokoko <[email protected]> * try updating setup-python to v5 Signed-off-by: tokoko <[email protected]> * pin macos image to macos-12 Signed-off-by: tokoko <[email protected]> * force rerun Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: Tornike Gurgenidze <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * chore: Install python dependencies with uv in workflows (feast-dev#4086) * install dependencies in unit-tests with uv Signed-off-by: tokoko <[email protected]> * install dependencies in unit-tests with uv Signed-off-by: tokoko <[email protected]> * enable caching, change linter job Signed-off-by: tokoko <[email protected]> * change local integration tests to uv Signed-off-by: tokoko <[email protected]> * change all installs to uv Signed-off-by: tokoko <[email protected]> * try adding uv cache Signed-off-by: tokoko <[email protected]> * fix lambda cache step name Signed-off-by: tokoko <[email protected]> * reenable caches for uv Signed-off-by: tokoko <[email protected]> * remove dangling cache step Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * feat: Make arrow primary interchange for online ODFV execution (feast-dev#4143) * rewrite online flow to use transform_arrow Signed-off-by: tokoko <[email protected]> * fix transformation server Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * feat: Feast/IKV online store documentation (feast-dev#4146) * feat: Feast/IKV online store documentation Signed-off-by: Pushkar Gupta <[email protected]> * functionality matric Signed-off-by: Pushkar Gupta <[email protected]> * more changes Signed-off-by: Pushkar Gupta <[email protected]> * mount dir Signed-off-by: Pushkar Gupta <[email protected]> --------- Signed-off-by: Pushkar Gupta <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Default value is not set in Redis connection string using environment variable (feast-dev#4136) Removed documentation of Redis connection string supporting default values when using environment variables as it isn't supported Fixes feast-dev#3669 Signed-off-by: Theodor Mihalache <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource (feast-dev#4131) * Fix get_table_query_string method for Snowflake datasource Signed-off-by: TomSteenbergen <[email protected]> * Add quotes to table string Signed-off-by: TomSteenbergen <[email protected]> --------- Signed-off-by: TomSteenbergen <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * fix: Change checkout action back to v3 from v5 which isn't released yet (feast-dev#4147) Signed-off-by: Jeremy Ary <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> * changed the code the way mysql container is initialized. Trying to fix the issue - feast-dev#4128 Also going to check if this change will be resolved in the github actions as well. Signed-off-by: Lokesh Rangineni <[email protected]> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <[email protected]> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <[email protected]> --------- Signed-off-by: tokoko <[email protected]> Signed-off-by: Lokesh Rangineni <[email protected]> Signed-off-by: Pushkar Gupta <[email protected]> Signed-off-by: Theodor Mihalache <[email protected]> Signed-off-by: TomSteenbergen <[email protected]> Signed-off-by: Jeremy Ary <[email protected]> Co-authored-by: Tornike Gurgenidze <[email protected]> Co-authored-by: Jeremy Ary <[email protected]> Co-authored-by: feast-ci-bot <[email protected]> Co-authored-by: Pushkar Gupta <[email protected]> Co-authored-by: Theodor Mihalache <[email protected]> Co-authored-by: Tom Steenbergen <[email protected]>
franciscojavierarceo
pushed a commit
that referenced
this pull request
May 24, 2024
# [0.38.0](v0.37.0...v0.38.0) (2024-05-24) ### Bug Fixes * Add vector database doc ([#4165](#4165)) ([37f36b6](37f36b6)) * Change checkout action back to v3 from v5 which isn't released yet ([#4147](#4147)) ([9523fff](9523fff)) * Change numpy version <1.25 dependency to <2 in setup.py ([#4085](#4085)) ([2ba71ff](2ba71ff)), closes [#4084](#4084) * Changed the code the way mysql container is initialized. ([#4140](#4140)) ([8b5698f](8b5698f)), closes [#4126](#4126) * Correct nightly install command, move all installs to uv ([#4164](#4164)) ([c86d594](c86d594)) * Default value is not set in Redis connection string using environment variable ([#4136](#4136)) ([95acfb4](95acfb4)), closes [#3669](#3669) * Get container host addresses from testcontainers (java) ([#4125](#4125)) ([9184dde](9184dde)) * Get rid of empty string `name_alias` during feature view projection deserialization ([#4116](#4116)) ([65056ce](65056ce)) * Helm chart `feast-feature-server`, improve Service template name ([#4161](#4161)) ([dedc164](dedc164)) * Improve the code related to on-demand-featureview. ([#4203](#4203)) ([d91d7e0](d91d7e0)) * Integration tests for async sdk method ([#4201](#4201)) ([08c44ae](08c44ae)) * Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource ([#4131](#4131)) ([c1579c7](c1579c7)) * Make sure schema is used when generating `from_expression` for Snowflake ([#4177](#4177)) ([5051da7](5051da7)) * Pass native input values to `get_online_features` from feature server ([#4117](#4117)) ([60756cb](60756cb)) * Pass region to S3 client only if set (Java) ([#4151](#4151)) ([b8087f7](b8087f7)) * Pgvector patch ([#4108](#4108)) ([ad45bb4](ad45bb4)) * Update doc ([#4153](#4153)) ([e873636](e873636)) * Update master-only benchmark bucket name due to credential update ([#4183](#4183)) ([e88f1e3](e88f1e3)) * Updating the instructions for quickstart guide. ([#4120](#4120)) ([0c30e96](0c30e96)) * Upgrading the test container so that local tests works with updated d… ([#4155](#4155)) ([93ddb11](93ddb11)) ### Features * Add a Kubernetes Operator for the Feast Feature Server ([#4145](#4145)) ([4a696dc](4a696dc)) * Add delta format to `FileSource`, add support for it in ibis/duckdb ([#4123](#4123)) ([2b6f1d0](2b6f1d0)) * Add materialization support to ibis/duckdb ([#4173](#4173)) ([369ca98](369ca98)) * Add optional private key params to Snowflake config ([#4205](#4205)) ([20f5419](20f5419)) * Add s3 remote storage export for duckdb ([#4195](#4195)) ([6a04c48](6a04c48)) * Adding DatastoreOnlineStore 'database' argument. ([#4180](#4180)) ([e739745](e739745)) * Adding get_online_features_async to feature store sdk ([#4172](#4172)) ([311efc5](311efc5)) * Adding support for dictionary writes to online store ([#4156](#4156)) ([abfac01](abfac01)) * Elasticsearch vector database ([#4188](#4188)) ([bf99640](bf99640)) * Enable other distance metrics for Vector DB and Update docs ([#4170](#4170)) ([ba9f4ef](ba9f4ef)) * Feast/IKV datetime edgecase errors ([#4211](#4211)) ([bdae562](bdae562)) * Feast/IKV documenation language changes ([#4149](#4149)) ([690a621](690a621)) * Feast/IKV online store contrib plugin integration ([#4068](#4068)) ([f2b4eb9](f2b4eb9)) * Feast/IKV online store documentation ([#4146](#4146)) ([73601e4](73601e4)) * Feast/IKV upgrade client version ([#4200](#4200)) ([0e42150](0e42150)) * Incorporate substrait ODFVs into ibis-based offline store queries ([#4102](#4102)) ([c3a102f](c3a102f)) * Isolate input-dependent calculations in `get_online_features` ([#4041](#4041)) ([2a6edea](2a6edea)) * Make arrow primary interchange for online ODFV execution ([#4143](#4143)) ([3fdb716](3fdb716)) * Move data source validation entrypoint to offline store ([#4197](#4197)) ([a17725d](a17725d)) * Upgrading python version to 3.11, adding support for 3.11 as well. ([#4159](#4159)) ([4b1634f](4b1634f)), closes [#4152](#4152) [#4114](#4114) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([#4112](#4112)) ([b66baa4](b66baa4)), closes [#3953](#3953)
franciscojavierarceo
pushed a commit
that referenced
this pull request
May 27, 2024
# [0.38.0](v0.37.0...v0.38.0) (2024-05-24) ### Bug Fixes * Add vector database doc ([#4165](#4165)) ([37f36b6](37f36b6)) * Change checkout action back to v3 from v5 which isn't released yet ([#4147](#4147)) ([9523fff](9523fff)) * Change numpy version <1.25 dependency to <2 in setup.py ([#4085](#4085)) ([2ba71ff](2ba71ff)), closes [#4084](#4084) * Changed the code the way mysql container is initialized. ([#4140](#4140)) ([8b5698f](8b5698f)), closes [#4126](#4126) * Correct nightly install command, move all installs to uv ([#4164](#4164)) ([c86d594](c86d594)) * Default value is not set in Redis connection string using environment variable ([#4136](#4136)) ([95acfb4](95acfb4)), closes [#3669](#3669) * Get container host addresses from testcontainers (java) ([#4125](#4125)) ([9184dde](9184dde)) * Get rid of empty string `name_alias` during feature view projection deserialization ([#4116](#4116)) ([65056ce](65056ce)) * Helm chart `feast-feature-server`, improve Service template name ([#4161](#4161)) ([dedc164](dedc164)) * Improve the code related to on-demand-featureview. ([#4203](#4203)) ([d91d7e0](d91d7e0)) * Integration tests for async sdk method ([#4201](#4201)) ([08c44ae](08c44ae)) * Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource ([#4131](#4131)) ([c1579c7](c1579c7)) * Make sure schema is used when generating `from_expression` for Snowflake ([#4177](#4177)) ([5051da7](5051da7)) * Pass native input values to `get_online_features` from feature server ([#4117](#4117)) ([60756cb](60756cb)) * Pass region to S3 client only if set (Java) ([#4151](#4151)) ([b8087f7](b8087f7)) * Pgvector patch ([#4108](#4108)) ([ad45bb4](ad45bb4)) * Update doc ([#4153](#4153)) ([e873636](e873636)) * Update master-only benchmark bucket name due to credential update ([#4183](#4183)) ([e88f1e3](e88f1e3)) * Updating the instructions for quickstart guide. ([#4120](#4120)) ([0c30e96](0c30e96)) * Upgrading the test container so that local tests works with updated d… ([#4155](#4155)) ([93ddb11](93ddb11)) ### Features * Add a Kubernetes Operator for the Feast Feature Server ([#4145](#4145)) ([4a696dc](4a696dc)) * Add delta format to `FileSource`, add support for it in ibis/duckdb ([#4123](#4123)) ([2b6f1d0](2b6f1d0)) * Add materialization support to ibis/duckdb ([#4173](#4173)) ([369ca98](369ca98)) * Add optional private key params to Snowflake config ([#4205](#4205)) ([20f5419](20f5419)) * Add s3 remote storage export for duckdb ([#4195](#4195)) ([6a04c48](6a04c48)) * Adding DatastoreOnlineStore 'database' argument. ([#4180](#4180)) ([e739745](e739745)) * Adding get_online_features_async to feature store sdk ([#4172](#4172)) ([311efc5](311efc5)) * Adding support for dictionary writes to online store ([#4156](#4156)) ([abfac01](abfac01)) * Elasticsearch vector database ([#4188](#4188)) ([bf99640](bf99640)) * Enable other distance metrics for Vector DB and Update docs ([#4170](#4170)) ([ba9f4ef](ba9f4ef)) * Feast/IKV datetime edgecase errors ([#4211](#4211)) ([bdae562](bdae562)) * Feast/IKV documenation language changes ([#4149](#4149)) ([690a621](690a621)) * Feast/IKV online store contrib plugin integration ([#4068](#4068)) ([f2b4eb9](f2b4eb9)) * Feast/IKV online store documentation ([#4146](#4146)) ([73601e4](73601e4)) * Feast/IKV upgrade client version ([#4200](#4200)) ([0e42150](0e42150)) * Incorporate substrait ODFVs into ibis-based offline store queries ([#4102](#4102)) ([c3a102f](c3a102f)) * Isolate input-dependent calculations in `get_online_features` ([#4041](#4041)) ([2a6edea](2a6edea)) * Make arrow primary interchange for online ODFV execution ([#4143](#4143)) ([3fdb716](3fdb716)) * Move data source validation entrypoint to offline store ([#4197](#4197)) ([a17725d](a17725d)) * Upgrading python version to 3.11, adding support for 3.11 as well. ([#4159](#4159)) ([4b1634f](4b1634f)), closes [#4152](#4152) [#4114](#4114) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([#4112](#4112)) ([b66baa4](b66baa4)), closes [#3953](#3953)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR is an online counterpart of #4083. online response object no longer needs to be converted to pandas for substrait transformations to be executed. Also cleans up some leftover unused code from
on_demand_feature_view.py.