Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix DeserializeStage to ensure output messages correctly contain the correct rows for each batch #2015

Merged
merged 7 commits into from
Oct 30, 2024

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Oct 29, 2024

Description

  • DeserializeStage now performs a copy of the rows needed for each batch, ensuring each output ControlMessage now has a unique MessageMeta and underlying DataFrame
  • Works-around issue where calling SlicedMessageMeta::get_mutable_info().checkout_obj() returns the entire DataFrame
  • This unfortunately means that DeserializeStage will require the GIL.
  • Remove unused make_output_message method
  • Avoid some redundant code in SlicedMessageMeta (unrelated but I was in this part of the code).

Closes #2002
Closes #2011

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@dagardner-nv dagardner-nv added bug Something isn't working non-breaking Non-breaking change DO NOT MERGE PR should not be merged; see PR for details skip-ci Optionally Skip CI for this PR labels Oct 29, 2024
@dagardner-nv dagardner-nv self-assigned this Oct 29, 2024
@dagardner-nv dagardner-nv requested a review from a team as a code owner October 29, 2024 19:05
@dagardner-nv dagardner-nv marked this pull request as draft October 29, 2024 19:05
@dagardner-nv dagardner-nv removed the skip-ci Optionally Skip CI for this PR label Oct 29, 2024
@dagardner-nv dagardner-nv changed the title Fix for calling set_meta on SlicedMeta Fix DeserializeStage to ensure output messages correctly contain the correct rows for each batch Oct 29, 2024
@dagardner-nv dagardner-nv marked this pull request as ready for review October 29, 2024 22:31
@dagardner-nv dagardner-nv requested a review from a team as a code owner October 29, 2024 22:31
@dagardner-nv dagardner-nv removed the DO NOT MERGE PR should not be merged; see PR for details label Oct 29, 2024
@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 6f89f83 into nv-morpheus:branch-24.10 Oct 30, 2024
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working non-breaking Non-breaking change
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants