streamingccl: heartbeat persisted frontier #84286
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
from
aed7c42 to
9c2ce6f
Compare
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
2 times, most recently
from
2bca602 to
a732e87
Compare
stevendanna
reviewed
Jul 13, 2022
pkg/ccl/streamingccl/streamingest/stream_ingestion_frontier_processor.go
Outdated
Show resolved
Hide resolved
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
2 times, most recently
from
d90164a to
24b505c
Compare
miretskiy
reviewed
Jul 20, 2022
pkg/ccl/streamingccl/streamingest/stream_ingestion_processor_planning.go
Outdated
Show resolved
Hide resolved
pkg/ccl/streamingccl/streamingest/stream_ingestion_frontier_processor.go
Show resolved
Hide resolved
pkg/ccl/streamingccl/streamingest/stream_replication_e2e_test.go
Outdated
Show resolved
Hide resolved
pkg/ccl/streamingccl/streamingest/stream_replication_e2e_test.go
Outdated
Show resolved
Hide resolved
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
2 times, most recently
from
4956065 to
3847dbc
Compare
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
from
3847dbc to
ba3994a
Compare
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
from
ba3994a to
77baf46
Compare
gh-casper
approved these changes
Jul 28, 2022
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
from
0afb75c to
4cc4cf4
Compare
|
bors r+ |
|
Build failed (retrying...): |
|
Has merge conflict. bors r- |
|
Canceled. |
Previously we would forward the ingestion frontier's resolved timestamp at the point of heartbeat, however this would result in the chance of the protected timestamp record of the producer to exceed that of the ingestion job's persisted frontier. This PR uses the last persisted frontier value instead. It also moves the highwater updating to the frontier processor. Release note (bug fix): The protected timestamp of the producer job is no longer able to exceed the persisted ingestion frontier.
samiskin
force-pushed
the
producer-pts-behind-checkpoint
branch
from
4cc4cf4 to
96149c2
Compare
|
bors r+ |
|
Build failed (retrying...): |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #84086
Previously we would forward the ingestion frontier's resolved timestamp
at the point of heartbeat, however this would result in the chance of
the protected timestamp record of the producer to exceed that of the
ingestion job's persisted frontier.
This PR uses the last persisted frontier value instead.
Release note (bug fix): The protected timestamp of the producer job is
no longer able to exceed the persisted ingestion frontier.