external: register all ExternalStorage, KMS and Sink providers as supported External Connections #84228
Closed
9 of 13 tasks
Labels
A-cdc
Change Data Capture
A-disaster-recovery
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-cdc
Comments
adityamaru
added
C-enhancement
|
cc @cockroachdb/bulk-io |
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this issue
Jul 14, 2022
This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of cockroachdb#84228. Furthermore, none of the permission model outlined in the RFC cockroachdb#84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: cockroachdb#84225
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this issue
Jul 16, 2022
This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of cockroachdb#84228. Furthermore, none of the permission model outlined in the RFC cockroachdb#84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: cockroachdb#84225
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this issue
Jul 19, 2022
This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of cockroachdb#84228. Furthermore, none of the permission model outlined in the RFC cockroachdb#84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: cockroachdb#84225
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this issue
Jul 20, 2022
This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of cockroachdb#84228. Furthermore, none of the permission model outlined in the RFC cockroachdb#84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: cockroachdb#84225
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this issue
Jul 20, 2022
This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of cockroachdb#84228. Furthermore, none of the permission model outlined in the RFC cockroachdb#84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: cockroachdb#84225
craig bot
pushed a commit
that referenced
this issue
Jul 20, 2022
84310: sql,externalconn: introduce CREATE EXTERNAL CONNECTION r=miretskiy,benbardin a=adityamaru systemschema: introduce system.external_connections table This change introduces the `system.external_connections` table and its associated migration. The system table will be responsible for storing External Connection objects that represent resources that reside outside of CockroachDB, for example, a bucket being backed up into or a sink being written to by a changefeed. Informs: #84225 Release note: None sql,cli,spanconfigccl: test updates when adding system table This is a purely mechanical commit to update all the tests that need updating when introducing a new system table. Release note: None sql,externalconn: introduce CREATE EXTERNAL CONNECTION This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of #84228. Furthermore, none of the permission model outlined in the RFC #84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: #84225 Co-authored-by: Aditya Maru <[email protected]>
craig bot
pushed a commit
that referenced
this issue
Jul 21, 2022
84310: sql,externalconn: introduce CREATE EXTERNAL CONNECTION r=adityamaru a=adityamaru systemschema: introduce system.external_connections table This change introduces the `system.external_connections` table and its associated migration. The system table will be responsible for storing External Connection objects that represent resources that reside outside of CockroachDB, for example, a bucket being backed up into or a sink being written to by a changefeed. Informs: #84225 Release note: None sql,cli,spanconfigccl: test updates when adding system table This is a purely mechanical commit to update all the tests that need updating when introducing a new system table. Release note: None sql,externalconn: introduce CREATE EXTERNAL CONNECTION This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of #84228. Furthermore, none of the permission model outlined in the RFC #84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: #84225 Co-authored-by: Aditya Maru <[email protected]>
dt
pushed a commit
to dt/cockroach
that referenced
this issue
Jul 21, 2022
This change introduces the `CREATE EXTERNAL CONNECTION` syntax to CockroachDB. This statement can be used to create an External Connection object that represents an external resource. Majority of this change is introducing the required interfaces to persist an External Connection object in the `system.external_connections` table. We only register `nodelocal` as a supported External Connection to allow for end to end testing of functionality. A user can now execute: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo';` All the other providers will be registered as part of cockroachdb#84228. Furthermore, none of the permission model outlined in the RFC cockroachdb#84209 has been implemented, to minimize the scope of this change. Release note (sql change): introduce `CREATE EXTERNAL CONNECTION` syntax that can be used to create an External Connection representing a resource that resides outside of CockroachDB. The only supported resource at the moment is a `nodelocal` URI that can be represented as an External Connection object using: `CREATE EXTERNAL CONNECTION foo AS 'nodelocal://1/foo'`; Fixes: cockroachdb#84225
|
cc @cockroachdb/cdc |
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this issue
Aug 22, 2022
Informs: cockroachdb#84228 Release note (sql change): Users can now `CREATE EXTERNAL CONNECTION` to represent an `aws-kms` scheme that represents an AWS KMS resource.
craig bot
pushed a commit
that referenced
this issue
Aug 23, 2022
86186: sql: do not distribute queries with subqueries returning OIDs r=yuzefovich a=yuzefovich If a subquery results in a DOid datum, the datum will get a type annotation (because DOids are ambiguous) when serializing the render expression involving the result of the subquery. As a result, we might need to perform a cast on a remote node which might fail, thus we prohibit the distribution of the main query. Fixes: #86075. Release justification: bug fix. Release note: None 86357: colmem: improve memory accounting when memory limit is exceeded r=yuzefovich a=yuzefovich **colmem: improve memory accounting when memory limit is exceeded** This commit improves the memory accounting when memory limit is exceeded. Previously, in several operators we could run into a situation where we perform some allocations and run into a memory limit error later, which results in those allocations being unaccounted for. In some cases this is acceptable (when the query results in an error), but in others the memory error is caught and spilling to disk occurs. In the latter scenarios we would under-account, and this commit fixes most of such situations. Now, each disk-spilling operator instantiates a "limited" allocator that will grow an unlimited memory account when a memory error is encountered. The idea is that even though the denied allocations cannot be registered with the main memory account (meaning the operator has exceeded its memory limit), we still will draw from the `--max-sql-memory` pool since the allocations can be live for non-trivial amount of time. If an error occurs when growing the unlimited memory account, then that error is returned (not the original memory error) so that the disk spiller doesn't catch it. This commit audits all operators in `execplan` to use the limited allocator where appropriate. The new accounting method is only used in a handful of places which cover most of the use cases. The goal is to make this commit backportable whereas the follow-up commit will audit usages of `AdjustMemoryUsage` and will not be backported. Addresses: #64906. Fixes: #86351. Addresses: https://github.com/cockroachlabs/support/issues/1762. Release justification: bug fix. Release note: None **colmem: audit callers of AdjustMemoryUsage** This commit audits all callers of `Allocator.AdjustMemoryUsage` to use the newly-exported `AdjustMemoryUsageAfterAllocation` where applicable (meaning that if an allocation occurs before the method is called, then the new method is now used). In many cases this won't result in a change in the behavior since the allocators are not instantiated with limited memory accounts, but in some cases it is still useful. Release justification: bug fix. Release note: None 86402: externalconn,amazon: support s3 KMS in External Connecetions r=benbardin a=adityamaru Informs: #84228 Release note (sql change): Users can now `CREATE EXTERNAL CONNECTION` to represent an `aws-kms` scheme that represents an AWS KMS resource. Release justification: low risk change to new functionality to register s3 KMS as a supported External Connection 86613: streamproducer: check the job type for replication stream r=yuzefovich a=yuzefovich Previously, we would panic if the job id corresponded to a job type different from the replication stream job, and this is now fixed. Fixes: #86508. Release justification: bug fix. Release note: None Co-authored-by: Yahor Yuzefovich <[email protected]> Co-authored-by: Aditya Maru <[email protected]>
|
Closing; not going to do remaining sinks (e.g. null). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This change tracks all the providers that need to be registered as resources that can be represented by an External Connection. Every provider will need to provide its own implementation of the External Connection interface to be considered supported.
External Storage
Sink
KMS
Jira issue: CRDB-17530
Epic CRDB-15001
The text was updated successfully, but these errors were encountered: