Skip to content

Commit

Permalink
feat & docs: include the UNSIGNED type from the MySQL database in the…
Browse files Browse the repository at this point in the history
… supported list and change CHN to ch-ZN (#32)
  • Loading branch information
Breeze0806 authored Feb 21, 2024
1 parent 3929c88 commit 8c7035a
Show file tree
Hide file tree
Showing 28 changed files with 103 additions and 58 deletions.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ Similar data synchronization capabilities to datax, implemented in the datax pac

Since I have limited energy, everyone is welcome to submit issues to discuss go-etl, let's make progress together!

+ [Chinese Version(中文版)](README_zh-CN.md)

## Data Synchronization Tool

This data synchronization tool has the synchronization capability for the following data sources.
Expand Down Expand Up @@ -112,7 +114,7 @@ release.bat
+ The datax/plugin directory contains the documentation for various plugins.
+ The bin directory houses the data synchronization program, named datax.
+ The examples directory includes configuration files for data synchronization in different scenarios.
+ README_USER.md is the user manual or guide.
+ README_USER is the user manual or guide in English.

## Module Introduction
### datax
Expand All @@ -133,7 +135,7 @@ This architecture allows for flexibility and scalability, as new data sources an
For detailed information, please refer to the [go-etl Data Synchronization Developer Documentation](datax/README.md). This documentation provides guidance on how to use the go-etl framework for data synchronization, including information on its architecture, plugin system, and how to develop custom Reader and Writer plugins.

### element
Currently, the data types and data type conversions in go-etl have been implemented. For more information, please refer to the [go-etl Data Type Descriptions]((element\README.md)). This documentation provides details on the supported data types, their usage, and how to perform conversions between different types within the go-etl framework.
Currently, the data types and data type conversions in go-etl have been implemented. For more information, please refer to the [go-etl Data Type Descriptions](element/README.md). This documentation provides details on the supported data types, their usage, and how to perform conversions between different types within the go-etl framework.

### storage

Expand Down
2 changes: 1 addition & 1 deletion README_USER.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ You can download the 64-bit version of the binary program for Windows or Linux o

Obtain the datax binary program from the downloaded package. On Linux, as shown in the Makefile, export LD_LIBRARY_PATH=/home/ibmdb/clidriver/lib. This library can be downloaded from [ibm db2](https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli), otherwise it will not run.

Additionally, for Oracle, you need to download the corresponding 64-bit version of the ODBC dependency from [Oracle](https://www.oracle.com/database/technologies/instant-client/downloads.html). It can also be found in the shared files of **QQ Group 185188648**.
Additionally, for Oracle, you need to download the corresponding 64-bit version of the ODBC dependency from [Oracle](https://www.oracle.com/database/technologies/instant-client/downloads.html).

### 2.1 Single-Task Data Synchronization

Expand Down
16 changes: 8 additions & 8 deletions README_USER_CHN.md → README_USER_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ data -c config.json
"parameter": {
"username": "postgres",
"password": "123456",
"writeMode": "insert",README_CHN
"writeMode": "insert",
"column": ["*"],
"preSql": [],
"connection": {
Expand All @@ -87,13 +87,13 @@ data -c config.json

| 类型 | 数据源 | Reader(读) | Writer(写) | 文档 |
| ------------ | ------------------ | ------------ | ---------- | ------------------------------------------------------------ |
| 关系型数据库 | MySQL/Mariadb/Tidb ||| [](datax/plugin/reader/mysql/README_CHN.md)[](datax/plugin/writer/mysql/README_CHN.md) |
| | Postgres/Greenplum ||| [](datax/plugin/reader/postgres/README_CHN.md)[](datax/plugin/writer/postgres/README_CHN.md) |
| | DB2 LUW ||| [](datax/plugin/reader/db2/README_CHN.md)[](datax/plugin/writer/db2/README_CHN.md) |
| | SQL Server ||| [](datax/plugin/reader/sqlserver/README_CHN.md)[](datax/plugin/writer/sqlserver/README_CHN.md) |
| | Oracle ||| [](datax/plugin/reader/oracle/README_CHN.md)[](datax/plugin/writer/oracle/README_CHN.md) |
| 无结构流 | CSV ||| [](datax/plugin/reader/csv/README_CHN.md)[](datax/plugin/writer/csv/README_CHN.md) |
| | XLSX(excel) ||| [](datax/plugin/reader/xlsx/README_CHN.md)[](datax/plugin/writer/xlsx/README_CHN.md) |
| 关系型数据库 | MySQL/Mariadb/Tidb ||| [](datax/plugin/reader/mysql/README_zh-CN.md)[](datax/plugin/writer/mysql/README_zh-CN.md) |
| | Postgres/Greenplum ||| [](datax/plugin/reader/postgres/README_zh-CN.md)[](datax/plugin/writer/postgres/README_zh-CN.md) |
| | DB2 LUW ||| [](datax/plugin/reader/db2/README_zh-CN.md)[](datax/plugin/writer/db2/README_zh-CN.md) |
| | SQL Server ||| [](datax/plugin/reader/sqlserver/README_zh-CN.md)[](datax/plugin/writer/sqlserver/README_zh-CN.md) |
| | Oracle ||| [](datax/plugin/reader/oracle/README_zh-CN.md)[](datax/plugin/writer/oracle/README_zh-CN.md) |
| 无结构流 | CSV ||| [](datax/plugin/reader/csv/README_zh-CN.md)[](datax/plugin/writer/csv/README_zh-CN.md) |
| | XLSX(excel) ||| [](datax/plugin/reader/xlsx/README_zh-CN.md)[](datax/plugin/writer/xlsx/README_zh-CN.md) |

#### 2.1.2 使用示例

Expand Down
34 changes: 17 additions & 17 deletions README_CHN.md → README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,21 +24,21 @@ go-etl将提供的etl能力如下:

| 类型 | 数据源 | Reader(读) | Writer(写) | 文档 |
| ------------ | ------------------ | ------------ | ---------- | ------------------------------------------------------------ |
| 关系型数据库 | MySQL/Mariadb/Tidb ||| [](datax/plugin/reader/mysql/README_CHN.md)[](datax/plugin/writer/mysql/README_CHN.md) |
| | Postgres/Greenplum ||| [](datax/plugin/reader/postgres/README_CHN.md)[](datax/plugin/writer/postgres/README_CHN.md) |
| | DB2 LUW ||| [](datax/plugin/reader/db2/README_CHN.md)[](datax/plugin/writer/db2/README_CHN.md) |
| | SQL Server ||| [](datax/plugin/reader/sqlserver/README_CHN.md)[](datax/plugin/writer/sqlserver/README_CHN.md) |
| | Oracle ||| [](datax/plugin/reader/oracle/README_CHN.md)[](datax/plugin/writer/oracle/README_CHN.md) |
| 无结构流 | CSV ||| [](datax/plugin/reader/csv/README_CHN.md)[](datax/plugin/writer/csv/README_CHN.md) |
| | XLSX(excel) ||| [](datax/plugin/reader/xlsx/README_CHN.md)[](datax/plugin/writer/xlsx/README_CHN.md) |
| 关系型数据库 | MySQL/Mariadb/Tidb ||| [](datax/plugin/reader/mysql/README_zh-CN.md)[](datax/plugin/writer/mysql/README_zh-CN.md) |
| | Postgres/Greenplum ||| [](datax/plugin/reader/postgres/README_zh-CN.md)[](datax/plugin/writer/postgres/README_zh-CN.md) |
| | DB2 LUW ||| [](datax/plugin/reader/db2/README_zh-CN.md)[](datax/plugin/writer/db2/README_zh-CN.md) |
| | SQL Server ||| [](datax/plugin/reader/sqlserver/README_zh-CN.md)[](datax/plugin/writer/sqlserver/README_zh-CN.md) |
| | Oracle ||| [](datax/plugin/reader/oracle/README_zh-CN.md)[](datax/plugin/writer/oracle/README_zh-CN.md) |
| 无结构流 | CSV ||| [](datax/plugin/reader/csv/README_zh-CN.md)[](datax/plugin/writer/csv/README_zh-CN.md) |
| | XLSX(excel) ||| [](datax/plugin/reader/xlsx/README_zh-CN.md)[](datax/plugin/writer/xlsx/README_zh-CN.md) |

### 数据同步用户手册

使用[go-etl数据同步用户手册](README_USER_CHN.md)开始数据同步
使用[go-etl数据同步用户手册](README_USER_zh-CN.md)开始数据同步

### 数据同步开发宝典

参考[go-etl数据同步开发者文档](datax/README_CHN.md)来帮助开发
参考[go-etl数据同步开发者文档](datax/README_zh-CN.md)来帮助开发

### 数据同步工具编译

Expand Down Expand Up @@ -92,25 +92,25 @@ release.bat
#### 编译产物

```
+---datax---|---plugin---+---reader--mysql---|--README.md
+---datax---|---plugin---+---reader--mysql---|--README_zh-CN.md
| | .......
| |
| |---writer--mysql---|--README.md
| |---writer--mysql---|--README_zh-CN.md
| | .......
|
+---bin----datax
+---exampales---+---csvpostgres----config.json
| |---db2------------config.json
| | .......
|
+---README_USER.md
+---README_USER_zh-CN.md
```

+ datax/plugin下是各插件的文档
+ bin下的是数据同步程序datax
+ exampales下是各场景的数据同步的配置文档
+ README_USER.md是用户使用手册
+ README_USER_zh-CN.md是中文用户使用手册

## 模块简介
### datax
Expand All @@ -127,24 +127,24 @@ readerPlugin(reader)—> Framework(Exchanger+Transformer) ->writerPlugin(riter)
+ Writer:Writer为数据写入模块,负责不断向Framework取数据,并将数据写入到目的端。
+ Framework:Framework用于连接reader和writer,作为两者的数据传输通道,并处理缓冲,流控,并发,数据转换等核心技术问题

具体可以参考[go-etl数据同步开发者文档](datax/README_CHN.md)
具体可以参考[go-etl数据同步开发者文档](datax/README_zh-CN.md)

### element

目前已经实现了go-etl中的数据类型以及数据类型转换,可以参考[go-etl数据类型说明](element\README_CHN.md)
目前已经实现了go-etl中的数据类型以及数据类型转换,可以参考[go-etl数据类型说明](element\README_zh-CN.md)

### storage

#### database

目前已经实现了数据库的基础集成,抽象了数据库方言(Dialect)接口,具体实现可以参考[数据库存储开发者指南](storage/database/README_CHN.md)
目前已经实现了数据库的基础集成,抽象了数据库方言(Dialect)接口,具体实现可以参考[数据库存储开发者指南](storage/database/README_zh-CN.md)

#### stream

主要用于字节流的解析,如文件,消息队列,elasticsearch等,字节流格式可以是cvs,json, xml等。

##### file
主要用于文件的解析,如cvs,excel等,抽象了输入流(InputStream)和输出流(OutputStream)接口,具体实现可以参考[类二维表文件存储开发者指南](storage/stream/file/README_CHN.md)
主要用于文件的解析,如cvs,excel等,抽象了输入流(InputStream)和输出流(OutputStream)接口,具体实现可以参考[类二维表文件存储开发者指南](storage/stream/file/README_zh-CN.md)
### tools

工具集用于编译,新增许可证等
Expand Down
20 changes: 10 additions & 10 deletions datax/README_CHN.md → datax/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ go run main.go -t reader -p Mysql

##### 3.1.5.1 数据库存储

查看[数据库存储开发者指南](../storage/database/README.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口
查看[数据库存储开发者指南](../storage/database/README_zh-CN.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口

##### 3.1.5.2 数据库读取器

Expand Down Expand Up @@ -176,7 +176,7 @@ type Querier interface {

##### 3.1.6.1 二维表文件流存储

查看[二维表文件流存储开发者指南](../storage/stream/file/README.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口
查看[二维表文件流存储开发者指南](../storage/stream/file/README_zh-CN.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口

##### 3.1.6.2 文件读取器

Expand Down Expand Up @@ -270,7 +270,7 @@ go run main.go -t writer -p Mysql

##### 3.2.5.1 数据库存储

查看[数据库存储开发者指南](../storage/database/README.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口
查看[数据库存储开发者指南](../storage/database/README_zh-CN.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口

##### 3.2.5.2 数据库写入器

Expand Down Expand Up @@ -308,7 +308,7 @@ type Execer interface {

##### 3.2.6.1 二维表文件流存储

查看[二维表文件流存储开发者指南](../storage/stream/file/README.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口
查看[二维表文件流存储开发者指南](../storage/stream/file/README_zh-CN.md),不仅能帮助你更快地实现Reader插件接口,而且能帮助你更快地实现Writer插件接口

##### 3.2.6.2 文件读取器

Expand Down Expand Up @@ -468,15 +468,15 @@ go generate ./...

跟一般的`生产者-消费者`模式一样,`Reader`插件和`Writer`插件之间也是通过`channel`来实现数据的传输的。`channel`可以是内存的,也可能是持久化的,插件不必关心。插件通过`RecordSender``channel`写入数据,通过`RecordReceiver``channel`读取数据。

`channel`中的一条数据为一个`Record`的对象,`Record`中可以放多个`Column`对象,这可以简单理解为数据库中的记录和列,`Record`原型具体见[文档](../element/README.md)的《记录》一章。
`channel`中的一条数据为一个`Record`的对象,`Record`中可以放多个`Column`对象,这可以简单理解为数据库中的记录和列,`Record`原型具体见[文档](../element/README_zh-CN.md)的《记录》一章。

因为`Record`是一个接口,`Reader`插件首先调用`RecordSender.createRecord()`创建一个`Record`实例,然后把`Column`一个个添加到`Record`中。

`Writer`插件调用`RecordReceiver.getFromReader()`方法获取`Record`,然后把`Column`遍历出来,写入目标存储中。当`Reader`尚未退出,传输还在进行时,如果暂时没有数据`RecordReceiver.getFromReader()`方法会阻塞直到有数据。如果传输已经结束,会返回`ErrTerminate``Writer`插件可以据此判断是否结束`startWrite`方法。

### 6.1 数据类型转化

为了规范源端和目的端类型转换操作,保证数据不失真,go-etl支持六种内部数据类型,具体见[文档](../element/README.md)的《数据类型转化》一章。
为了规范源端和目的端类型转换操作,保证数据不失真,go-etl支持六种内部数据类型,具体见[文档](../element/README_zh-CN.md)的《数据类型转化》一章。

## 7. 插件文档

Expand Down Expand Up @@ -547,24 +547,24 @@ release.bat
### 8.3 编译产物

```
+---datax---|---plugin---+---reader--mysql---|--README.md
+---datax---|---plugin---+---reader--mysql---|--README_zh-CN.md
| | .......
| |
| |---writer--mysql---|--README.md
| |---writer--mysql---|--README_zh-CN.md
| | .......
|
+---bin----datax
+---exampales---+---csvpostgres----config.json
| |---db2------------config.json
| | .......
|
+---README_USER.md
+---README_USER_zh-CN.md
```
+ datax/plugin下是各插件的文档
+ bin下的是数据同步程序datax
+ exampales下是各场景的数据同步的配置文档
+ README_USER.md是用户使用手册
+ README_USER_zh-CN.md是用户使用手册

## 9. 调试http接口

Expand Down
File renamed without changes.
File renamed without changes.
8 changes: 1 addition & 7 deletions datax/plugin/reader/mysql/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
当然可以,请提供您需要翻译的内容。

(如果您指的是上面关于“MysqlReader插件文档”的内容,那么该段内容的英文翻译如下:)

---

# MysqlReader Plugin Documentation

## Quick Introduction
Expand Down Expand Up @@ -163,7 +157,7 @@ Below is a conversion table for MysqlReader regarding MySQL types:

| go-etl Type | MySQL Data Type |
| --- | --- |
| bigInt | int, tinyint, smallint, mediumint, bigint, year |
| bigInt | int, tinyint, smallint, mediumint, bigint, year,unsigned int, unsigned bigint, unsigned smallint, unsigned tinyint |
| decimal | float, double, decimal |
| string | varchar, char, tinytext, text, mediumtext, longtext |
| time | date, datetime, timestamp, time |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ MysqlReader通过使用dbmsreader中定义的查询流程调用go-etl自定义

| go-etl的类型 | mysql数据类型 |
| ------------ | --------------------------------------------------- |
| bigInt | int, tinyint, smallint, mediumint, bigint,year |
| bigInt | int, tinyint, smallint, mediumint, bigint,year,unsigned int, unsigned bigint, unsigned smallint, unsigned tinyint |
| decimal | float, double, decimal |
| string | varchar, char, tinytext, text, mediumtext, longtext |
| time | date, datetime, timestamp, time |
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
请将以下中文翻译成英文
# PostgresReader插件文档

## 快速介绍
Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
将以下中文翻译成英文
# CsvWriter插件文档

## 快速介绍
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion datax/plugin/writer/mysql/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ Below is a conversion table for MysqlWriter and Mysql data types:

| go-etl Type | Mysql Data Type |
| ---------- | --------------------------------------------------- |
| bigInt | int, tinyint, smallint, mediumint, bigint, year |
| bigInt | int, tinyint, smallint, mediumint, bigint, year, unsigned int, unsigned bigint, unsigned smallint, unsigned tinyint |
| decimal | float, double, decimal |
| string | varchar, char, tinytext, text, mediumtext, longtext |
| time | date, datetime, timestamp, time |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ MysqlWriter通过使用dbmswriter中定义的查询流程调用go-etl自定义

| go-etl的类型 | mysql数据类型 |
| ------------ | --------------------------------------------------- |
| bigInt | int, tinyint, smallint, mediumint,bigint,year |
| bigInt | int, tinyint, smallint, mediumint,bigint,year,unsigned int, unsigned bigint, unsigned smallint, unsigned tinyint |
| decimal | float, double, decimal |
| string | varchar, char, tinytext, text, mediumtext, longtext |
| time | date, datetime, timestamp, time |
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
1 change: 1 addition & 0 deletions storage/database/mysql/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,5 @@
// The Scanner uses BaseScanner to simplify its implementation.
// The Valuer uses the implementation approach of GoValuer.

// Package mysql implements the Dialect of mysql database, supporting the corresponding database of mysql 5.7+
package mysql
Loading

0 comments on commit 8c7035a

Please sign in to comment.