Skip to content

Commit

Permalink
Gradle build - some submodules are skipped
Browse files Browse the repository at this point in the history
### Motivation

In order to complete migration to Gradle we must build all the subprojects.

### Changes

- Enabled `sh` integration tests with gradle, located in `tests/scripts/src/test/bash/gradle`
- Added these modules to the build
    - `bookkeeper-http:servlet-http-server` 
    - `metadata-drivers:etcd`
    - `tests:backward-compat:*`
    - `tests:shaded:*`
    - `stream:bk-grpc-name-resolver`
- DL shading process is now performed (before it didn't build any jar)
- Groovy tests (`tests:backward-compat:*`) now are triggered by the build/tests itself; with Maven, there is a "runner" project (`tests/integration-tests-base-groovy`); in Gradle is useless so it is skipped


### Test

- Both `bin/bookkeper standalone` and `bin/bookkeper_gradle standalone` work locally
- Tests are passing locally  

Master Issue: #2849



Reviewers: Henry Saputra <[email protected]>, Prashant Kumar <None>

This closes #2850 from nicoloboschi/fix/2849/gradle and squashes the following commits:

00b49f4 [Nicolò Boschi] Fix common_gradle.sh regex
bd739fd [Nicolò Boschi] fix sh tests
43230ba [Nicolò Boschi] revert sh files. Avoid to modify maven files, create gradle versions to faciltate migration
d1f95e4 [Nicolò Boschi] fix shaded deps
bcab40d [Nicolò Boschi] fix build
5fd0341 [Nicolò Boschi] fix build
0082e0e [Nicolò Boschi] fix build
2c32ac1 [Nicolò Boschi] fixes
3bc0b26 [Nicolò Boschi] bookkeeper-server-shaded-tests
ba89132 [Nicolò Boschi] shaded tests
6d39e33 [Nicolò Boschi] sh tests
e0032bc [Nicolò Boschi] actually run arquillian groovy tests
08dcc39 [Nicolò Boschi] backwards
2361f79 [Nicolò Boschi] hierarchical-ledger-manager
8388e11 [Nicolò Boschi] current-server-old-clients
6a24344 [Nicolò Boschi] bc-non-fips
2faca01 [Nicolò Boschi] bk-grpc-name-resolver
991bc11 [Nicolò Boschi] servlet-http-server
675ef7b [Nicolò Boschi] etcd
b1d5e14 [ZhangJian He] A empty implement in EtcdLedgerManagerFactory to let the project can compile (#2845)
bd5c50b [shustsud] Add error handling to readLedgerMetadata in over-replicated ledger GC (#2844)
746f9f6 [Matteo Merli] Remove direct ZK access for Auditor (#2842)
4117200 [ZhangJian He] the compare should be >= instead of > (#2782)
14ef56f [Prashant Kumar] BookieId can not be cast to BookieSocketAddress (#2843)
e10f3fe [ZhangJian He] Forget to close preAllocator log on shutdown (#2819)
53954ca [shustsud] Add ensemble check to over-replicated ledger GC (#2813)
919fdd3 [Prashant Kumar] Issue:2840 Create bookie shellscript for gradle (#2841)
031d168 [gaozhangmin] fix-npe-when-pulsar-ZkBookieRackAffinityMapping-getBookieAddressResolver (#2788)
3dd671c [Prashant Kumar] Migrate bookkeepr-server:test to gradle run unit tests excepts org.apache.bookkeeper.bookie. org.apache.bookkeeper.client org.apache.bookkeeper.replication org.apache.bookkeeper.tls. (#2812)
f6903b8 [Jack Vanlightly] BP-44 USE metrics
a4afaa4 [Matteo Merli] Eliminate direct ZK access in ScanAndCompareGarbageCollector (#2833)
a9b576d [Yunze Xu] Release semaphore when addEntry accepts the same entries (#2832)
148bf22 [Yun Tang] Ensure to release cache during KeyValueStorageRocksDB#closec (#2821)
4dc4260 [gaozhangmin] Heap memory leak problem when ledger replication failed (#2794)
a522fa3 [Raúl Gracia] Issue 2815: Upgrade to log4j2 to get rid of CVE-2019-17571 (#2816)
0465052 [Nicolò Boschi] Upgrade httpclient from 4.5.5 to 4.5.13 (#2793)
594a056 [Raúl Gracia] Issue 2795: Bookkeeper upgrade using Bookie ID may fail due to cookie mismatch (#2796)
354cf37 [Raúl Gracia] Upgraded dependencies with CVEs (#2792)
e413c70 [Raúl Gracia] Issue 2728: Entry Log GC may get blocked when using entryLogPerLedgerEnabled option (#2779)
883231e [pradeepbn] Building bookkeeper with gradle on java11
  • Loading branch information
nicoloboschi authored Oct 29, 2021
1 parent 6e3c82c commit 98ddf81
Show file tree
Hide file tree
Showing 20 changed files with 1,125 additions and 8 deletions.
1 change: 1 addition & 0 deletions bin/common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,7 @@ add_maven_deps_to_classpath() {
# clean it up and force us create a new one.
f="${BK_HOME}/${MODULE_PATH}/target/cached_classpath.txt"
output="${BK_HOME}/${MODULE_PATH}/target/build_classpath.out"

if [ ! -f ${f} ]; then
echo "the classpath of module '${MODULE_PATH}' is not found, generating it ..." >&2
echo "see output at ${output} for the progress ..." >&2
Expand Down
9 changes: 2 additions & 7 deletions bin/common_gradle.sh
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ is_released_binary() {
find_module_jar_at() {
DIR=$1
MODULE=$2
REGEX="^${MODULE}.jar$"
REGEX="^${MODULE}[-0-9\\.]*((-[a-zA-Z]*(-[0-9]*)?)|(-SNAPSHOT))?.jar$"
if [ -d ${DIR} ]; then
cd ${DIR}
for f in *.jar; do
Expand Down Expand Up @@ -213,17 +213,12 @@ find_module_jar() {
echo "Could not find module '${MODULE_JAR}' jar." >&2
exit 1
fi
##echo ${MODULE_JAR}
echo "${MODULE_JAR}"
return
}

add_maven_deps_to_classpath() {
MODULE_PATH=$1
MVN="mvn"
if [ "$MAVEN_HOME" != "" ]; then
MVN=${MAVEN_HOME}/bin/mvn
fi

# Need to generate classpath from maven pom. This is costly so generate it
# and cache it. Save the file into our target dir so a mvn clean will get
# clean it up and force us create a new one.
Expand Down
44 changes: 44 additions & 0 deletions bookkeeper-http/servlet-http-server/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
plugins {
id 'java'
id 'com.adarshr.test-logger'
}

dependencies {
implementation project(':bookkeeper-http:http-server')
implementation depLibs.slf4j
implementation depLibs.javaxServlet
implementation depLibs.commonsIO

compileOnly depLibs.lombok
annotationProcessor depLibs.lombok

testImplementation depLibs.junit
testImplementation depLibs.jettyWebapp
}

publishing {
publications {
maven(MavenPublication) {
groupId = 'org.apache.bookkeeper.http'
artifactId = 'servlet-http-server'
}
}
}
5 changes: 5 additions & 0 deletions dependencies.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ depVersions = [
dockerJava: "3.2.5",
dropwizard: "3.2.5",
errorprone: "2.4.0",
etcd: "0.5.4",
freebuilder: "2.7.0",
googleHTTPClient: "1.34.0",
gradleTooling: "4.0.1",
Expand Down Expand Up @@ -112,6 +113,9 @@ depLibs = [
},
dockerJava: "com.github.docker-java:docker-java:${depVersions.dockerJava}",
errorprone: "com.google.errorprone:error_prone_annotations:${depVersions.errorprone}",
etcd: dependencies.create("io.etcd:jetcd-core:${depVersions.etcd}") {
exclude group: 'io.grpc'
},
freebuilder: "org.inferred:freebuilder:${depVersions.freebuilder}",
googleHTTPClient: dependencies.create("com.google.http-client:google-http-client:${depVersions.googleHTTPClient}") {
exclude group: 'com.google.code.findbugs', module: 'jsr305'
Expand Down Expand Up @@ -152,6 +156,7 @@ depLibs = [
jettyServlet: dependencies.create("org.eclipse.jetty:jetty-servlet:${depVersions.jetty}") {
exclude group: 'javax.servlet', module: 'javax.servlet-api'
},
jettyWebapp: "org.eclipse.jetty:jetty-webapp:${depVersions.jetty}",
jmock: "org.jmock:jmock:${depVersions.jmock}",
jmhCore: "org.openjdk.jmh:jmh-core:${depVersions.jmh}",
jmhGeneratorAnnprocess: "org.openjdk.jmh:jmh-generator-annprocess:${depVersions.jmh}",
Expand Down
55 changes: 55 additions & 0 deletions metadata-drivers/etcd/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
plugins {
id 'java'
id 'com.adarshr.test-logger'
}

dependencies {
implementation project(":bookkeeper-server")
implementation project(":bookkeeper-common")
implementation project(":bookkeeper-stats")
implementation depLibs.etcd
implementation depLibs.commonsConfiguration
implementation depLibs.grpc
implementation depLibs.zookeeper

compileOnly depLibs.lombok

testImplementation depLibs.arquillianCubeDocker
testImplementation depLibs.arquillianJunitStandalone
testImplementation depLibs.testcontainers
testImplementation project(path: ':bookkeeper-common', configuration: 'testArtifacts')
testImplementation project(path: ':bookkeeper-server', configuration: 'testArtifacts')
testImplementation depLibs.dockerJava
testCompileOnly depLibs.lombok

annotationProcessor depLibs.lombok
testAnnotationProcessor depLibs.lombok
}

publishing {
publications {
maven(MavenPublication) {
groupId = 'org.apache.bookkeeper.metadata.drivers'
artifactId = 'metadata-stores-etcd'
}
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
*/
package org.apache.bookkeeper.metadata.etcd;

import com.beust.jcommander.internal.Lists;
import com.google.common.collect.Lists;
import io.etcd.jetcd.Client;
import java.io.IOException;
import java.util.List;
Expand Down
17 changes: 17 additions & 0 deletions settings.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ include(':bookkeeper-benchmark',
'bookkeeper-dist-src',
'bookkeeper-dist:storage-bench',
'bookkeeper-http:http-server',
'bookkeeper-http:servlet-http-server',
'bookkeeper-http:vertx-http-server',
'bookkeeper-proto',
'bookkeeper-server',
Expand All @@ -59,6 +60,7 @@ include(':bookkeeper-benchmark',
'cpu-affinity:src:main:affinity',
'dev:release',
'microbenchmarks',
'metadata-drivers:etcd',
'shaded:distributedlog-core-shaded',
'stats:utils',
'stream:api',
Expand All @@ -69,18 +71,33 @@ include(':bookkeeper-benchmark',
'stream:distributedlog:common',
'stream:distributedlog:core',
'stream:distributedlog:protocol',
'stream:bk-grpc-name-resolver',
'stream:proto',
'stream:server',
'stream:statelib',
'stream:storage:api',
'stream:storage:impl',
'stream:tests-common',
'tests:backward-compat',
'tests:backward-compat:bc-non-fips',
'tests:backward-compat:current-server-old-clients',
'tests:backward-compat:hierarchical-ledger-manager',
'tests:backward-compat:hostname-bookieid',
'tests:backward-compat:old-cookie-new-cluster',
'tests:backward-compat:recovery-no-password',
'tests:backward-compat:upgrade',
'tests:backward-compat:upgrade-direct',
'tests:backward-compat:yahoo-custom-version',
'tests:integration-tests-base',
'tests:integration-tests-topologies',
'tests:integration-tests-utils',
'tests:integration:cluster',
'tests:integration:smoke',
'tests:integration:standalone',
'tests:scripts',
'tests:shaded:bookkeeper-server-shaded-test',
'tests:shaded:bookkeeper-server-tests-shaded-test',
'tests:shaded:distributedlog-core-shaded-test',
'tools:framework')

project(':bookkeeper-tools').projectDir = file('tools/all')
Expand Down
36 changes: 36 additions & 0 deletions shaded/bookkeeper-server-tests-shaded/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

plugins {
id 'java'
id 'com.github.johnrengelman.shadow'
id 'com.github.hierynomus.license'
}

dependencies {
implementation project(path: ':bookkeeper-server', configuration: 'testArtifacts')
}

shadowJar {
dependencies {
}
relocate 'com.google', 'org.apache.bookkeeper.shaded.com.google'
archiveBaseName.set("bookkeeper-server-tests-shaded")
archiveClassifier.set("")
}
54 changes: 54 additions & 0 deletions stream/bk-grpc-name-resolver/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

plugins {
id 'java'
id 'com.adarshr.test-logger'
}

dependencies {
implementation project(':bookkeeper-server')
implementation project(':bookkeeper-common')
implementation project(':bookkeeper-stats')
implementation project(':stream:common')
implementation depLibs.grpc
implementation depLibs.slf4j
implementation depLibs.commonsConfiguration

compileOnly depLibs.jsr305
compileOnly depLibs.lombok
annotationProcessor depLibs.lombok

testImplementation project(':stream:clients:java:base')
testImplementation project(path: ':bookkeeper-common', configuration: 'testArtifacts')
testImplementation project(path: ':bookkeeper-server', configuration: 'testArtifacts')
testImplementation depLibs.junit
testImplementation depLibs.zookeeper

testCompileOnly depLibs.lombok
testAnnotationProcessor depLibs.lombok
}

publishing {
publications {
maven(MavenPublication) {
artifactId = 'bk-grpc-name-resolver'
}
}
}
37 changes: 37 additions & 0 deletions tests/backward-compat/bc-non-fips/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

plugins {
id 'java'
id 'com.adarshr.test-logger'
}

dependencies {

testImplementation project(path: ':bookkeeper-common', configuration: 'testArtifacts')
testImplementation project(path: ':bookkeeper-server', configuration: 'testArtifacts')
testImplementation depLibs.junit
testImplementation depLibs.slf4j
testImplementation depLibs.bcpkixJdk15on
testImplementation depLibs.bcproveExtJdk15on

testCompileOnly depLibs.lombok
testAnnotationProcessor depLibs.lombok
}

43 changes: 43 additions & 0 deletions tests/backward-compat/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

plugins {
id 'groovy'
id 'com.adarshr.test-logger'
}

subprojects {
if (project.name == "bc-non-fips") {
return
}

apply plugin: 'groovy'
apply plugin: 'com.adarshr.test-logger'

dependencies {
testImplementation 'org.codehaus.groovy:groovy-all:2.4.15'
testImplementation project(":tests:integration-tests-utils")
testImplementation project(":tests:integration-tests-topologies")
testImplementation depLibs.arquillianJunitContainer
testImplementation depLibs.junit
testImplementation depLibs.slf4j
testImplementation depLibs.arquillianCubeDocker
}

}
Loading

0 comments on commit 98ddf81

Please sign in to comment.