Introduce GoogleDrive Fetcher for tika-pipes

[bartek]

This introduces a Google Drive fetcher into the tika-pipes project

[bartek]

@tballison Cleaned up version of #2074 now that I sorted out what's going on with the tika-grpc-3x-features branch (it will require cleanup)

[THausherr]

The pom.xml has several properties that are not used, e.g. kiota, wiremock, nimbus, etc. You probably copied them from another (older) pom.xml. I think you only need the first one.

[bartek]

The pom.xml has several properties that are not used, e.g. kiota, wiremock, nimbus, etc. You probably copied them from another (older) pom.xml. I think you only need the first one.

Thanks, I was indeed copying and did not review the pom.xml too deeply. Cleaned up: 880a2ea

[THausherr]

There are still 3 that you don't need, one that isn't used and two that are defined in the parent.

[bartek]

There are still 3 that you don't need, one that isn't used and two that are defined in the parent.

I think I got them in the final push. Is there a way to identify these during build? I didn't notice it in the output

[THausherr]

I got these by looking at the source code. This is just me, I like smaller pom.xml files that are easier to understand and maintain.

Is it possible for you to create some sort of unit test, or is this impossible because one would need some google drive access?

[bartek]

I got these by looking at the source code. This is just me, I like smaller pom.xml files that are easier to understand and maintain.

Thanks for the commits and notes. I'm not too familiar with the project and am entering through tika-pipes and its fetcher requirements, so I appreciate your patience.

Is it possible for you to create some sort of unit test, or is this impossible because one would need some google drive access?

I imagine we could mock the response from Google Drive, so at least we test happy/sad paths. Let me have a try at it.

[THausherr]

So I was able to fix the google driver fetcher pom.xml, but now the pipes gRPC server is failing with dependency convergence errors. 😬 I'll try to fix that too.

[THausherr]

I managed to do a complete build locally, mostly by moving the dependencyManagement stuff I introduced to the parent. I'll do another test locally and then add this here.

[bartek]

@THausherr Looks like you got this building. Are you happy with your changes? If so I will squash them into a single commit.

[THausherr]

Yes!

[bartek]

@THausherr Great. Btw, since these changes, I am unable to build tika-pipes (which is what I am building, not the whole project). It looks like the pom.xml that was previously expected no longer is applicable. Are you able to help?

Here's the error:

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:33 min
[INFO] Finished at: 2024-12-09T12:47:12-04:00
[INFO] ------------------------------------------------------------------------
+ mvn dependency:copy-dependencies -f /Users/bartek/workspace/tika/tika-pipes/tika-grpc/example-dockerfile/../../../tika-pipes/tika-grpc
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-readable POM /Users/bartek/workspace/tika/tika-pipes/tika-grpc/example-dockerfile/../../../tika-pipes/tika-grpc/pom.xml: /Users/bartek/workspace/tika/tika-pipes/tika-grpc/example-dockerfile/../../../tika-pipes/tika-grpc/pom.xml (No such file or directory) @ 

And here's my build script:

set -x

TAG_NAME=$1

if [ -z "${TAG_NAME}" ]; then
    echo "Single command line argument is required which will be used as the -t parameter of the docker build command"
    exit 1
fi

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
TIKA_SRC_PATH=${SCRIPT_DIR}/../../..
OUT_DIR=${TIKA_SRC_PATH}/tika-pipes/tika-grpc/target/tika-docker

mvn clean install -Dossindex.skip -DskipTests=true -Denforcer.skip=true -Dossindex.skip=true -f "${TIKA_SRC_PATH}" || exit
mvn dependency:copy-dependencies -f "${TIKA_SRC_PATH}/tika-pipes/tika-grpc" || exit
rm -rf "${OUT_DIR}"
mkdir -p "${OUT_DIR}"

project_version=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout -f "${TIKA_SRC_PATH}")

cp -r "${TIKA_SRC_PATH}/tika-pipes/tika-grpc/target/dependency" "${OUT_DIR}/libs"
cp -r "${TIKA_SRC_PATH}/tika-pipes/tika-fetchers/tika-fetcher-gcs/target/tika-fetcher-gcs-${project_version}.jar" "${OUT_DIR}/libs"
cp -r "${TIKA_SRC_PATH}/tika-pipes/tika-fetchers/tika-fetcher-az-blob/target/tika-fetcher-az-blob-${project_version}.jar" "${OUT_DIR}/libs"
cp -r "${TIKA_SRC_PATH}/tika-pipes/tika-fetchers/tika-fetcher-http/target/tika-fetcher-http-${project_version}.jar" "${OUT_DIR}/libs"
cp -r "${TIKA_SRC_PATH}/tika-pipes/tika-fetchers/tika-fetcher-microsoft-graph/target/tika-fetcher-microsoft-graph-${project_version}.jar" "${OUT_DIR}/libs"
cp -r "${TIKA_SRC_PATH}/tika-pipes/tika-fetchers/tika-fetcher-s3/target/tika-fetcher-s3-${project_version}.jar" "${OUT_DIR}/libs"

cp "${TIKA_SRC_PATH}/tika-pipes/tika-grpc/target/tika-grpc-${project_version}.jar" "${OUT_DIR}/libs"
cp "${TIKA_SRC_PATH}/tika-pipes/tika-grpc/src/test/resources/log4j2.xml" "${OUT_DIR}"
cp "${TIKA_SRC_PATH}/tika-pipes/tika-grpc/src/test/resources/tika-pipes-test-config.xml" "${OUT_DIR}/tika-config.xml"
cp "${TIKA_SRC_PATH}/tika-pipes/tika-grpc/example-dockerfile/Dockerfile" "${OUT_DIR}/Dockerfile"

cd "${OUT_DIR}" || exit

# build single arch
#docker build "${OUT_DIR}" -t "${TAG_NAME}"

# Or we can build multi-arch - https://www.docker.com/blog/multi-arch-images/
docker buildx create --name tikabuilder
# see https://askubuntu.com/questions/1339558/cant-build-dockerfile-for-arm64-due-to-libc-bin-segmentation-fault/1398147#1398147
docker run --rm --privileged tonistiigi/binfmt --install amd64
docker run --rm --privileged tonistiigi/binfmt --install arm64
docker buildx build --builder=tikabuilder "${OUT_DIR}" -t "${TAG_NAME}" --platform linux/amd64,linux/arm64 --push
docker buildx stop tikabuilder

[THausherr]

I didn't touch tika-grpc/pom.xml at all.

Your script has "tika-pipes/tika-grpc" however "tika-grpc" is at the top level.

[bartek]

@THausherr Thanks, I sorted that out. Looks like my paths are based on tika-grpc-3x-features branch paths.

[nddipiazza]

I am porting this into https://github.com/nddipiazza/tika-pipes
starting now.