FAQs

Table of Contents

• [General Questions](General Questions)
• [Technical Questions](Technical Questions)

General Questions

What is Gobblin?

Gobblin is a universal ingestion framework. It's goal is to pull data from any source into an arbitrary data store. One major use case for Gobblin is pulling data into Hadoop. Gobblin can pull data from file systems, SQL stores, and data that is exposed by a REST API. See the Gobblin Home page for more information.

What programming languages does Gobblin support?

Gobblin currently only supports Java 6 and up.

Does Gobblin require any external software to be installed?

The machine that Gobblin is built on must have Java installed, and the $JAVA_HOME environment variable must be set.

What Hadoop version can Gobblin run on?

Gobblin can run on both Hadoop 1.x and Hadoop 2.x. By default, Gobblin compiles against Hadoop 1.2.1, and can compiled against Hadoop 2.3.0 by running ./gradlew -PuseHadoop2 clean build.

How do I run and schedule a Gobblin job?

Check out the [Deployment](Gobblin Deployment) page for information on how to run and schedule Gobblin jobs. Check out the [Configuration](Configuration Properties Glossary) page for information on how to set proper configuration properties for a job.

How is Gobblin different from Sqoop?

Sqoop main focus bulk import and export of data from relational databases to HDFS, it lacks the ETL functionality of data cleansing, data transformation, and data quality checks that Gobblin provides. Gobblin is also capable of pulling from any data source (e.g. file systems, RDMS, REST APIs).

Technical Questions

When running on Hadoop, each map task quickly reaches 100% completion, but then stalls for a long time. Why does this happen?

Gobblin currently uses Hadoop map tasks as a container for running Gobblin tasks. Each map task runs 1 or more Gobblin workunits, and the progress of each workunit is not hooked into the progress of each map task. Even though the Hadoop job reports 100% completion, Gobblin is still doing work. See the [Gobblin Deployment](Gobblin Deployment) page for more information.

Why does Gobblin on Hadoop stall for a long time between adding files to the DistrbutedCache, and launching the actual job?

Gobblin takes all WorkUnits created by the Source class and serializes each one into a file on Hadoop. These files are read by each map task, and are deserialized into Gobblin Tasks. These Tasks are then run by the map-task. The reason the job stalls is that Gobblin is writing all these files to HDFS, which can take a while especially if there are a lot of tasks to run. See the [Gobblin Deployment](Gobblin Deployment) page for more information.