Skip to content

Send table:import requests to api-import endpoints directly#226

Merged
tagomoris merged 9 commits into
masterfrom
send-table-import-requests-to-api-import-directly
Dec 19, 2019
Merged

Send table:import requests to api-import endpoints directly#226
tagomoris merged 9 commits into
masterfrom
send-table-import-requests-to-api-import-directly

Conversation

@tagomoris
Copy link
Copy Markdown
Contributor

Now Treasure Data is providing api-import.* endpoints only for streaming import requests.
api.* endpoints can route/process those requests now and in future, but it's better to send those requests to api-import endpoints.

@tagomoris tagomoris requested a review from nurse December 18, 2019 07:07
@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage increased (+0.2%) to 52.682% when pulling 1b9d06c on send-table-import-requests-to-api-import-directly into d6e8b5d on master.

3 similar comments
@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage increased (+0.2%) to 52.682% when pulling 1b9d06c on send-table-import-requests-to-api-import-directly into d6e8b5d on master.

@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage increased (+0.2%) to 52.682% when pulling 1b9d06c on send-table-import-requests-to-api-import-directly into d6e8b5d on master.

@coveralls
Copy link
Copy Markdown

Coverage Status

Coverage increased (+0.2%) to 52.682% when pulling 1b9d06c on send-table-import-requests-to-api-import-directly into d6e8b5d on master.

@coveralls
Copy link
Copy Markdown

coveralls commented Dec 18, 2019

Coverage Status

Coverage increased (+0.08%) to 52.558% when pulling 34c7b4c on send-table-import-requests-to-api-import-directly into d6e8b5d on master.

@tagomoris tagomoris force-pushed the send-table-import-requests-to-api-import-directly branch from 4a0d5af to bb0f0dd Compare December 18, 2019 07:57
@tagomoris tagomoris force-pushed the send-table-import-requests-to-api-import-directly branch from 2bbf98c to beab44d Compare December 18, 2019 08:23
@tagomoris
Copy link
Copy Markdown
Contributor Author

Updated the CI problems (and created #227 )

Comment thread lib/td/command/runner.rb Outdated
endpoint = e
}

op.on('-E', '--import-endpoint API_IMPORT_SERVER', "specify the URL for API Import server to use (default: https://api-import.treasuredata.com).") { |e|
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you avoid -E?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean to use different character? (for example, -n?)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a commit to use -i for "import".
@nurse How do you feel on it?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean just remove one character option.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed the commit to remove it.

Comment thread lib/td/command/runner.rb Outdated
Config.cl_endpoint = true
end
if import_endpoint
Config.endpoint = endpoint
Copy link
Copy Markdown
Contributor

@nurse nurse Dec 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Config.endpoint = endpoint
Config.import_endpoint = import_endpoint

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@tagomoris
Copy link
Copy Markdown
Contributor Author

@nurse Thanks!

@tagomoris tagomoris merged commit d4bf378 into master Dec 19, 2019
@tagomoris
Copy link
Copy Markdown
Contributor Author

I think we should update import endpoint automatically (like workflow endpoint), if users will specify --endpoint but not specify --import-endpoint, at least for our tokyo/eu sites.

@nurse nurse deleted the send-table-import-requests-to-api-import-directly branch December 25, 2019 04:55
@nurse
Copy link
Copy Markdown
Contributor

nurse commented Dec 25, 2019

I agree I wish we have a REST API which returns endpoints

@tagomoris
Copy link
Copy Markdown
Contributor Author

How about having a JSON file for lists of TD public endpoints (for sites)? That can solve parts of problems, and it's not worse than the current situation (with just one default, api.treasuredata.com).

@nurse
Copy link
Copy Markdown
Contributor

nurse commented Dec 25, 2019

Customers are already running td CLI with td.conf and --endpoint option, and I think we want them to use new import endpoint. If so, the import endpoint needs to automatically follow endpoint.

@tagomoris
Copy link
Copy Markdown
Contributor Author

Created #229

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants