Send a Hadoop Job via gcloud
0
This is my current Hadoop job.
java -cp `hadoop classpath`:/usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
How to convert it to a gcloud dataproc jobs submit hadoop job?
I tried:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar --class com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I keep getting:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) argument --class: Exactly one of (--class | --jar) must be specified.
Usage: gcloud dataproc jobs submit hadoop --cluster=CLUSTER (--class=MAIN_CLASS | --jar=MAIN_JAR) [optional flags] [-- JOB_ARGS ...]
optional flags may be --archives | --async | --bucket | --class |
--driver-log-levels | --files | --help | --jar |
--jars | --labels | --max-failures-per-hour |
--properties | --region
For detailed information on this command and its flags, run:
gcloud dataproc jobs submit hadoop --help
If I pass:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I get:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) unrecognized arguments: com.linkedin.tony.cli.ClusterSubmitter
Reference here.
python hadoop
google-cloud-platform google-cloud-dataproc asked Nov 12 '18 at 6:18
spicyramen
3,18523369
add a comment |
0
This is my current Hadoop job.
java -cp `hadoop classpath`:/usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
How to convert it to a gcloud dataproc jobs submit hadoop job?
I tried:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar --class com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I keep getting:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) argument --class: Exactly one of (--class | --jar) must be specified.
Usage: gcloud dataproc jobs submit hadoop --cluster=CLUSTER (--class=MAIN_CLASS | --jar=MAIN_JAR) [optional flags] [-- JOB_ARGS ...]
optional flags may be --archives | --async | --bucket | --class |
--driver-log-levels | --files | --help | --jar |
--jars | --labels | --max-failures-per-hour |
--properties | --region
For detailed information on this command and its flags, run:
gcloud dataproc jobs submit hadoop --help
If I pass:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I get:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) unrecognized arguments: com.linkedin.tony.cli.ClusterSubmitter
Reference here.
python hadoop
google-cloud-platform google-cloud-dataproc asked Nov 12 '18 at 6:18
spicyramen
3,18523369
add a comment |
0
0
0
This is my current Hadoop job.
java -cp `hadoop classpath`:/usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
How to convert it to a gcloud dataproc jobs submit hadoop job?
I tried:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar --class com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I keep getting:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) argument --class: Exactly one of (--class | --jar) must be specified.
Usage: gcloud dataproc jobs submit hadoop --cluster=CLUSTER (--class=MAIN_CLASS | --jar=MAIN_JAR) [optional flags] [-- JOB_ARGS ...]
optional flags may be --archives | --async | --bucket | --class |
--driver-log-levels | --files | --help | --jar |
--jars | --labels | --max-failures-per-hour |
--properties | --region
For detailed information on this command and its flags, run:
gcloud dataproc jobs submit hadoop --help
If I pass:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I get:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) unrecognized arguments: com.linkedin.tony.cli.ClusterSubmitter
Reference here.
python hadoop
google-cloud-platform google-cloud-dataproc asked Nov 12 '18 at 6:18
spicyramen
3,18523369
This is my current Hadoop job.
java -cp `hadoop classpath`:/usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
How to convert it to a gcloud dataproc jobs submit hadoop job?
I tried:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar --class com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I keep getting:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) argument --class: Exactly one of (--class | --jar) must be specified.
Usage: gcloud dataproc jobs submit hadoop --cluster=CLUSTER (--class=MAIN_CLASS | --jar=MAIN_JAR) [optional flags] [-- JOB_ARGS ...]
optional flags may be --archives | --async | --bucket | --class |
--driver-log-levels | --files | --help | --jar |
--jars | --labels | --max-failures-per-hour |
--properties | --region
For detailed information on this command and its flags, run:
gcloud dataproc jobs submit hadoop --help
If I pass:
gcloud dataproc jobs submit hadoop --cluster tony-dev
--jar /usr/local/src/jobs/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter --
--python_venv=/usr/local/src/jobs/MyJob/mnist_venv.zip
--src_dir=/usr/local/src/jobs/MyJob/
--executes=/usr/local/src/jobs/MyJob/src/mnist_distributed.py
--conf_file=/usr/local/src/jobs/MyJob/tony.xml
--python_binary_path=venv/bin/python3.5
I get:
ERROR: (gcloud.dataproc.jobs.submit.hadoop) unrecognized arguments: com.linkedin.tony.cli.ClusterSubmitter
Reference here.
python hadoop
google-cloud-platform google-cloud-dataproc python hadoop
google-cloud-platform google-cloud-dataproc asked Nov 12 '18 at 6:18
spicyramen
3,18523369
asked Nov 12 '18 at 6:18
spicyramen
3,18523369
asked Nov 12 '18 at 6:18
spicyramen
3,18523369
asked Nov 12 '18 at 6:18
spicyramen
3,18523369
asked Nov 12 '18 at 6:18
spicyramen
3,18523369
3,18523369
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
3
It was a simple change:
Changed --jar to --jars and now it works.
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53256800%2fsend-a-hadoop-job-via-gcloud%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
3
It was a simple change:
Changed --jar to --jars and now it works.
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
add a comment |
3
It was a simple change:
Changed --jar to --jars and now it works.
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
add a comment |
3
3
3
It was a simple change:
Changed --jar to --jars and now it works.
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
It was a simple change:
Changed --jar to --jars and now it works.
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
answered Nov 12 '18 at 6:32
spicyramen
3,18523369
3,18523369
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53256800%2fsend-a-hadoop-job-via-gcloud%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
