Paramiko Download, process and re-upload the same file
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
asked Nov 11 at 19:09
canofwhoopass
155
add a comment |
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
asked Nov 11 at 19:09
canofwhoopass
155
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingputandputfoin this respect. Actually all thatputdoes is that it callsopenandputfo, just like your code does.
– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
add a comment |
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
asked Nov 11 at 19:09
canofwhoopass
155
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
python sftp paramiko
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
asked Nov 11 at 19:09
canofwhoopass
155
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
asked Nov 11 at 19:09
canofwhoopass
155
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
edited Nov 12 at 7:00
Martin Prikryl
85.4k22163355
85.4k22163355
asked Nov 11 at 19:09
canofwhoopass
155
asked Nov 11 at 19:09
canofwhoopass
155
asked Nov 11 at 19:09
canofwhoopass
155
155
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingputandputfoin this respect. Actually all thatputdoes is that it callsopenandputfo, just like your code does.
– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
add a comment |
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingputandputfoin this respect. Actually all thatputdoes is that it callsopenandputfo, just like your code does.
– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.– Martin Prikryl
Nov 11 at 22:39
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
add a comment |
1 Answer
1
active
oldest
votes
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
answered Nov 11 at 19:53
Martin Prikryl
85.4k22163355
85.4k22163355
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
putandputfoin this respect. Actually all thatputdoes is that it callsopenandputfo, just like your code does.– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54