Iterating over two pandas lists
I'm working on cleaning up some of my code to make it a bit more pythonic, but I'm wondering if the below could be written in a nicer way with something like an itertools or pandas method. The below code works, however I'm hoping to remove the double for-loop and consolidate a bit of the code for performance reasons.
Ultimately, I'm working with a list of indices that call a Pandas column.
def foo(dataset):
api_reshaped = pd.DataFrame(columns=['foo', 'bar'])
k = 0
for index, _ in dataset.iterrows():
for key in dataset.iloc[index][0][0]:
api_reshaped.loc[k, 'foo'] = key
api_reshaped.loc[k, 'bar'] = dataset.iloc[index][0][0][key]
k += 1
return api_reshaped
Below is the expected input/output from this function:
foo_input = pd.dataframe({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
print foo_input(foo_input)
# expected_output = pd.dataframe({
# 'foo': 'foo_query',
# 'bar': [{'bar_query': 'data'}]
# })
Many thanks!
python pandas
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
add a comment |
I'm working on cleaning up some of my code to make it a bit more pythonic, but I'm wondering if the below could be written in a nicer way with something like an itertools or pandas method. The below code works, however I'm hoping to remove the double for-loop and consolidate a bit of the code for performance reasons.
Ultimately, I'm working with a list of indices that call a Pandas column.
def foo(dataset):
api_reshaped = pd.DataFrame(columns=['foo', 'bar'])
k = 0
for index, _ in dataset.iterrows():
for key in dataset.iloc[index][0][0]:
api_reshaped.loc[k, 'foo'] = key
api_reshaped.loc[k, 'bar'] = dataset.iloc[index][0][0][key]
k += 1
return api_reshaped
Below is the expected input/output from this function:
foo_input = pd.dataframe({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
print foo_input(foo_input)
# expected_output = pd.dataframe({
# 'foo': 'foo_query',
# 'bar': [{'bar_query': 'data'}]
# })
Many thanks!
python pandas
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
2
Could you show some data input and expected output?
– Franco Piccolo
Nov 13 '18 at 4:48
Yes, of course. I just added some additional information on this. I believe I simplified it enough to drop the complexities but please let me know if any additional information will be helpful. Many thanks, Franco.
– sokeefe
Nov 13 '18 at 5:06
1
There are typos there could you validate the that code runs and it gives the expected input and output?
– Franco Piccolo
Nov 13 '18 at 5:11
Yes, good catch. Believe I just fixed them all...
– sokeefe
Nov 13 '18 at 5:21
2
@sokeefe, Nope you haven't fixed them at all. Your function doesn't work withfoo_input. In fact, you haven't tested your code at all, e.g.pd.dataframedoesn't work.
– jpp
Nov 13 '18 at 15:11
add a comment |
I'm working on cleaning up some of my code to make it a bit more pythonic, but I'm wondering if the below could be written in a nicer way with something like an itertools or pandas method. The below code works, however I'm hoping to remove the double for-loop and consolidate a bit of the code for performance reasons.
Ultimately, I'm working with a list of indices that call a Pandas column.
def foo(dataset):
api_reshaped = pd.DataFrame(columns=['foo', 'bar'])
k = 0
for index, _ in dataset.iterrows():
for key in dataset.iloc[index][0][0]:
api_reshaped.loc[k, 'foo'] = key
api_reshaped.loc[k, 'bar'] = dataset.iloc[index][0][0][key]
k += 1
return api_reshaped
Below is the expected input/output from this function:
foo_input = pd.dataframe({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
print foo_input(foo_input)
# expected_output = pd.dataframe({
# 'foo': 'foo_query',
# 'bar': [{'bar_query': 'data'}]
# })
Many thanks!
python pandas
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
I'm working on cleaning up some of my code to make it a bit more pythonic, but I'm wondering if the below could be written in a nicer way with something like an itertools or pandas method. The below code works, however I'm hoping to remove the double for-loop and consolidate a bit of the code for performance reasons.
Ultimately, I'm working with a list of indices that call a Pandas column.
def foo(dataset):
api_reshaped = pd.DataFrame(columns=['foo', 'bar'])
k = 0
for index, _ in dataset.iterrows():
for key in dataset.iloc[index][0][0]:
api_reshaped.loc[k, 'foo'] = key
api_reshaped.loc[k, 'bar'] = dataset.iloc[index][0][0][key]
k += 1
return api_reshaped
Below is the expected input/output from this function:
foo_input = pd.dataframe({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
print foo_input(foo_input)
# expected_output = pd.dataframe({
# 'foo': 'foo_query',
# 'bar': [{'bar_query': 'data'}]
# })
Many thanks!
python pandas
python pandas
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
edited Nov 13 '18 at 5:16
sokeefe
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
asked Nov 13 '18 at 4:45
sokeefesokeefe
399422
399422
2
Could you show some data input and expected output?
– Franco Piccolo
Nov 13 '18 at 4:48
Yes, of course. I just added some additional information on this. I believe I simplified it enough to drop the complexities but please let me know if any additional information will be helpful. Many thanks, Franco.
– sokeefe
Nov 13 '18 at 5:06
1
There are typos there could you validate the that code runs and it gives the expected input and output?
– Franco Piccolo
Nov 13 '18 at 5:11
Yes, good catch. Believe I just fixed them all...
– sokeefe
Nov 13 '18 at 5:21
2
@sokeefe, Nope you haven't fixed them at all. Your function doesn't work withfoo_input. In fact, you haven't tested your code at all, e.g.pd.dataframedoesn't work.
– jpp
Nov 13 '18 at 15:11
add a comment |
2
Could you show some data input and expected output?
– Franco Piccolo
Nov 13 '18 at 4:48
Yes, of course. I just added some additional information on this. I believe I simplified it enough to drop the complexities but please let me know if any additional information will be helpful. Many thanks, Franco.
– sokeefe
Nov 13 '18 at 5:06
1
There are typos there could you validate the that code runs and it gives the expected input and output?
– Franco Piccolo
Nov 13 '18 at 5:11
Yes, good catch. Believe I just fixed them all...
– sokeefe
Nov 13 '18 at 5:21
2
@sokeefe, Nope you haven't fixed them at all. Your function doesn't work withfoo_input. In fact, you haven't tested your code at all, e.g.pd.dataframedoesn't work.
– jpp
Nov 13 '18 at 15:11
2
2
Could you show some data input and expected output?
– Franco Piccolo
Nov 13 '18 at 4:48
Could you show some data input and expected output?
– Franco Piccolo
Nov 13 '18 at 4:48
Yes, of course. I just added some additional information on this. I believe I simplified it enough to drop the complexities but please let me know if any additional information will be helpful. Many thanks, Franco.
– sokeefe
Nov 13 '18 at 5:06
Yes, of course. I just added some additional information on this. I believe I simplified it enough to drop the complexities but please let me know if any additional information will be helpful. Many thanks, Franco.
– sokeefe
Nov 13 '18 at 5:06
1
1
There are typos there could you validate the that code runs and it gives the expected input and output?
– Franco Piccolo
Nov 13 '18 at 5:11
There are typos there could you validate the that code runs and it gives the expected input and output?
– Franco Piccolo
Nov 13 '18 at 5:11
Yes, good catch. Believe I just fixed them all...
– sokeefe
Nov 13 '18 at 5:21
Yes, good catch. Believe I just fixed them all...
– sokeefe
Nov 13 '18 at 5:21
2
2
@sokeefe, Nope you haven't fixed them at all. Your function doesn't work with
foo_input. In fact, you haven't tested your code at all, e.g. pd.dataframe doesn't work.– jpp
Nov 13 '18 at 15:11
@sokeefe, Nope you haven't fixed them at all. Your function doesn't work with
foo_input. In fact, you haven't tested your code at all, e.g. pd.dataframe doesn't work.– jpp
Nov 13 '18 at 15:11
add a comment |
1 Answer
1
active
oldest
votes
You can use list comprehension with transpose:
# your input data
foo_input = pd.DataFrame({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
# use list comprehension with transpose
df = pd.DataFrame([item for item in foo_input['batch_data']]).T.reset_index()
# rename your columns
df.columns = ['Foo', 'Bar']
Foo Bar
0 foo_query [{'bar_query': 'data'}]
you can use applymap with a lambda function if you want to remove the list and just have a dict
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53273967%2fiterating-over-two-pandas-lists%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can use list comprehension with transpose:
# your input data
foo_input = pd.DataFrame({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
# use list comprehension with transpose
df = pd.DataFrame([item for item in foo_input['batch_data']]).T.reset_index()
# rename your columns
df.columns = ['Foo', 'Bar']
Foo Bar
0 foo_query [{'bar_query': 'data'}]
you can use applymap with a lambda function if you want to remove the list and just have a dict
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
add a comment |
You can use list comprehension with transpose:
# your input data
foo_input = pd.DataFrame({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
# use list comprehension with transpose
df = pd.DataFrame([item for item in foo_input['batch_data']]).T.reset_index()
# rename your columns
df.columns = ['Foo', 'Bar']
Foo Bar
0 foo_query [{'bar_query': 'data'}]
you can use applymap with a lambda function if you want to remove the list and just have a dict
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
add a comment |
You can use list comprehension with transpose:
# your input data
foo_input = pd.DataFrame({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
# use list comprehension with transpose
df = pd.DataFrame([item for item in foo_input['batch_data']]).T.reset_index()
# rename your columns
df.columns = ['Foo', 'Bar']
Foo Bar
0 foo_query [{'bar_query': 'data'}]
you can use applymap with a lambda function if you want to remove the list and just have a dict
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
You can use list comprehension with transpose:
# your input data
foo_input = pd.DataFrame({
'batch_data': [{'foo_query': [{'bar_query': 'data'}]}],
'query_spell': ['foo']
})
# use list comprehension with transpose
df = pd.DataFrame([item for item in foo_input['batch_data']]).T.reset_index()
# rename your columns
df.columns = ['Foo', 'Bar']
Foo Bar
0 foo_query [{'bar_query': 'data'}]
you can use applymap with a lambda function if you want to remove the list and just have a dict
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
edited Nov 13 '18 at 15:20
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
answered Nov 13 '18 at 15:06
ChrisChris
2,1812418
2,1812418
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53273967%2fiterating-over-two-pandas-lists%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Could you show some data input and expected output?
– Franco Piccolo
Nov 13 '18 at 4:48
Yes, of course. I just added some additional information on this. I believe I simplified it enough to drop the complexities but please let me know if any additional information will be helpful. Many thanks, Franco.
– sokeefe
Nov 13 '18 at 5:06
1
There are typos there could you validate the that code runs and it gives the expected input and output?
– Franco Piccolo
Nov 13 '18 at 5:11
Yes, good catch. Believe I just fixed them all...
– sokeefe
Nov 13 '18 at 5:21
2
@sokeefe, Nope you haven't fixed them at all. Your function doesn't work with
foo_input. In fact, you haven't tested your code at all, e.g.pd.dataframedoesn't work.– jpp
Nov 13 '18 at 15:11