How can I use loops/recursion with promise-chains?
up vote
3
down vote
favorite
Imagine for example that you want to store paginated data from an API to a database.
let db;
let pageitems = 35
var offset = 0;
dbConnect //establish connection to database
.then( fetch(apiLink+?offset=2)
.then( res => res.json())
.then( res => {
var total = res.count
return collection.insertMany(res.data, {ordered: false})
// If offset is less than total, I want to increase offset and go back to the fetch-event.
.catch( err => {
if(err.code !== 11000){log(err)}
else{log({completed: err.result.nInserted, duplicates:
err.result.result.writeErrors.length});}
})
.then(() => {
connection.close();
})
javascript node.js ecmascript-6
asked Nov 11 at 16:31
Himmators
5,5012684171
|
show 1 more comment
up vote
3
down vote
favorite
Imagine for example that you want to store paginated data from an API to a database.
let db;
let pageitems = 35
var offset = 0;
dbConnect //establish connection to database
.then( fetch(apiLink+?offset=2)
.then( res => res.json())
.then( res => {
var total = res.count
return collection.insertMany(res.data, {ordered: false})
// If offset is less than total, I want to increase offset and go back to the fetch-event.
.catch( err => {
if(err.code !== 11000){log(err)}
else{log({completed: err.result.nInserted, duplicates:
err.result.result.writeErrors.length});}
})
.then(() => {
connection.close();
})
javascript node.js ecmascript-6
asked Nov 11 at 16:31
Himmators
5,5012684171
1
Your fetch only fetchespageitems(15) items? Isres.countthe number of fetched records, or the total size of the data set at the backend?
– trincot
Nov 11 at 16:35
2
You can use async/await within a while loop. Or async/await is not an option here?
– Sim Dim
Nov 11 at 16:37
2
Warning: your code is callingfetchimmediately, not whendbConnectresolves. You must put() => fetch(...there.
– trincot
Nov 11 at 16:40
2
Yes, recursion is the way to go. But just like in the non-promise case, for that you first need a function that you could call recursively.
– Bergi
Nov 11 at 16:43
1
@JonasWilms Sure, but it seems the OP tries to learn ES6 promise chaining first. My comment was just meant to be encouraging, that recursion is a correct approach, and the OP should try it. Later when adoptingasync/awaityou can of course use a loop.
– Bergi
Nov 11 at 17:00
|
show 1 more comment
up vote
3
down vote
favorite
up vote
3
down vote
favorite
Imagine for example that you want to store paginated data from an API to a database.
let db;
let pageitems = 35
var offset = 0;
dbConnect //establish connection to database
.then( fetch(apiLink+?offset=2)
.then( res => res.json())
.then( res => {
var total = res.count
return collection.insertMany(res.data, {ordered: false})
// If offset is less than total, I want to increase offset and go back to the fetch-event.
.catch( err => {
if(err.code !== 11000){log(err)}
else{log({completed: err.result.nInserted, duplicates:
err.result.result.writeErrors.length});}
})
.then(() => {
connection.close();
})
javascript node.js ecmascript-6
asked Nov 11 at 16:31
Himmators
5,5012684171
Imagine for example that you want to store paginated data from an API to a database.
let db;
let pageitems = 35
var offset = 0;
dbConnect //establish connection to database
.then( fetch(apiLink+?offset=2)
.then( res => res.json())
.then( res => {
var total = res.count
return collection.insertMany(res.data, {ordered: false})
// If offset is less than total, I want to increase offset and go back to the fetch-event.
.catch( err => {
if(err.code !== 11000){log(err)}
else{log({completed: err.result.nInserted, duplicates:
err.result.result.writeErrors.length});}
})
.then(() => {
connection.close();
})
javascript node.js ecmascript-6
javascript node.js ecmascript-6
asked Nov 11 at 16:31
Himmators
5,5012684171
asked Nov 11 at 16:31
Himmators
5,5012684171
asked Nov 11 at 16:31
Himmators
5,5012684171
asked Nov 11 at 16:31
Himmators
5,5012684171
asked Nov 11 at 16:31
Himmators
5,5012684171
5,5012684171
1
Your fetch only fetchespageitems(15) items? Isres.countthe number of fetched records, or the total size of the data set at the backend?
– trincot
Nov 11 at 16:35
2
You can use async/await within a while loop. Or async/await is not an option here?
– Sim Dim
Nov 11 at 16:37
2
Warning: your code is callingfetchimmediately, not whendbConnectresolves. You must put() => fetch(...there.
– trincot
Nov 11 at 16:40
2
Yes, recursion is the way to go. But just like in the non-promise case, for that you first need a function that you could call recursively.
– Bergi
Nov 11 at 16:43
1
@JonasWilms Sure, but it seems the OP tries to learn ES6 promise chaining first. My comment was just meant to be encouraging, that recursion is a correct approach, and the OP should try it. Later when adoptingasync/awaityou can of course use a loop.
– Bergi
Nov 11 at 17:00
|
show 1 more comment
1
Your fetch only fetchespageitems(15) items? Isres.countthe number of fetched records, or the total size of the data set at the backend?
– trincot
Nov 11 at 16:35
2
You can use async/await within a while loop. Or async/await is not an option here?
– Sim Dim
Nov 11 at 16:37
2
Warning: your code is callingfetchimmediately, not whendbConnectresolves. You must put() => fetch(...there.
– trincot
Nov 11 at 16:40
2
Yes, recursion is the way to go. But just like in the non-promise case, for that you first need a function that you could call recursively.
– Bergi
Nov 11 at 16:43
1
@JonasWilms Sure, but it seems the OP tries to learn ES6 promise chaining first. My comment was just meant to be encouraging, that recursion is a correct approach, and the OP should try it. Later when adoptingasync/awaityou can of course use a loop.
– Bergi
Nov 11 at 17:00
1
1
Your fetch only fetches
pageitems (15) items? Is res.count the number of fetched records, or the total size of the data set at the backend?– trincot
Nov 11 at 16:35
Your fetch only fetches
pageitems (15) items? Is res.count the number of fetched records, or the total size of the data set at the backend?– trincot
Nov 11 at 16:35
2
2
You can use async/await within a while loop. Or async/await is not an option here?
– Sim Dim
Nov 11 at 16:37
You can use async/await within a while loop. Or async/await is not an option here?
– Sim Dim
Nov 11 at 16:37
2
2
Warning: your code is calling
fetch immediately, not when dbConnect resolves. You must put () => fetch(... there.– trincot
Nov 11 at 16:40
Warning: your code is calling
fetch immediately, not when dbConnect resolves. You must put () => fetch(... there.– trincot
Nov 11 at 16:40
2
2
Yes, recursion is the way to go. But just like in the non-promise case, for that you first need a function that you could call recursively.
– Bergi
Nov 11 at 16:43
Yes, recursion is the way to go. But just like in the non-promise case, for that you first need a function that you could call recursively.
– Bergi
Nov 11 at 16:43
1
1
@JonasWilms Sure, but it seems the OP tries to learn ES6 promise chaining first. My comment was just meant to be encouraging, that recursion is a correct approach, and the OP should try it. Later when adopting
async/await you can of course use a loop.– Bergi
Nov 11 at 17:00
@JonasWilms Sure, but it seems the OP tries to learn ES6 promise chaining first. My comment was just meant to be encouraging, that recursion is a correct approach, and the OP should try it. Later when adopting
async/await you can of course use a loop.– Bergi
Nov 11 at 17:00
|
show 1 more comment
2 Answers
2
active
oldest
votes
up vote
1
down vote
accepted
You could just use a regular loop:
(async function() {
const conn = await dbConnect;
for(let offset = 0; true; offset++) {
const { data, count } = await (await fetch(`api?page=${offset}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
To speed that up you could execute multiple requests in parallel:
const chunkSize = 10; // 10 in parallel
for(let offset = 0; offset < chunkSize; offset++) {
(async function() {
const conn = await dbConnect;
for(let offset2 = 0; true; offset2 += chunkSize) {
const { data, count } = await (await fetch(`api?page=${offset + offset2}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
}
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
this looks nice, I have to read up on await/async more...(though I don't want to call them all at once, I think the API will hate me if I do and it's a scrape anyways...)
– Himmators
Nov 11 at 21:10
1
@himmators definetly depends on the usecase, glad to help :)
– Jonas Wilms
Nov 12 at 15:47
add a comment |
up vote
1
down vote
Basically, you will want to wrap your fetch and insert into a function that you will call many times. See the below as an example to illustrate my point...
let db;
let pageitems = 35
var offset = 0;
var db = dbConnect() //establish connection to database
function fetch_and_insert(offset) {
db
.then(fetch(apiLink + "?" + offset))
.then(res => res.json())
.then(res => {
var total = res.count
collection.insertMany(res.data, { ordered: false })
.catch(err => {
if (err.code !== 11000) { log(err) }
else {
log({
completed: err.result.nInserted, duplicates: err.result.result.writeErrors.length
});
}
})
if (offset < total) return fetch_and_insert(offset + pageitems)
return null;
})
}
fetch_and_insert(offset)
.then(() => {
connection.close();
})
answered Nov 11 at 16:42
Paul
3,71922238
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
You could just use a regular loop:
(async function() {
const conn = await dbConnect;
for(let offset = 0; true; offset++) {
const { data, count } = await (await fetch(`api?page=${offset}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
To speed that up you could execute multiple requests in parallel:
const chunkSize = 10; // 10 in parallel
for(let offset = 0; offset < chunkSize; offset++) {
(async function() {
const conn = await dbConnect;
for(let offset2 = 0; true; offset2 += chunkSize) {
const { data, count } = await (await fetch(`api?page=${offset + offset2}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
}
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
this looks nice, I have to read up on await/async more...(though I don't want to call them all at once, I think the API will hate me if I do and it's a scrape anyways...)
– Himmators
Nov 11 at 21:10
1
@himmators definetly depends on the usecase, glad to help :)
– Jonas Wilms
Nov 12 at 15:47
add a comment |
up vote
1
down vote
accepted
You could just use a regular loop:
(async function() {
const conn = await dbConnect;
for(let offset = 0; true; offset++) {
const { data, count } = await (await fetch(`api?page=${offset}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
To speed that up you could execute multiple requests in parallel:
const chunkSize = 10; // 10 in parallel
for(let offset = 0; offset < chunkSize; offset++) {
(async function() {
const conn = await dbConnect;
for(let offset2 = 0; true; offset2 += chunkSize) {
const { data, count } = await (await fetch(`api?page=${offset + offset2}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
}
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
this looks nice, I have to read up on await/async more...(though I don't want to call them all at once, I think the API will hate me if I do and it's a scrape anyways...)
– Himmators
Nov 11 at 21:10
1
@himmators definetly depends on the usecase, glad to help :)
– Jonas Wilms
Nov 12 at 15:47
add a comment |
up vote
1
down vote
accepted
up vote
1
down vote
accepted
You could just use a regular loop:
(async function() {
const conn = await dbConnect;
for(let offset = 0; true; offset++) {
const { data, count } = await (await fetch(`api?page=${offset}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
To speed that up you could execute multiple requests in parallel:
const chunkSize = 10; // 10 in parallel
for(let offset = 0; offset < chunkSize; offset++) {
(async function() {
const conn = await dbConnect;
for(let offset2 = 0; true; offset2 += chunkSize) {
const { data, count } = await (await fetch(`api?page=${offset + offset2}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
}
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
You could just use a regular loop:
(async function() {
const conn = await dbConnect;
for(let offset = 0; true; offset++) {
const { data, count } = await (await fetch(`api?page=${offset}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
To speed that up you could execute multiple requests in parallel:
const chunkSize = 10; // 10 in parallel
for(let offset = 0; offset < chunkSize; offset++) {
(async function() {
const conn = await dbConnect;
for(let offset2 = 0; true; offset2 += chunkSize) {
const { data, count } = await (await fetch(`api?page=${offset + offset2}`)).json();
// Exit if the page is empty
if(count === 0) break;
await collection.insertMany(data, { ordered: false });
}
})();
}
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
answered Nov 11 at 16:45
Jonas Wilms
53.5k42547
53.5k42547
this looks nice, I have to read up on await/async more...(though I don't want to call them all at once, I think the API will hate me if I do and it's a scrape anyways...)
– Himmators
Nov 11 at 21:10
1
@himmators definetly depends on the usecase, glad to help :)
– Jonas Wilms
Nov 12 at 15:47
add a comment |
this looks nice, I have to read up on await/async more...(though I don't want to call them all at once, I think the API will hate me if I do and it's a scrape anyways...)
– Himmators
Nov 11 at 21:10
1
@himmators definetly depends on the usecase, glad to help :)
– Jonas Wilms
Nov 12 at 15:47
this looks nice, I have to read up on await/async more...(though I don't want to call them all at once, I think the API will hate me if I do and it's a scrape anyways...)
– Himmators
Nov 11 at 21:10
this looks nice, I have to read up on await/async more...(though I don't want to call them all at once, I think the API will hate me if I do and it's a scrape anyways...)
– Himmators
Nov 11 at 21:10
1
1
@himmators definetly depends on the usecase, glad to help :)
– Jonas Wilms
Nov 12 at 15:47
@himmators definetly depends on the usecase, glad to help :)
– Jonas Wilms
Nov 12 at 15:47
add a comment |
up vote
1
down vote
Basically, you will want to wrap your fetch and insert into a function that you will call many times. See the below as an example to illustrate my point...
let db;
let pageitems = 35
var offset = 0;
var db = dbConnect() //establish connection to database
function fetch_and_insert(offset) {
db
.then(fetch(apiLink + "?" + offset))
.then(res => res.json())
.then(res => {
var total = res.count
collection.insertMany(res.data, { ordered: false })
.catch(err => {
if (err.code !== 11000) { log(err) }
else {
log({
completed: err.result.nInserted, duplicates: err.result.result.writeErrors.length
});
}
})
if (offset < total) return fetch_and_insert(offset + pageitems)
return null;
})
}
fetch_and_insert(offset)
.then(() => {
connection.close();
})
answered Nov 11 at 16:42
Paul
3,71922238
add a comment |
up vote
1
down vote
Basically, you will want to wrap your fetch and insert into a function that you will call many times. See the below as an example to illustrate my point...
let db;
let pageitems = 35
var offset = 0;
var db = dbConnect() //establish connection to database
function fetch_and_insert(offset) {
db
.then(fetch(apiLink + "?" + offset))
.then(res => res.json())
.then(res => {
var total = res.count
collection.insertMany(res.data, { ordered: false })
.catch(err => {
if (err.code !== 11000) { log(err) }
else {
log({
completed: err.result.nInserted, duplicates: err.result.result.writeErrors.length
});
}
})
if (offset < total) return fetch_and_insert(offset + pageitems)
return null;
})
}
fetch_and_insert(offset)
.then(() => {
connection.close();
})
answered Nov 11 at 16:42
Paul
3,71922238
add a comment |
up vote
1
down vote
up vote
1
down vote
Basically, you will want to wrap your fetch and insert into a function that you will call many times. See the below as an example to illustrate my point...
let db;
let pageitems = 35
var offset = 0;
var db = dbConnect() //establish connection to database
function fetch_and_insert(offset) {
db
.then(fetch(apiLink + "?" + offset))
.then(res => res.json())
.then(res => {
var total = res.count
collection.insertMany(res.data, { ordered: false })
.catch(err => {
if (err.code !== 11000) { log(err) }
else {
log({
completed: err.result.nInserted, duplicates: err.result.result.writeErrors.length
});
}
})
if (offset < total) return fetch_and_insert(offset + pageitems)
return null;
})
}
fetch_and_insert(offset)
.then(() => {
connection.close();
})
answered Nov 11 at 16:42
Paul
3,71922238
Basically, you will want to wrap your fetch and insert into a function that you will call many times. See the below as an example to illustrate my point...
let db;
let pageitems = 35
var offset = 0;
var db = dbConnect() //establish connection to database
function fetch_and_insert(offset) {
db
.then(fetch(apiLink + "?" + offset))
.then(res => res.json())
.then(res => {
var total = res.count
collection.insertMany(res.data, { ordered: false })
.catch(err => {
if (err.code !== 11000) { log(err) }
else {
log({
completed: err.result.nInserted, duplicates: err.result.result.writeErrors.length
});
}
})
if (offset < total) return fetch_and_insert(offset + pageitems)
return null;
})
}
fetch_and_insert(offset)
.then(() => {
connection.close();
})
answered Nov 11 at 16:42
Paul
3,71922238
answered Nov 11 at 16:42
Paul
3,71922238
answered Nov 11 at 16:42
Paul
3,71922238
answered Nov 11 at 16:42
Paul
3,71922238
3,71922238
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53250797%2fhow-can-i-use-loops-recursion-with-promise-chains%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Your fetch only fetches
pageitems(15) items? Isres.countthe number of fetched records, or the total size of the data set at the backend?– trincot
Nov 11 at 16:35
2
You can use async/await within a while loop. Or async/await is not an option here?
– Sim Dim
Nov 11 at 16:37
2
Warning: your code is calling
fetchimmediately, not whendbConnectresolves. You must put() => fetch(...there.– trincot
Nov 11 at 16:40
2
Yes, recursion is the way to go. But just like in the non-promise case, for that you first need a function that you could call recursively.
– Bergi
Nov 11 at 16:43
1
@JonasWilms Sure, but it seems the OP tries to learn ES6 promise chaining first. My comment was just meant to be encouraging, that recursion is a correct approach, and the OP should try it. Later when adopting
async/awaityou can of course use a loop.– Bergi
Nov 11 at 17:00