Intermediate result storage for Spring Batch processing REST service
up vote
0
down vote
favorite
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service without saving anything to any database.
Considering each Step needs it's ItemReader and ItemWriter, I imagine I would need to implement one ItemReader that will download documents from input service, one ItemWriter that will somehow store intermediate results in memory, then an ItemReader that will read such results from memory for next steps and the last ItemWriter that will send results once all processing is done.
Would that be a good approach or does anyone have any better ideas?
spring rest spring-batch
asked Nov 11 at 12:15
Leonid Bor
5151920
add a comment |
up vote
0
down vote
favorite
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service without saving anything to any database.
Considering each Step needs it's ItemReader and ItemWriter, I imagine I would need to implement one ItemReader that will download documents from input service, one ItemWriter that will somehow store intermediate results in memory, then an ItemReader that will read such results from memory for next steps and the last ItemWriter that will send results once all processing is done.
Would that be a good approach or does anyone have any better ideas?
spring rest spring-batch
asked Nov 11 at 12:15
Leonid Bor
5151920
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service without saving anything to any database.
Considering each Step needs it's ItemReader and ItemWriter, I imagine I would need to implement one ItemReader that will download documents from input service, one ItemWriter that will somehow store intermediate results in memory, then an ItemReader that will read such results from memory for next steps and the last ItemWriter that will send results once all processing is done.
Would that be a good approach or does anyone have any better ideas?
spring rest spring-batch
asked Nov 11 at 12:15
Leonid Bor
5151920
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service without saving anything to any database.
Considering each Step needs it's ItemReader and ItemWriter, I imagine I would need to implement one ItemReader that will download documents from input service, one ItemWriter that will somehow store intermediate results in memory, then an ItemReader that will read such results from memory for next steps and the last ItemWriter that will send results once all processing is done.
Would that be a good approach or does anyone have any better ideas?
spring rest spring-batch
spring rest spring-batch
asked Nov 11 at 12:15
Leonid Bor
5151920
asked Nov 11 at 12:15
Leonid Bor
5151920
asked Nov 11 at 12:15
Leonid Bor
5151920
asked Nov 11 at 12:15
Leonid Bor
5151920
asked Nov 11 at 12:15
Leonid Bor
5151920
5151920
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service
You can do it in one chunk-oriented step. The reader to get data from the REST service, a processor to process data and a write to post/put data to the other REST service.
without saving anything to any database.
Even though you don't want to save anything to a database (I guess you mean to a separate database server), you can still use an in-memory database and save intermediate processing results to a staging table. A first step would get the data from the REST service, process it and write it to the staging table. The second step reads data from the staging table and send it to the second REST service. The advantage of this technique is that you can do aggregation on processing results on the staging table before starting the second step.
Would that be a good approach or does anyone have any better ideas?
A third way to do it is to use a staging in-memory java.util.Queue. Here, you can use two steps like with the staging table but using the queue as the staging storage.
Hope this helps.
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
What if i run two or more such jobs at the same time? I guess i’ll have to use locking ob the table, which might affect performance. What about storing data in job context? Is it possible via ItemProcessor interface? I need to have more than one processor btw
– Leonid Bor
Nov 13 at 21:22
no need for locking, you can have a staging table per job instance, or a discriminator column to distinguish between records. Storing data in job context is not recommended if the data is big enough because the execution context end up serialized and persisted in the database.
– Mahmoud Ben Hassine
Nov 13 at 22:57
You mean the context is persisted after the job is done?
– Leonid Bor
Nov 15 at 10:29
no, the step execution context is persisted after each chunk, whereas the job execution context is persisted after each step. Please refer to last paragraph of this section: docs.spring.io/spring-batch/4.1.x/reference/html/….
– Mahmoud Ben Hassine
Nov 15 at 10:33
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service
You can do it in one chunk-oriented step. The reader to get data from the REST service, a processor to process data and a write to post/put data to the other REST service.
without saving anything to any database.
Even though you don't want to save anything to a database (I guess you mean to a separate database server), you can still use an in-memory database and save intermediate processing results to a staging table. A first step would get the data from the REST service, process it and write it to the staging table. The second step reads data from the staging table and send it to the second REST service. The advantage of this technique is that you can do aggregation on processing results on the staging table before starting the second step.
Would that be a good approach or does anyone have any better ideas?
A third way to do it is to use a staging in-memory java.util.Queue. Here, you can use two steps like with the staging table but using the queue as the staging storage.
Hope this helps.
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
What if i run two or more such jobs at the same time? I guess i’ll have to use locking ob the table, which might affect performance. What about storing data in job context? Is it possible via ItemProcessor interface? I need to have more than one processor btw
– Leonid Bor
Nov 13 at 21:22
no need for locking, you can have a staging table per job instance, or a discriminator column to distinguish between records. Storing data in job context is not recommended if the data is big enough because the execution context end up serialized and persisted in the database.
– Mahmoud Ben Hassine
Nov 13 at 22:57
You mean the context is persisted after the job is done?
– Leonid Bor
Nov 15 at 10:29
no, the step execution context is persisted after each chunk, whereas the job execution context is persisted after each step. Please refer to last paragraph of this section: docs.spring.io/spring-batch/4.1.x/reference/html/….
– Mahmoud Ben Hassine
Nov 15 at 10:33
add a comment |
up vote
0
down vote
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service
You can do it in one chunk-oriented step. The reader to get data from the REST service, a processor to process data and a write to post/put data to the other REST service.
without saving anything to any database.
Even though you don't want to save anything to a database (I guess you mean to a separate database server), you can still use an in-memory database and save intermediate processing results to a staging table. A first step would get the data from the REST service, process it and write it to the staging table. The second step reads data from the staging table and send it to the second REST service. The advantage of this technique is that you can do aggregation on processing results on the staging table before starting the second step.
Would that be a good approach or does anyone have any better ideas?
A third way to do it is to use a staging in-memory java.util.Queue. Here, you can use two steps like with the staging table but using the queue as the staging storage.
Hope this helps.
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
What if i run two or more such jobs at the same time? I guess i’ll have to use locking ob the table, which might affect performance. What about storing data in job context? Is it possible via ItemProcessor interface? I need to have more than one processor btw
– Leonid Bor
Nov 13 at 21:22
no need for locking, you can have a staging table per job instance, or a discriminator column to distinguish between records. Storing data in job context is not recommended if the data is big enough because the execution context end up serialized and persisted in the database.
– Mahmoud Ben Hassine
Nov 13 at 22:57
You mean the context is persisted after the job is done?
– Leonid Bor
Nov 15 at 10:29
no, the step execution context is persisted after each chunk, whereas the job execution context is persisted after each step. Please refer to last paragraph of this section: docs.spring.io/spring-batch/4.1.x/reference/html/….
– Mahmoud Ben Hassine
Nov 15 at 10:33
add a comment |
up vote
0
down vote
up vote
0
down vote
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service
You can do it in one chunk-oriented step. The reader to get data from the REST service, a processor to process data and a write to post/put data to the other REST service.
without saving anything to any database.
Even though you don't want to save anything to a database (I guess you mean to a separate database server), you can still use an in-memory database and save intermediate processing results to a staging table. A first step would get the data from the REST service, process it and write it to the staging table. The second step reads data from the staging table and send it to the second REST service. The advantage of this technique is that you can do aggregation on processing results on the staging table before starting the second step.
Would that be a good approach or does anyone have any better ideas?
A third way to do it is to use a staging in-memory java.util.Queue. Here, you can use two steps like with the staging table but using the queue as the staging storage.
Hope this helps.
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
I would like to get a document out of REST service, process with Spring Batch and then send to other REST service
You can do it in one chunk-oriented step. The reader to get data from the REST service, a processor to process data and a write to post/put data to the other REST service.
without saving anything to any database.
Even though you don't want to save anything to a database (I guess you mean to a separate database server), you can still use an in-memory database and save intermediate processing results to a staging table. A first step would get the data from the REST service, process it and write it to the staging table. The second step reads data from the staging table and send it to the second REST service. The advantage of this technique is that you can do aggregation on processing results on the staging table before starting the second step.
Would that be a good approach or does anyone have any better ideas?
A third way to do it is to use a staging in-memory java.util.Queue. Here, you can use two steps like with the staging table but using the queue as the staging storage.
Hope this helps.
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
answered Nov 11 at 19:21
Mahmoud Ben Hassine
3,4551714
3,4551714
What if i run two or more such jobs at the same time? I guess i’ll have to use locking ob the table, which might affect performance. What about storing data in job context? Is it possible via ItemProcessor interface? I need to have more than one processor btw
– Leonid Bor
Nov 13 at 21:22
no need for locking, you can have a staging table per job instance, or a discriminator column to distinguish between records. Storing data in job context is not recommended if the data is big enough because the execution context end up serialized and persisted in the database.
– Mahmoud Ben Hassine
Nov 13 at 22:57
You mean the context is persisted after the job is done?
– Leonid Bor
Nov 15 at 10:29
no, the step execution context is persisted after each chunk, whereas the job execution context is persisted after each step. Please refer to last paragraph of this section: docs.spring.io/spring-batch/4.1.x/reference/html/….
– Mahmoud Ben Hassine
Nov 15 at 10:33
add a comment |
What if i run two or more such jobs at the same time? I guess i’ll have to use locking ob the table, which might affect performance. What about storing data in job context? Is it possible via ItemProcessor interface? I need to have more than one processor btw
– Leonid Bor
Nov 13 at 21:22
no need for locking, you can have a staging table per job instance, or a discriminator column to distinguish between records. Storing data in job context is not recommended if the data is big enough because the execution context end up serialized and persisted in the database.
– Mahmoud Ben Hassine
Nov 13 at 22:57
You mean the context is persisted after the job is done?
– Leonid Bor
Nov 15 at 10:29
no, the step execution context is persisted after each chunk, whereas the job execution context is persisted after each step. Please refer to last paragraph of this section: docs.spring.io/spring-batch/4.1.x/reference/html/….
– Mahmoud Ben Hassine
Nov 15 at 10:33
What if i run two or more such jobs at the same time? I guess i’ll have to use locking ob the table, which might affect performance. What about storing data in job context? Is it possible via ItemProcessor interface? I need to have more than one processor btw
– Leonid Bor
Nov 13 at 21:22
What if i run two or more such jobs at the same time? I guess i’ll have to use locking ob the table, which might affect performance. What about storing data in job context? Is it possible via ItemProcessor interface? I need to have more than one processor btw
– Leonid Bor
Nov 13 at 21:22
no need for locking, you can have a staging table per job instance, or a discriminator column to distinguish between records. Storing data in job context is not recommended if the data is big enough because the execution context end up serialized and persisted in the database.
– Mahmoud Ben Hassine
Nov 13 at 22:57
no need for locking, you can have a staging table per job instance, or a discriminator column to distinguish between records. Storing data in job context is not recommended if the data is big enough because the execution context end up serialized and persisted in the database.
– Mahmoud Ben Hassine
Nov 13 at 22:57
You mean the context is persisted after the job is done?
– Leonid Bor
Nov 15 at 10:29
You mean the context is persisted after the job is done?
– Leonid Bor
Nov 15 at 10:29
no, the step execution context is persisted after each chunk, whereas the job execution context is persisted after each step. Please refer to last paragraph of this section: docs.spring.io/spring-batch/4.1.x/reference/html/….
– Mahmoud Ben Hassine
Nov 15 at 10:33
no, the step execution context is persisted after each chunk, whereas the job execution context is persisted after each step. Please refer to last paragraph of this section: docs.spring.io/spring-batch/4.1.x/reference/html/….
– Mahmoud Ben Hassine
Nov 15 at 10:33
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53248654%2fintermediate-result-storage-for-spring-batch-processing-rest-service%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
