Geeks With Blogs
Rahul Anand's Blog If my mind can conceive it, and my heart can believe it, I know I can achieve it.

Many articles on implementing a loop in PDI suggest to put a ‘Wait For’ step and join the output hop back to a previous step.

Something as suggested by below screen shot:

image

But I would like to warn about this implementation of looping that this causes recursive stack allocation by JVM during job execution and the system may run out of memory after a high number of iterations (depending the system available available memory).

In a test run on my laptop in which I continuously ran the above job with a ‘Wait For’ configuration of 1 second, within minutes the JVM crashed. I also noticed a continuous increase in JVM memory usage before it crashed.

Therefore to conclude it is ok to run few iterations (like retrying on error condition) with above mechanism but make sure you do so for a finite number of iteration (may be lesser than 100), but not infinite looping.

So what is the suggested approach for infinite looping – One of the possible way is to use the settings of ‘Start’ step. Set the ‘Repeat’ flag and add interval configuration. This cause the job to be re-initialize completely as a new instance and does not cause any memory issue.

 

image

Posted on Friday, February 8, 2013 7:45 AM ETL , pentaho , kettle , PDI , Datawarehouse , Pentaho Data Integration | Back to top


Comments on this post: Pentaho Data Integration (Kettle)–Continuous processing by implementing a loop

# re: Pentaho Data Integration (Kettle)–Continuous processing by implementing a loop
Requesting Gravatar...
Hello, Thank you for the information on Pentaho Data Integration. You can get some more information on Pentaho Consulting on PDI at http://pentaho.graymatter.co.in/
Left by Pentaho Consulting on Jan 06, 2017 11:44 AM

# re: Pentaho Data Integration (Kettle)–Continuous processing by implementing a loop
Requesting Gravatar...
Nice article on Pentaho Data Integration and Useful Information. Get some more details on Pentaho at Pentaho Consulting
Left by Pentaho Consulting on Jan 06, 2017 11:45 AM

Your comment:
 (will show your gravatar)


Copyright © Rahul Anand | Powered by: GeeksWithBlogs.net