Lice Exposure And What To Do

Subsonic 3.0 [Francis Jehl]: Scalability, some advice.

In his post

[SSIS] Scalability, some advice.

Francis, offers us some very good advice to optimize SSIS packages.

For once he does not speak of Data Mining (someone understands what he says at least DMX), I am going to intervene / add one point. I hope it does not bother you Francis.

to Francis said

... SSIS is optimized memory So to you to save! Already, select only the relevant columns ...

In Data Flow Sources, it is indeed advisable to select only the columns that will be actually used later. However, later in your data stream, some transformations (eg Derived Column, Data Conversion etc. ...) will duplicate the first column is used most likely.

One solution often put in place to remove unnecessary columns, is to use the Union All task. Output Column by removing unnecessary, we think rightly reduce the memory used. The worst solution would be to use synchronous aggregate spots or lots to remove these columns. But then if I agree that it does reduce the memory usage, why is still not a good idea? Simply by the fact that deleting a column induces a copy of the buffers of the pipeline into new buffers. And unfortunately, experience shows that this copying action is too costly in terms of CPU cycles to compensate for reduced memory consumption.

Darren Green - May 10, 2007 at said Something That people, Including me get a bit worried about, Is The Fact That You End Up With Big thesis wide buffers. The classic scenario is when you need to convert between string and Unicode string or visa versa. You end up duplicating all your columns, which seems really horrid. Well it may not be great, but the alternative is even worse. Data Conversion and Derived Column transforms can do the conversion, but are synchronous, so we have duplicate columns in effect. Would it be better to have a custom component that worked asynchronously, so you didn't end up with two columns for all your strings? The short answer is no. I wrote a test component, cunningly named DeUnicodeAsynchTestComponent, and simply put it was 2 - 2.5 times slower than a data conversion transform. The cost of copying data between buffers for the asynchronous nature is far more expensive than the extra baggage of the "duplicate" columns.

That Francis, I hope this little clarification intervention does not bother you too ... I promise I'll leave you alone until you talk about data mining .... As for me, I return to my C # code.