There are there ways to remove duplicates from flat file, 1)Using Unix command in session. 2)Using Dynamic look up feature of look up. 3)Using Sorter transformation Using Sorter transformation is the easiest way to remove duplicates.But you need to compromise with performance if you are loading huge amount of data. If you loading huge amount of data it is better to use dynamic look up. Now lets discuss how to do it first using sorter, All you have to do is,import sorter transformation in mapping and enable the distinct property in the properties tab of sorter.