Python Program to Remove Duplicate Items from a List of Dictionaries

Written by
Vinish Kapoor is a seasoned software development professional and a fervent enthusiast of artificial intelligence (AI). His impressive career spans over 20 years, marked by a relentless pursuit of innovation and excellence in the field of information technology. As an Oracle ACE, Vinish has distinguished himself as a leading expert in Oracle technologies, a title awarded to individuals who have demonstrated their deep commitment, leadership, and expertise in the Oracle community.
  • Python
  • 1 min read

Here is a Python program to remove duplicate items from a list of dictionaries while keeping the order.

Remove Duplicate Items From a Dictionary in Python and Keep The Order

The following Python program will remove the two duplicate records for {'APPLE': 20, 'MANGO': 30} entries from the list of directories.

def dedupe(items, key=None):
    seen = set()
    for item in items:
        val = item if key is None else key(item)
        if val not in seen:
            yield item
            seen.add(val)

if __name__ == '__main__':
    a = [
        {'APPLE': 20, 'MANGO': 30},
        {'APPLE': 10, 'MANGO': 40},
        {'APPLE': 20, 'MANGO': 30},
        {'APPLE': 20, 'MANGO': 30},
        {'APPLE': 10, 'MANGO': 15}
        ]
#print(a)
    print(list(dedupe(a, key=lambda a: (a['APPLE'],a['MANGO']))))

Output:

[{'APPLE': 20, 'MANGO': 30}, {'APPLE': 10, 'MANGO': 40}, {'APPLE': 10, 'MANGO': 15}]

Process finished with exit code 0

See also: