A database with duplicate records can be very problematic for your application. They may produce inaccurate results, use storage space, and impair the functionality of the application. Due to this, it’s crucial to be able to rapidly and effectively detect and get rid of duplicate records.

Introduction

For .NET developers, Entity Framework is a potent Object-Relational Mapping (ORM) tool. It may assist you with operations like querying, updating, and removing data in a database and enables you to work with databases using strongly-typed.NET objects. This post will concentrate on using Entity Framework to identify and get rid of redundant records in a database.

Understanding Duplicate Records

When two or more rows in a database table have the same values for every column, duplicate entries are created. Multiple things, including data entry errors, software problems, and hardware issues, might result in duplicate records. Duplicate records might make it harder for your application to acquire accurate data from the database, which can lead to issues.

Install Required Software

• NET Core SDK from here
• Visual Studio or Visual Studio Code from here

Setup the Project

Make sure we have a functioning ASP.NET Core project with Entity Framework installed before we begin. You can start a new project in Visual Studio or by using the dotnet new command if you don’t already have one.

Add the Entity Framework Core package to your project once it has been configured. Run the following command in the Package Manager Console to accomplish this:

Install-Package Microsoft.EntityFrameworkCore

Find Duplicate Records Using Entity Framework in Asp.net Core

In Entity Framework, there are numerous methods for locating duplicate records. Below, we’ll look at a few of the more popular techniques.

Sample Data

 List<Employee> Employees = new List<Employee>
        {
         new Employee{ EmpId= 1, FirstName = "Vicky",LastName= "Pointing" },
         new Employee{ EmpId= 2, FirstName = "John",LastName= "Astle" },
         new Employee{ EmpId= 3, FirstName = "Vicky",LastName= "Pointing" },
         new Employee{ EmpId= 4, FirstName = "Fleming",LastName= "Mick" },
         new Employee{ EmpId= 5, FirstName = "Vicky",LastName= "Pointing" },
         new Employee{ EmpId= 6, FirstName = "Jonty",LastName= "M" },
         new Employee{ EmpId= 7, FirstName = "Vicky",LastName= "Pointing" },
         new Employee{ EmpId= 8, FirstName = "Fleming",LastName= "Mick" }
        };

Group By

The GroupBy approach is one way to identify duplicate records. Based on a given key, this method groups the table’s rows and returns the groups as a collection. We can group the rows by the columns we want to check for duplicates in order to discover duplicate records, and then filter the groups that contain more than one row.

   public IActionResult Index()
        {

            var Employee = Employees.GroupBy(d => new { d.FirstName, d.LastName })
                .Where(g => g.Count() > 1).Select(g => new { g.Key.FirstName, g.Key.LastName });

            ViewBag.DupRecord = Employee;

            return View();
        }
    }

According to columns FirstName and LastName, the rows in the Employee table are grouped by this code, which then filters the groups with more than one row. A list of groups with duplicate records is the outcome.

Using Any

The Any approach is an additional technique for locating duplicate entries. If any element in a sequence matches a given criterion, this method returns true. We can use the Any method to see if any rows in a table meet the requirements for duplicate records in order to find duplicate records. Here’s an illustration:

  public IActionResult Index()
        {
 
             var dupemployee = Employees.GroupBy(d => new { d.FirstName, d.LastName })
               .Any(g => g.Count() > 1);                

            

            return View();
        }

The Any method is used to determine whether any groups of rows in the Employee table that are grouped by Columns FirstName and LastName have more than one row. The outcome is a boolean value that denotes if the table contains duplicate records.

Conclusion

In this post, we looked at numerous methods for finding duplicate records in ASP.NET Core using Entity Framework. You can quickly and efficiently detect and eliminate duplicate records in your database by using the methods outlined here. This will help to ensure that your application functions properly and gives accurate data to your users.

The post How to Find Duplicate Records Using Entity Framework in Asp.net Core appeared first on Tech Insights.

We need to create sample table

 CREATE TABLE [dbo].[Customers](
	[id] [int] NOT NULL,
	[firstName] [nvarchar](max) NOT NULL,
	[lastName] [nvarchar](max) NOT NULL,
	[job] [nvarchar](max) NOT NULL,
	[amount] [real] NOT NULL,
	[tdate] [datetime2](7) NOT NULL,
	[email] [varchar](25) NULL
	)

 INSERT [dbo].[Customers] ([id], [firstName], [lastName], [job], [amount], [tdate], [email]) VALUES (1, N'Astle', N'Vicky', N'Software Engineer', 50000, CAST(N'2021-01-29T00:00:00.0000000' AS DateTime2), N'astle@example.com')
INSERT [dbo].[Customers] ([id], [firstName], [lastName], [job], [amount], [tdate], [email]) VALUES (2, N'John', N'Vicky', N'software engineer', 55000, CAST(N'2022-12-03T00:00:00.0000000' AS DateTime2), N'John@example.com')
INSERT [dbo].[Customers] ([id], [firstName], [lastName], [job], [amount], [tdate], [email]) VALUES (3, N'Fleming', N'Stuart', N'software engineer', 55000, CAST(N'2022-12-29T00:00:00.0000000' AS DateTime2), N'astle@example.com')
INSERT [dbo].[Customers] ([id], [firstName], [lastName], [job], [amount], [tdate], [email]) VALUES (4, N'Philip', N'John', N'Manager', 650000, CAST(N'2022-12-29T00:00:00.0000000' AS DateTime2), N'ricky@example.com')
INSERT [dbo].[Customers] ([id], [firstName], [lastName], [job], [amount], [tdate], [email]) VALUES (5, N'Ricky', N'Fleming', N'DBA Administrator', 78000, CAST(N'2022-12-29T00:00:00.0000000' AS DateTime2), N'ricky@example.com')
INSERT [dbo].[Customers] ([id], [firstName], [lastName], [job], [amount], [tdate], [email]) VALUES (6, N'mack', N'Mack', N'Manager', 56000, CAST(N'2022-12-20T00:00:00.0000000' AS DateTime2), N'nack@exampl.com')
INSERT [dbo].[Customers] ([id], [firstName], [lastName], [job], [amount], [tdate], [email]) VALUES (7, N'steva', N'margh', N'Team Lead', 456000, CAST(N'2022-12-23T00:00:00.0000000' AS DateTime2), N'steva@example.com')

Here are few SQL methods for finding and removing duplicate records in a table using SQL

GROUP BY and HAVING CLAUSE

Suppose we have a table customer with columns id, firstname, email etc. 

Suppose we have a table customer with columns id, firstname, email etc.  To find and remove duplicate records based on the “email”  column, you can use the following SQL query

SELECT EMAIL,COUNT(*)
FROM Customers GROUP BY EMAIL
HAVING COUNT(*)>1

This query will group the rows in the “Customers” table by the “email” column, and the return the email and count of each group that has more than on row. To delete the duplicate records , you can use the following SQL query

DELETE FROM Customers WHERE id NOT IN(SELECT MAX(id) FROM Customers GROUP BY EMAIL)

This query will delete all records in the customers table that have a duplicate email, except for the one with highest id value

EXISTS CLAUSE

Suppose we have a table customer with columns id, firstname, email etc  To find and remove duplicate records based on the “firstname”  column, you can use the following SQL query

SELECT c.id,c.firstName
FROM Customers c WHERE EXISTS (
SELECT * FROM Customers cust
WHERE c.firstName=cust.firstName
AND c.id > cust.id
)

This query will select all rows in the customers table where there exists another row with the same firstname and a lower id value. To delete the duplicate records , you can use the following SQL query

DELETE c FROM Customers c,Customers cust
WHERE c.firstName=cust.firstName
AND c.id > cust.id

This query will delete all records in the customers table that have a duplicate firstname, except for the one with lowest id value

ROW_NUMBER Function

In this query , I’m using the same example as in the above one, but this time I’m using the ROW_NUMBER Function to find the duplicate data

SELECT id, email , ROW_NUMBER() OVER (
     PARTITION BY email ORDER BY id
	 ) AS row_dup
	 FROM Customers

The query return the id, email and row number for each row from the email column. To remove duplicate rows, modify the query as follows

WITH CT AS(
SELECT id, email , ROW_NUMBER() OVER (
     PARTITION BY email ORDER BY id
	 ) AS row_dup
	 FROM Customers
	 )
	 DELETE FROM CT WHERE row_dup>1

This query will use a common table expression (CTE) to assign row numbers to each records in the customers table based on the email column, then delete any rows with a row number greater than 1

The post How to Find and Remove Duplicate records in a table in SQL appeared first on Tech Insights.