ASP.NET Core 5.0 - Analytic Data Collection

This article will describe the implementation of middleware which captures usage metrics for the current request. I will assume you have downloaded the ASP.NET Core 5.0 - Homegrown Analytics Project or created a new ASP.NET Core 5.0 Razor Pages project. See Tutorial: Get started with Razor Pages in ASP.NET Core.

The project captures the HttpContext. Connection. RemoteIpAddress but I make it anonymous by blanking the Host ID (the last segment) with zero. See How-To Geek - How Do IP Addresses Work?

Services > AnonymizeIpAddressExtention:

public static class AnonymizeIpAddressExtention
{
    //const string IPV4_NETMASK = "255.255.255.0";
    //const string IPV6_NETMASK = "ffff:ffff:ffff:0000:0000:0000:0000:0000";

    /// <summary>
    /// Removes the unique part of an <see cref="IPAddress" />.
    /// </summary>
    /// <param name="ipAddress"></param>
    /// <returns><see cref="string" /></returns>
    public static string AnonymizeIP(this IPAddress ipAddress)
    {
        string anonymizedIp;
        if (ipAddress != null)
        {
            if (ipAddress.AddressFamily == AddressFamily.InterNetwork)
            {
                var ipString = ipAddress.ToString();
                string[] octets = ipString.Split('.');
                octets[3] = "0";
                anonymizedIp = string.Join(".", octets);
            }
            else if (ipAddress.AddressFamily == AddressFamily.InterNetworkV6)
            {
                var ipString = ipAddress.ToString();
                string[] hextets = ipString.Split(':');
                var hl = hextets.Length;
                if (hl > 3) { for (var i = 3; i < hl; i++) { if (hextets[i].Length > 0) { hextets[i] = "0"; } } }
                anonymizedIp = string.Join(":", hextets);
            }
            else { anonymizedIp = $"Not Valid - {ipAddress}"; }
        }
        else
        {
            anonymizedIp = "Is Null";
        }
        return anonymizedIp;
    }
}

The project captures whether the user is authenticated and the request's method, path, query string, and user agent. I developed AnalyticsMiddleware, a service to capture and store the data in a PageHit entity. I developed the AnalyticsService for database operations. The AnalyticsMiddleware depends on AnalyticsServices using dependency injection.

I have upgraded the PageHit entity's UserAgent property from a string to an int. The UserAgentId relates to a child UserAgent entity. The AnalyticsMiddleware and AnalyticsServices creates a new UserAgent if it does not exist. By adding a Session entity, I can capture the first or landing page the user browses on my site. The project implements a SessionId cookie which expires when the browsing session ends. Because of the confusion and interpretations of the GDPR and other privacy laws, my code includes an option to set the SessionId cookie's essential state. If the SessionId cookie does not exist, the AnalyticsMiddleware and AnalyticsServices creates a new Session with the properties collected for the next PageHit and sets a SessionId cookie with the new Id.

The AnalyticsMiddleware is fault tolerant. The initial migration and CreateAnalyticsSchema SQL scripts create a default UserAgent and a default Session. Both have the primary Id property set to 1. If the AnalyticsMiddleware encounters an issue, the default principal record is employed.

The project displays PageHits on an admin page. The page filters on properties and implements a paged table which only takes 10 records per request. The PageHit depends on the Session. The PageHit, Session, NotFoundError depend on UserAgent. The relationships allow efficient storage and filtering. I address the NotFoundError in the ASP.NET Core 5.0 - Not Found Errors article.

The admin pages for listing PageHits, Sessions, NotFoundErrors, and CaptchaChallenges are paged tables. I address the CaptchaChallenge in the ASP.NET Core 5.0 - Is Human with Google reCAPTCHA article. All listing pages have a count based on a history filter of 1, 3, 7, or 30 days. The paging and CreateDate index produce responsive queries. I implemented the new schema when I integrated the project with KenHaggerty.Com. I am evaluating the responsiveness as I gain history.

KenHaggerty.Com has been recording PageHits for almost 2 years. I started out purging the log with SQL Server Management Studio (SSMS). I often neglected the task which made matters worse. I developed functions to alert on the listing pages when the number of stale records passed a threshold. Maintaining a single table log is not difficult. Maintaining tables which depend on child (principal) tables is not difficult. Purging records from the principal tables is a little more complicated. The project implements functions to purge orphaned principal records without deleting the default principal records.

Attempts to purge the records from the application either failed or became long running tasks. Hangfire is better than Background tasks with hosted services for fire and forget tasks. Hangfire is easy to implement. The Hangfire dashboard provides detailed status on running, completed, and failed jobs. The project has an option to enable Hangfire. The option is disabled by default. I address Hangfire in the ASP.NET Core 5.0 - Log Maintenance with Hangfire article.