ASP.NET Core 6.0 - Data Collection

KenHaggerty.Com has been logging requests for over 3 years. Early on, I stored the UserAgent as a string with every PageHit entity. I implemented a UserAgent entity and upgraded the PageHit's property to the UserAgentId. Analytics middleware creates a new UserAgent if it does not exist. I added a Session entity to capture the first or landing page the user browses on my site. The HAP implements a SessionId cookie which expires when the browsing session ends. If the SessionId cookie does not exist, Analytics middleware creates a new Session and sets a SessionId cookie with the new Id. The HAP implements a cookie consent banner and CheckConsentNeeded from Microsoft. AspNetCore. CookiePolicy to request user consent for non-essential cookies. If granted, I use cookies to record the device screen size and the client's GMT offset. The HAP includes a sample privacy policy page with a revoke cookie consent function. I developed AnonymizeIpAddressExtention.cs to record an anonymous IP address from the HttpContext. Connection. RemoteIpAddress. The AnonymizeIpAddressExtention sets the Host ID (the last segment) of the IP address to zero. See How-To Geek - How Do IP Addresses Work?.

AnonymizeIpAddressExtention.cs

public static class AnonymizeIpAddressExtention
{
    //const string IPV4_NETMASK = "255.255.255.0";
    //const string IPV6_NETMASK = "ffff:ffff:ffff:0000:0000:0000:0000:0000";

    /// <summary>
    /// Removes the unique part of an <see cref="IPAddress" />.
    /// </summary>
    /// <param name="ipAddress"></param>
    /// <returns><see cref="string" /></returns>
    public static string AnonymizeIP(this IPAddress ipAddress)
    {
        string anonymizedIp;
        if (ipAddress != null)
        {
            if (ipAddress.AddressFamily == AddressFamily.InterNetwork)
            {
                var ipString = ipAddress.ToString();
                string[] octets = ipString.Split('.');
                octets[3] = "0";
                anonymizedIp = string.Join(".", octets);
            }
            else if (ipAddress.AddressFamily == AddressFamily.InterNetworkV6)
            {
                var ipString = ipAddress.ToString();
                string[] hextets = ipString.Split(':');
                var hl = hextets.Length;
                if (hl > 3) { for (var i = 3; i < hl; i++) { if (hextets[i].Length > 0) { hextets[i] = "0"; } } }
                anonymizedIp = string.Join(":", hextets);
            }
            else { anonymizedIp = $"Not Valid - {ipAddress}"; }
        }
        else
        {
            anonymizedIp = "Is Null";
        }
        return anonymizedIp;
    }
}

Over time, I added more properties to the PageHit like cookieConsent, screenSize, minutesOffset, method, queryString, refererHeader, and a nullable appUserId. AnalyticsMiddleware collects and analyzes the properties of the HttpContext and Request. AnalyticsServices implements database operations. AnalyticsMiddleware depends on AnalyticsServices using dependency injection. The AnalyticsMiddleware is fault tolerant. The EnsureAnalyticsDefaultData HostedService creates a default UserAgent and a default Session if not found. Both have the primary Id property set to 1. If the AnalyticsMiddleware encounters an issue, the default UserAgent or Session is employed.

AnalyticsMiddleware.cs

public class AnalyticsMiddleware
{
    private readonly RequestDelegate _next;
    public AnalyticsMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(HttpContext context, IAnalyticsService _analyticsService)
    {
        int pageHitId = 0;
        int newSessionId = 0;
        string path = context.Request.Path.ToString().ToLower();
        string method = context.Request.Method.ToUpper();
        string? uaString = context.Request.Headers["User-Agent"].FirstOrDefault();
        if (string.IsNullOrEmpty(uaString)) uaString = "Not Found";
        else
        {
            if (uaString.Length > 512) uaString = uaString[..512];
        }

        int userAgentId = 0;
        try
        {
            userAgentId = await _analyticsService.GetUserAgentIdByUAStringAsync(uaString).ConfigureAwait(false);
        }
        catch (Exception ex)
        {
            if (path != "/error")
                throw new InvalidOperationException("An error occurred getting a UserAgentId. Verify the database connection.", ex);
        }

        // Default UserAgent with UserAgentId = 1 is created with the EnsureAnalyticsDefaultData HostedService.
        if (userAgentId == 0)
        {
            UserAgent userAgent = new(uaString);
            if (await _analyticsService.AddUserAgentAsync(userAgent).ConfigureAwait(false) && userAgent.Id > 0)
                userAgentId = userAgent.Id;
            else userAgentId = 1;
        }

        if (path.Length > 512) path = path[..512];

        string queryString = "Not Found";
        if (context.Request.QueryString.HasValue)
        {
            queryString = context.Request.QueryString.ToString();
            if (queryString.Length > 1024) queryString = queryString[..1024];
        }

        string refererHeader = context.Request.Headers["Referer"].ToString();
        if (string.IsNullOrEmpty(refererHeader)) refererHeader = "Not Found";
        if (refererHeader.Length > 1024) refererHeader = refererHeader[..1024];

        string anonymizedIp = context.Connection.RemoteIpAddress!.AnonymizeIP();

        bool cookieConsent = true;
        ITrackingConsentFeature? consentFeature = context.Features.Get<ITrackingConsentFeature>();
        if (!consentFeature?.CanTrack ?? false) cookieConsent = false;

        int sessionId = 0;
        if (context.Request.Cookies.TryGetValue(AnalyticsSettings.SessionIdCookieName, out string? sessionIdString))
            if (!int.TryParse(sessionIdString, out sessionId)) sessionId = 1;
        double minutesOffset = 0;
        if (context.Request.Cookies.TryGetValue(AnalyticsSettings.TimeOffsetCookieName, out string? minutesOffsetString))
            if (!double.TryParse(minutesOffsetString, out minutesOffset)) minutesOffset = 0;
        _ = context.Request.Cookies.TryGetValue(AnalyticsSettings.ScreenSizeCookieName, out string? screenSize);
        if (string.IsNullOrEmpty(screenSize)) screenSize = "Not Found";

        bool userAuthenticated = context.User.Identity!.IsAuthenticated;
        int? appUserId = userAuthenticated ? int.Parse(context.User.FindFirstValue(ClaimTypes.NameIdentifier)) : null;

        // Default Session with SessionId = 1 is created with the EnsureAnalyticsDefaultData HostedService.
        if (sessionId == 0)
        {
            if (!context.Response.Headers.IsReadOnly && (cookieConsent || AnalyticsSettings.SessionIdCookieIsEssential))
            {
                Session session = new(method, path, queryString, refererHeader, userAgentId, anonymizedIp,
                    cookieConsent, screenSize, Convert.ToInt32(minutesOffset), appUserId);
                if (!await _analyticsService.AddSessionAsync(session).ConfigureAwait(false) || session.Id == 0)
                    sessionId = 1;
                else
                {
                    sessionId = session.Id;
                    context.Response.Cookies.Append(AnalyticsSettings.SessionIdCookieName, sessionId.ToString(),
                        new CookieOptions()
                        {
                            Path = "/",
                            IsEssential = AnalyticsSettings.SessionIdCookieIsEssential,
                            Secure = true,
                            HttpOnly = true
                        });
                }
            }
            else
                sessionId = 1;
        }

        if (!path.Contains("/onlinecount") && !path.Equals("/error/404"))
        {
            PageHit pageHit = new(method, path, queryString, refererHeader, userAgentId, sessionId,
                anonymizedIp, cookieConsent, screenSize, Convert.ToInt32(minutesOffset), appUserId);
            if (await _analyticsService.AddPageHitAsync(pageHit).ConfigureAwait(false)) 
                pageHitId = pageHit.Id;
            else throw new InvalidOperationException($"An error occurred adding a PageHit. Close all browser windows and try again.");
        }

        await _next(context).ConfigureAwait(true);

        if (context.Response.StatusCode == 404 && !path.Equals("/error/404"))
        {
            if (pageHitId > 0)
                if (!await _analyticsService.SetPageHitNotFoundAsync(pageHitId).ConfigureAwait(false)) 
                    throw new InvalidOperationException($"An error occurred setting a PageHit.NotFound.");
            if (newSessionId > 0)
                if (!await _analyticsService.SetSessionStartNotFoundAsync(newSessionId).ConfigureAwait(false)) 
                    throw new InvalidOperationException($"An error occurred setting a Session.NotFound.");
        }
    }
}

Early on, I generated and stored NotFoundError entities from the Error page. Notice the pageHitId and newSessionId which are set on the Request side the pipeline. If the Response side has a 404 StatusCode (Not Found), I update the NotFound property for the PageHit or new Session. See MS Docs - ASP.NET Core Middleware.

I implemented an analytics database schema version to manage the integration with my live sites. I increment the minor version if I add a table column. When I dropped the NotFoundErrors table in HAP@2.0.0, I updated the schema version to 2.0.0. When I migrated a static list of MaliciousIps to a database table which has no relationships, I updated the schema version to 2.1.0. I consider v2.1.0 to be very stable. I implemented an analytics UI version because I plan to add new features based on schema v2.1.0.