ASP.NET Core 6.0 - Data Collection

This article will describe the implementation of middleware which captures usage metrics for the current request. I will assume you have downloaded the ASP.NET Core 6.0 - Homegrown Analytics Project.

Homegrown Analytics Project and Article Series

The Homegrown Analytics Project (HAP) is designed and organized to allow easy integration with existing projects. V2 simplifies the process with an Analytics area folder. The analytics schema can be implemented with SQL scripts or EF Core migration. See ASP.NET Core 6.0 - Analytics Schema and Integration. The HAP details page includes the version change log.

ASP.NET Core 6.0

Updated with links as the articles become published.

KenHaggerty.Com has been logging requests for over 3 years. Early on, I stored the UserAgent as a string with every PageHit entity. I implemented a UserAgent entity and upgraded the PageHit's property to the UserAgentId. Analytics middleware creates a new UserAgent if it does not exist. I added a Session entity to capture the first or landing page the user browses on my site. The HAP implements a SessionId cookie which expires when the browsing session ends. If the SessionId cookie does not exist, Analytics middleware creates a new Session and sets a SessionId cookie with the new Id. The HAP implements a cookie consent banner and CheckConsentNeeded from Microsoft. AspNetCore. CookiePolicy to request user consent for non-essential cookies. If granted, I use cookies to record the device screen size and the client's GMT offset. The HAP includes a sample privacy policy page with a revoke cookie consent function. I developed AnonymizeIpAddressExtention.cs to record an anonymous IP address from the HttpContext. Connection. RemoteIpAddress. The AnonymizeIpAddressExtention sets the Host ID (the last segment) of the IP address to zero. See How-To Geek - How Do IP Addresses Work?.

AnonymizeIpAddressExtention.cs
public static class AnonymizeIpAddressExtention
{
    //const string IPV4_NETMASK = "255.255.255.0";
    //const string IPV6_NETMASK = "ffff:ffff:ffff:0000:0000:0000:0000:0000";

    /// <summary>
    /// Removes the unique part of an <see cref="IPAddress" />.
    /// </summary>
    /// <param name="ipAddress"></param>
    /// <returns><see cref="string" /></returns>
    public static string AnonymizeIP(this IPAddress ipAddress)
    {
        string anonymizedIp;
        if (ipAddress != null)
        {
            if (ipAddress.AddressFamily == AddressFamily.InterNetwork)
            {
                var ipString = ipAddress.ToString();
                string[] octets = ipString.Split('.');
                octets[3] = "0";
                anonymizedIp = string.Join(".", octets);
            }
            else if (ipAddress.AddressFamily == AddressFamily.InterNetworkV6)
            {
                var ipString = ipAddress.ToString();
                string[] hextets = ipString.Split(':');
                var hl = hextets.Length;
                if (hl > 3) { for (var i = 3; i < hl; i++) { if (hextets[i].Length > 0) { hextets[i] = "0"; } } }
                anonymizedIp = string.Join(":", hextets);
            }
            else { anonymizedIp = $"Not Valid - {ipAddress}"; }
        }
        else
        {
            anonymizedIp = "Is Null";
        }
        return anonymizedIp;
    }
}

Over time, I added more properties to the PageHit like cookieConsent, screenSize, minutesOffset, method, queryString, refererHeader, and a nullable appUserId. AnalyticsMiddleware collects and analyzes the properties of the HttpContext and Request. AnalyticsServices implements database operations. AnalyticsMiddleware depends on AnalyticsServices using dependency injection. The AnalyticsMiddleware is fault tolerant. The EnsureAnalyticsDefaultData HostedService creates a default UserAgent and a default Session if not found. Both have the primary Id property set to 1. If the AnalyticsMiddleware encounters an issue, the default UserAgent or Session is employed.

AnalyticsMiddleware.cs
public class AnalyticsMiddleware
{
    private readonly RequestDelegate _next;
    public AnalyticsMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(HttpContext context, IAnalyticsService _analyticsService)
    {
        int pageHitId = 0;
        int newSessionId = 0;
        string path = context.Request.Path.ToString().ToLower();
        string method = context.Request.Method.ToUpper();
        string? uaString = context.Request.Headers["User-Agent"].FirstOrDefault();
        if (string.IsNullOrEmpty(uaString)) uaString = "Not Found";
        else
        {
            if (uaString.Length > 512) uaString = uaString[..512];
        }

        int userAgentId = 0;
        try
        {
            userAgentId = await _analyticsService.GetUserAgentIdByUAStringAsync(uaString).ConfigureAwait(false);
        }
        catch (Exception ex)
        {
            if (path != "/error")
                throw new InvalidOperationException("An error occurred getting a UserAgentId. Verify the database connection.", ex);
        }

        // Default UserAgent with UserAgentId = 1 is created with the EnsureAnalyticsDefaultData HostedService.
        if (userAgentId == 0)
        {
            UserAgent userAgent = new(uaString);
            if (await _analyticsService.AddUserAgentAsync(userAgent).ConfigureAwait(false) && userAgent.Id > 0)
                userAgentId = userAgent.Id;
            else userAgentId = 1;
        }

        if (path.Length > 512) path = path[..512];

        string queryString = "Not Found";
        if (context.Request.QueryString.HasValue)
        {
            queryString = context.Request.QueryString.ToString();
            if (queryString.Length > 1024) queryString = queryString[..1024];
        }

        string refererHeader = context.Request.Headers["Referer"].ToString();
        if (string.IsNullOrEmpty(refererHeader)) refererHeader = "Not Found";
        if (refererHeader.Length > 1024) refererHeader = refererHeader[..1024];

        string anonymizedIp = context.Connection.RemoteIpAddress!.AnonymizeIP();

        bool cookieConsent = true;
        ITrackingConsentFeature? consentFeature = context.Features.Get<ITrackingConsentFeature>();
        if (!consentFeature?.CanTrack ?? false) cookieConsent = false;

        int sessionId = 0;
        if (context.Request.Cookies.TryGetValue(AnalyticsSettings.SessionIdCookieName, out string? sessionIdString))
            if (!int.TryParse(sessionIdString, out sessionId)) sessionId = 1;
        double minutesOffset = 0;
        if (context.Request.Cookies.TryGetValue(AnalyticsSettings.TimeOffsetCookieName, out string? minutesOffsetString))
            if (!double.TryParse(minutesOffsetString, out minutesOffset)) minutesOffset = 0;
        _ = context.Request.Cookies.TryGetValue(AnalyticsSettings.ScreenSizeCookieName, out string? screenSize);
        if (string.IsNullOrEmpty(screenSize)) screenSize = "Not Found";

        bool userAuthenticated = context.User.Identity!.IsAuthenticated;
        int? appUserId = userAuthenticated ? int.Parse(context.User.FindFirstValue(ClaimTypes.NameIdentifier)) : null;

        // Default Session with SessionId = 1 is created with the EnsureAnalyticsDefaultData HostedService.
        if (sessionId == 0)
        {
            if (!context.Response.Headers.IsReadOnly && (cookieConsent || AnalyticsSettings.SessionIdCookieIsEssential))
            {
                Session session = new(method, path, queryString, refererHeader, userAgentId, anonymizedIp,
                    cookieConsent, screenSize, Convert.ToInt32(minutesOffset), appUserId);
                if (!await _analyticsService.AddSessionAsync(session).ConfigureAwait(false) || session.Id == 0)
                    sessionId = 1;
                else
                {
                    sessionId = session.Id;
                    context.Response.Cookies.Append(AnalyticsSettings.SessionIdCookieName, sessionId.ToString(),
                        new CookieOptions()
                        {
                            Path = "/",
                            IsEssential = AnalyticsSettings.SessionIdCookieIsEssential,
                            Secure = true,
                            HttpOnly = true
                        });
                }
            }
            else
                sessionId = 1;
        }

        if (!path.Contains("/onlinecount") && !path.Equals("/error/404"))
        {
            PageHit pageHit = new(method, path, queryString, refererHeader, userAgentId, sessionId,
                anonymizedIp, cookieConsent, screenSize, Convert.ToInt32(minutesOffset), appUserId);
            if (await _analyticsService.AddPageHitAsync(pageHit).ConfigureAwait(false)) 
                pageHitId = pageHit.Id;
            else throw new InvalidOperationException($"An error occurred adding a PageHit. Close all browser windows and try again.");
        }

        await _next(context).ConfigureAwait(true);

        if (context.Response.StatusCode == 404 && !path.Equals("/error/404"))
        {
            if (pageHitId > 0)
                if (!await _analyticsService.SetPageHitNotFoundAsync(pageHitId).ConfigureAwait(false)) 
                    throw new InvalidOperationException($"An error occurred setting a PageHit.NotFound.");
            if (newSessionId > 0)
                if (!await _analyticsService.SetSessionStartNotFoundAsync(newSessionId).ConfigureAwait(false)) 
                    throw new InvalidOperationException($"An error occurred setting a Session.NotFound.");
        }
    }
}

Early on, I generated and stored NotFoundError entities from the Error page. Notice the pageHitId and newSessionId which are set on the Request side the pipeline. If the Response side has a 404 StatusCode (Not Found), I update the NotFound property for the PageHit or new Session. See MS Docs - ASP.NET Core Middleware.

I implemented an analytics database schema version to manage the integration with my live sites. I increment the minor version if I add a table column. When I dropped the NotFoundErrors table in HAP@2.0.0, I updated the schema version to 2.0.0. When I migrated a static list of MaliciousIps to a database table which has no relationships, I updated the schema version to 2.1.0. I consider v2.1.0 to be very stable. I implemented an analytics UI version because I plan to add new features based on schema v2.1.0.

Manage Analytics Desktop. Manage Analytics Mobile.
Width
Ken Haggerty
Created 08/28/22
Updated 09/19/22 19:27 GMT

Log In or Reset Quota to read more.

Article Tags:

Analytics EF Core SQL
Successfully completed. Thank you for contributing.
Contribute to enjoy content without advertisments.
Something went wrong. Please try again.

Comments(0)

Loading...
Loading...

Not accepting new comments.

Submit your comment. Comments are moderated.

User Image.
DisplayedName - Member Since ?