ASP.NET Core 6.0 - Homegrown Analytics

I always thought of logging page hits as complimentary to Google Analytics. By adding a SessionId cookie which expires when the browsing session ends and ASP.NET Core SignalR for monitoring active connections (online users), I have gotten closer to a Google Analytics alternative. This series will explore some of the issues and considerations for collecting and analyzing usage metrics.

The project has been upgraded to ASP.NET Core 6.0 with Visual Studio 2022 from the ASP.NET Core 5.0 project. I migrated the Startup.cs to top-level statements in Program.cs. I also implemented global usings and file scoped namespaces. I have enabled the nullable context and mitigated all warnings and issues. I have updated the project with Twitter Bootstrap v5. See ASP.NET Core 5.0 - Migrate To Bootstrap v5. This project implements Bootstrap Native. The article series continues the ASP.NET Core 5.0 series with an introduction to the ASP.NET Core 6.0 - Homegrown Analytics Project. Version 2 implements a new schema and UI.

The Analytics middleware collects and analyzes the properties of the HttpContext and Request. The middleware creates and stores a new Session if the SessionId cookie is not found. The middleware looks for a known UserAgent and creates and stores a new UserAgent if not found. The middleware creates and stores a new PageHit and updates the NotFound property on 404 Response status.

PageHit pageHit = new(method, path, queryString, refererHeader, userAgentId, sessionId, anonymizedIp, cookieConsent, screenSize, Convert.ToInt32(minutesOffset), userId);

Integration

After implementing and updating analytics with my live sites multiple times, I have refactored the HAP V2 with an updated analytics schema and UI. The integration process is simplified with an Analytics area folder. After the AnalyticsDbContext is configured, you can use the Package Manager Console to to execute "update-database -Context AnalyticsDbContext" to apply the analytics schema migration. The CreateDatabaseAndAnalyticsSchema.sql and CreateAnalyticsSchema.sql scripts are located in the Areas/ Analytics/ Data/ SqlScripts folder. The PurgeExpiredAnalytics script deletes expired related records. The RemoveAnalyticsSchema will remove all analytics and Hangfire data and schema. The EnsureAnalyticsDefaultData HostedService sets the AnalyticsSchema version and creates the required default UserAgent and Session. The KenHaggerty. Com. SingleUser. Net6 NuGet package segregates the user logic. See ASP.NET Core 6.0 - Analytics Schema and Integration.

Data Collection

The HAP implements a cookie consent banner and CheckConsentNeeded from Microsoft. AspNetCore. CookiePolicy to request user consent for non-essential cookies. If granted, I use cookies to capture screen size and time zone. The project includes a sample privacy policy page with a revoke cookie consent function. I consider the SessionId cookie which expires when the browsing session ends as essential. The middleware collects and analyzes the properties of the HttpContext and Request. The middleware creates and stores a new Session if the SessionId cookie is not found. The middleware looks for a known UserAgent and creates and stores a new UserAgent if not found. The middleware creates and stores a new PageHit and updates the NotFound property on 404 Response status. If cookie consent is granted, the screen size and GMT time offset are recorded. I capture the HttpContext. Connection. RemoteIpAddress but I make it anonymous by blanking the Host ID (the last segment) with zero. See How-To Geek - How Do IP Addresses Work?.

Data Integrity

The analytics schema includes foreign keys which allow Fluent API queries on related entities. I use a nullable foreign key on AppUserId which allows anonymous and authenticated sessions and page hits. Finding accurate information on creating indexes, foreign keys, and nullable foreign keys is difficult. Deleting related records must be carefully implemented and tested. If a user deletes themselves, the AppUserId on Session and PageHit must be set to null before the AppUser can be deleted.

Data Visualization

I added filters to chronicle lists of sessions and page hits but this did not represent trends over time. HAP V1 had a client screen size page which displayed a bubble chart implemented with D3 - Data-Driven Documents. HAP V2 implements Chart.js. Both are JavaScript libraries. Chart.js is canvas based and D3.js is svg based. I updated the Dashboard to display a line chart of the number of PageHits over time. The time period is selected from Segments: 24 Hours, 7 Days, 5 Weeks, 6 Months, and 4 Quarters. When data history allows, a data series of the previous segment is automictically added to the line chart. HAP V2 implements a svg world map from amCharts - Free SVG Maps, NuGet - MaxMind.Db, and IP to Country Lite database to highlight the map with the top 10 countries with the most page hits. The optional _AnalyticsLayout .cshtml with a 1900px wide display container has been updated for easier integration.

Preferred UserAgents

The UserAgent has a Preferred property. I developed a UserAgent chart page which displays the UserAgent's PageHits count over time and provides the functions to add or remove the preferred status. Rather than trying to detect robot page hits, I can filter on Preferred UserAgents.

Homegrown Tag Helpers

Filtering a boolean property with a checkbox facilitates 2 states: include and only. I developed a group of radio buttons to allow 3 states: include, exclude, and only. I use this boolean filter on many pages for a lot of properties. Creating a custom TagHelper is easier than initial research suggests. After implementing the BoolFilterTagHelper, I developed SegmentFilterTagHelper, DaysFilterTagHelper, and CurrentIpFilterTagHelper. The DatalistSelectTagHelper generates a select dropdown with an auto complete feature.

Client End Of Day

The data is stored with a created DateTimeOffset set to the UTC date and time. When the TimeOffset cookie is not found, the charts use the GMT end of day. With the TimeOffset, the charts and Segments represent the client's end of day. To accurately calculate the end of week, month, and quarter you need to start with an accurate client's end of day. My hosting server is in the Pacific time zone and I develop in the Eastern time zone. I had to publish and test multiple times before getting this right.

ASP.NET Core SignalR

I use Google Analytic's Active Users feature mainly to detect if any users are online before I take the site offline for updates. I implemented and deployed an online user feature based on a couple of examples I found which count online users with SignalR. AnalyticsSettings.cs has an option to enable SignalR and the online user feature. The option is disabled by default.

Google reCAPTCHA

Before I implemented Google reCAPTCHA v3 on the contact and register pages, I was getting emails and users created by robots. Since, I haven't noticed any. You must register your site to receive an API key and secret. You set the secret, key, and a pass/fail threshold in AnalyticsSettings.cs. reCAPTCHA v3 returns a score (1.0 is very likely a good interaction, 0.0 is very likely a bot). AnalyticsSettings.cs has an option to enable Google reCAPTCHA. The option is disabled by default.

Hangfire

Online log maintenance like purging old records are usually long running tasks. Hangfire is better than Background tasks for fire and forget tasks. Hangfire is easy to implement. The Hangfire dashboard provides detailed status on running, completed, and failed jobs. AnalyticsSettings.cs has an option to enable Hangfire. The option is disabled by default.