ASP.NET Core 6.0 - Homegrown Analytics
Mac Mini Fund
At least 90% of all new contributions and access fees goes to the fund.
This article introduces the migrated ASP.NET Core 6.0 - Homegrown Analytics V2 project and continues the article series about collecting and analyzing a web application's usage metrics. I will assume you have downloaded the ASP.NET Core 6.0 - Homegrown Analytics Project.
Homegrown Analytics Project and Article Series
The Homegrown Analytics Project (HAP) is designed and organized to allow easy integration with existing projects. V2 simplifies the process with an Analytics area folder. The analytics schema can be implemented with SQL scripts or EF Core migration. See ASP.NET Core 6.0 - Analytics Schema and Integration. The HAP details page includes the version change log.
ASP.NET Core 6.0
- ASP.NET Core 6.0 - Homegrown Analytics
- ASP.NET Core 6.0 - Analytics Schema and Integration
- ASP.NET Core 6.0 - Data Collection
- ASP.NET Core 6.0 - Data Filters
- ASP.NET Core 6.0 - Data Visualization
- ASP.NET Core 6.0 - World Heat Map
- ASP.NET Core 6.0 - Homegrown TagHelpers
- ASP.NET Core 6.0 - Client End Of Day
ASP.NET Core 5.0
- ASP.NET Core 5.0 - Homegrown Analytics
- ASP.NET Core 5.0 - Analytics Schema and Integration
- ASP.NET Core 5.0 - Cookie Consent and GDPR
- ASP.NET Core 5.0 - Analytic Data Collection
- ASP.NET Core 5.0 - Not Found Errors
- ASP.NET Core 5.0 - Preferred UserAgents
- ASP.NET Core 5.0 - Analytic Dashboard
- ASP.NET Core 5.0 - Online User Count with SignalR
- ASP.NET Core 5.0 - Is Human with Google reCAPTCHA
- ASP.NET Core 5.0 - Log Maintenance with Hangfire
I always thought of logging page hits as complimentary to Google Analytics. By adding a SessionId cookie which expires when the browsing session ends and ASP.NET Core SignalR for monitoring active connections (online users), I have gotten closer to a Google Analytics alternative. This series will explore some of the issues and considerations for collecting and analyzing usage metrics.
The project has been upgraded to ASP.NET Core 6.0 with Visual Studio 2022 from the ASP.NET Core 5.0 project. I migrated the Startup.cs to top-level statements in Program.cs. I also implemented global usings and file scoped namespaces. I have enabled the nullable context and mitigated all warnings and issues. I have updated the project with Twitter Bootstrap v5. See ASP.NET Core 5.0 - Migrate To Bootstrap v5. This project implements Bootstrap Native. The article series continues the ASP.NET Core 5.0 series with an introduction to the ASP.NET Core 6.0 - Homegrown Analytics Project. Version 2 implements a new schema and UI.
The Analytics middleware collects and analyzes the properties of the HttpContext and Request. The middleware creates and stores a new Session if the SessionId cookie is not found. The middleware looks for a known UserAgent and creates and stores a new UserAgent if not found. The middleware creates and stores a new PageHit and updates the NotFound property on 404 Response status.
PageHit pageHit = new(method, path, queryString, refererHeader, userAgentId, sessionId, anonymizedIp, cookieConsent, screenSize, Convert.ToInt32(minutesOffset), userId);
After implementing and updating analytics with my live sites multiple times, I have refactored the HAP V2 with an updated analytics schema and UI. The integration process is simplified with an Analytics area folder. After the AnalyticsDbContext is configured, you can use the Package Manager Console to to execute "update-database -Context AnalyticsDbContext" to apply the analytics schema migration. The CreateDatabaseAndAnalyticsSchema.sql and CreateAnalyticsSchema.sql scripts are located in the Areas/ Analytics/ Data/ SqlScripts folder. The PurgeExpiredAnalytics script deletes expired related records. The RemoveAnalyticsSchema will remove all analytics and Hangfire data and schema. The EnsureAnalyticsDefaultData HostedService sets the AnalyticsSchema version and creates the required default UserAgent and Session. The KenHaggerty. Com. SingleUser. Net6 NuGet package segregates the user logic. See ASP.NET Core 6.0 - Analytics Schema and Integration.
The analytics schema includes foreign keys which allow Fluent API queries on related entities. I use a nullable foreign key on AppUserId which allows anonymous and authenticated sessions and page hits. Finding accurate information on creating indexes, foreign keys, and nullable foreign keys is difficult. Deleting related records must be carefully implemented and tested. If a user deletes themselves, the AppUserId on Session and PageHit must be set to null before the AppUser can be deleted.
The UserAgent has a Preferred property. I developed a UserAgent chart page which displays the UserAgent's PageHits count over time and provides the functions to add or remove the preferred status. Rather than trying to detect robot page hits, I can filter on Preferred UserAgents.
Homegrown Tag Helpers
Filtering a boolean property with a checkbox facilitates 2 states: include and only. I developed a group of radio buttons to allow 3 states: include, exclude, and only. I use this boolean filter on many pages for a lot of properties. Creating a custom TagHelper is easier than initial research suggests. After implementing the BoolFilterTagHelper, I developed SegmentFilterTagHelper, DaysFilterTagHelper, and CurrentIpFilterTagHelper. The DatalistSelectTagHelper generates a select dropdown with an auto complete feature.
Client End Of Day
The data is stored with a created DateTimeOffset set to the UTC date and time. When the TimeOffset cookie is not found, the charts use the GMT end of day. With the TimeOffset, the charts and Segments represent the client's end of day. To accurately calculate the end of week, month, and quarter you need to start with an accurate client's end of day. My hosting server is in the Pacific time zone and I develop in the Eastern time zone. I had to publish and test multiple times before getting this right.
ASP.NET Core SignalR
I use Google Analytic's Active Users feature mainly to detect if any users are online before I take the site offline for updates. I implemented and deployed an online user feature based on a couple of examples I found which count online users with SignalR. AnalyticsSettings.cs has an option to enable SignalR and the online user feature. The option is disabled by default.
Before I implemented Google reCAPTCHA v3 on the contact and register pages, I was getting emails and users created by robots. Since, I haven't noticed any. You must register your site to receive an API key and secret. You set the secret, key, and a pass/fail threshold in AnalyticsSettings.cs. reCAPTCHA v3 returns a score (1.0 is very likely a good interaction, 0.0 is very likely a bot). AnalyticsSettings.cs has an option to enable Google reCAPTCHA. The option is disabled by default.
Online log maintenance like purging old records are usually long running tasks. Hangfire is better than Background tasks for fire and forget tasks. Hangfire is easy to implement. The Hangfire dashboard provides detailed status on running, completed, and failed jobs. AnalyticsSettings.cs has an option to enable Hangfire. The option is disabled by default.