ASP.NET Core 5.0 - Log Maintenance with Hangfire

Ken Haggerty
Created 01/06/2021 - Updated 01/06/2021 06:05

This article will describe the implementation of purging stale analytic records with Hangfire. I will assume you have downloaded the ASP.NET Core 5.0 - Homegrown Analytics Project or created a new ASP.NET Core 5.0 Razor Pages project. See Tutorial: Get started with Razor Pages in ASP.NET Core.

Homegrown Analytics Project and Article Series

The project is feature complete and will increment the version with updates. The details page includes the version change log. This project is designed and organized to allow easy integration with existing projects. The KenHaggerty. Com. SingleUser NuGet package segregates the user logic. SQL scripts to create the schema, tables, and default data are included in the package. Details, screenshots, and related articles can be found at ASP.NET Core 5.0 - Homegrown Analytics Project.

Online log maintenance like purging old records are usually long running tasks. Hangfire is better than Background tasks with hosted services for fire and forget tasks. Hangfire is easy to implement. The project has an option to enable Hangfire. The Startup class has static properties for options.

Startup.cs:
// Enables the Hangfire database and Dashboard.
// Search project for EnableHangfire to locate commented declarations and functions
// The database must exist for Hangfire migration
public static bool EnableHangfire { get; } = true;

Hangfire with SQL requires 3 NuGet packages: Hangfire. AspNetCore, Hangfire. Core, and Hangfire. SqlServer.

Hangfire NuGet Packages

The project's Hangfire declarations and functions are commented. Uncomment the AddHangfire extention to add Hangfire to the services collection. The UseSqlServerStorage extension creates the database schema and tables if not found. Hangfire can utilize an existing or a seperate database by setting the connection string in appsettings.json.

Startup > ConfigureServices:
var analyticsConnection = Configuration.GetConnectionString("HangfireConnection");
// EnableHangfire services.
services.AddHangfire(configuration => configuration
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSqlServerStorage(analyticsConnection, new SqlServerStorageOptions
    {
        CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
        SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
        QueuePollInterval = TimeSpan.Zero,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true
    }));

// Add the processing server as IHostedService
services.AddHangfireServer();

The project maps the Hangfire Dashboard to /admin/hangfire. The Hangfire dashboard provides detailed status on running, completed, and failed jobs. The AppPath sets the Dashboard's Back to site link. The RequireAuthorization extension adds the default Authorization policy. The Authorization = new List<IDashboardAuthorizationFilter> { } seems to be required if deployed. See Securing Hangfire Dashboard in ASP.NET Core with a Custom Auth Policy

Startup > Configure:
app.UseEndpoints(endpoints =>
{
    endpoints.MapRazorPages();

    if (EnableSignalR) endpoints.MapHub<OnlineCountHub>("/onlinecount");

    if (EnableHangfire)
    {
        endpoints.MapHangfireDashboard("/admin/hangfire", new DashboardOptions()
        {
            Authorization = new List<IDashboardAuthorizationFilter> { },
            AppPath = "/admin",
            DashboardTitle = "Homegrown Analytics Hangfire"
        }).RequireAuthorization();
    }
});
Sample Database Tables
Hangfire Dashboard

The project implements a HangfireService with the injected IServiceScopeFactory to create an instance of the AnalyticsService. The AnalyticsService implements functions to purge records older than the keepDate.

Startup > ConfigureServices:
if (EnableHangfire)
    services.AddScoped<IHangfireService, HangfireService>();
Services > HangfireService:
public async Task PurgePageHitsAsync(DateTimeOffset keepDate)
{
    using var scope = _serviceScopeFactory.CreateScope();
    var _analyticsService = scope.ServiceProvider.GetRequiredService<IAnalyticsService>();
    var result = await _analyticsService.PurgePageHitsAsync(keepDate).ConfigureAwait(false);
}

The project uses AnalyticsService functions to count records older than the keepDate. If the count is greater than a purge threshold, a button with the purge count is displayed.

Pages > Admin > Analytics > Pagehits.cshtml:
@if (Startup.EnableHangfire && Model.PurgeCount > configuration.GetValue<int>("PurgeThresholdLarge"))
{
    <form class="d-inline" asp-page-handler="PurgePageHits" method="post">
        <button type="submit" class="btn btn-danger btn-sm">Purge (@Model.PurgeCount.ToString())</button>
    </form>
}
PageHits With Purge Button
PageHits With Purge Button Mobile

The PurgePageHits method with the injected Hangfire IBackgroundJobClient sends the _hangfireService. PurgePageHitsAsync task to Hangfire for execution.

Pages > Admin > Analytics > Pagehits.cshtml:
public async Task<IActionResult> OnPostPurgePageHitsAsync()
{
    // fire and forget
    var jobId = _backgroundJobs
        .Enqueue(() => _hangfireService
        .PurgePageHitsAsync(DateTimeOffset.UtcNow.AddDays(-_configuration.GetValue<int>("PurgeKeepDays"))));

    return RedirectToPage(new { purging = true });
}

PageHits depends on Sessions by foreign key. Sessions is not purged by date but when the principal record is no longer referenced by the dependent table. The project implements the System. Linq. Queryable GroupJoin extension to select orphaned Sessions records with Id > 1. The default Session's Id = 1. PageHits, Sessions, and NotFoundErrors depend on UserAgents by foreign key. The project implements GroupJoin to select UserAgents records which are orphaned by all and with Id > 1. The default UserAgent's Id = 1.

Services > AnalyticsService:
public async Task<int> GetSessionPurgeCountAsync()
{
    return await _dbSessionSet
        .GroupJoin(_dbPageHitSet, s => s.Id, p => p.SessionId, (s, grouping) => new { s, grouping })
        .SelectMany(p => p.grouping.DefaultIfEmpty(), (sp, p) => new { Session = sp.s, PageHit = p })
        .Where(both => both.PageHit == null && both.Session.Id > 1)
        .CountAsync()
        .ConfigureAwait(false);
}

The project implements a Shrink Database function. After purging the analytic data, you can shrink the database if the SQL user has the db_owner role.

Services > AnalyticsService:
public async Task<int> ShrinkDatabaseAsync()
{
    var dbName = _dbContext.Database.GetDbConnection().Database;
    return await _dbContext.Database.ExecuteSqlInterpolatedAsync($"DBCC SHRINKDATABASE({dbName})");
}
After Purge
Database Files Properties After Purge
After Shrink
Database Files Properties After Shrink

After purging and shrinking, Hangfire displays Succeeded Jobs.

Hangfire Purge Jobs Succeeded
Analytics Index Page
Analytics Index Page Mobile

Article Tags:

Analytics EF Core Model

Comment Count = 0

Please log in to comment.

Login Register
Logged in users receive web notifications.
Web Notifications