ASP.NET Core 5.0 - Log Maintenance with Hangfire

This article will describe the implementation of purging stale analytic records with Hangfire. I will assume you have downloaded the ASP.NET Core 5.0 - Homegrown Analytics Project or created a new ASP.NET Core 5.0 Razor Pages project. See Tutorial: Get started with Razor Pages in ASP.NET Core.

Online log maintenance like purging old records are usually long running tasks. Hangfire is better than Background tasks with hosted services for fire and forget tasks. Hangfire is easy to implement. The project has an option to enable Hangfire. The Startup class has static properties for options.

Startup.cs:

// Enables the Hangfire database and Dashboard.
// Search project for EnableHangfire to locate commented declarations and functions
// The database must exist for Hangfire migration
public static bool EnableHangfire { get; } = true;

Hangfire with SQL requires 3 NuGet packages: Hangfire. AspNetCore, Hangfire. Core, and Hangfire. SqlServer.

The project's Hangfire declarations and functions are commented. Uncomment the AddHangfire extention to add Hangfire to the services collection. The UseSqlServerStorage extension creates the database schema and tables if not found. Hangfire can utilize an existing or a seperate database by setting the connection string in appsettings.json.

Startup > ConfigureServices:

var analyticsConnection = Configuration.GetConnectionString("HangfireConnection");
// EnableHangfire services.
services.AddHangfire(configuration => configuration
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSqlServerStorage(analyticsConnection, new SqlServerStorageOptions
    {
        CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
        SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
        QueuePollInterval = TimeSpan.Zero,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true
    }));

// Add the processing server as IHostedService
services.AddHangfireServer();

The project maps the Hangfire Dashboard to /admin/hangfire. The Hangfire dashboard provides detailed status on running, completed, and failed jobs. The AppPath sets the Dashboard's Back to site link. The RequireAuthorization extension adds the default Authorization policy. The Authorization = new List<IDashboardAuthorizationFilter> { } seems to be required if deployed. See Securing Hangfire Dashboard in ASP.NET Core with a Custom Auth Policy

Startup > Configure:

app.UseEndpoints(endpoints =>
{
    endpoints.MapRazorPages();

    if (EnableSignalR) endpoints.MapHub<OnlineCountHub>("/onlinecount");

    if (EnableHangfire)
    {
        endpoints.MapHangfireDashboard("/admin/hangfire", new DashboardOptions()
        {
            Authorization = new List<IDashboardAuthorizationFilter> { },
            AppPath = "/admin",
            DashboardTitle = "Homegrown Analytics Hangfire"
        }).RequireAuthorization();
    }
});

The project implements a HangfireService with the injected IServiceScopeFactory to create an instance of the AnalyticsService. The AnalyticsService implements functions to purge records older than the keepDate.

Startup > ConfigureServices:

if (EnableHangfire)
    services.AddScoped<IHangfireService, HangfireService>();

Services > HangfireService:

public async Task PurgePageHitsAsync(DateTimeOffset keepDate)
{
    using var scope = _serviceScopeFactory.CreateScope();
    var _analyticsService = scope.ServiceProvider.GetRequiredService<IAnalyticsService>();
    var result = await _analyticsService.PurgePageHitsAsync(keepDate).ConfigureAwait(false);
}

The project uses AnalyticsService functions to count records older than the keepDate. If the count is greater than a purge threshold, a button with the purge count is displayed.

Pages > Admin > Analytics > Pagehits.cshtml:

@if (Startup.EnableHangfire && Model.PurgeCount > configuration.GetValue<int>("PurgeThresholdLarge"))
{
    <form class="d-inline" asp-page-handler="PurgePageHits" method="post">
        <button type="submit" class="btn btn-danger btn-sm">Purge (@Model.PurgeCount.ToString())</button>
    </form>
}

The PurgePageHits method with the injected Hangfire IBackgroundJobClient sends the _hangfireService. PurgePageHitsAsync task to Hangfire for execution.

Pages > Admin > Analytics > Pagehits.cshtml:

public async Task<IActionResult> OnPostPurgePageHitsAsync()
{
    // fire and forget
    var jobId = _backgroundJobs
        .Enqueue(() => _hangfireService
        .PurgePageHitsAsync(DateTimeOffset.UtcNow.AddDays(-_configuration.GetValue<int>("PurgeKeepDays"))));

    return RedirectToPage(new { purging = true });
}

PageHits depends on Sessions by foreign key. Sessions is not purged by date but when the principal record is no longer referenced by the dependent table. The project implements the System. Linq. Queryable GroupJoin extension to select orphaned Sessions records with Id > 1. The default Session's Id = 1. PageHits, Sessions, and NotFoundErrors depend on UserAgents by foreign key. The project implements GroupJoin to select UserAgents records which are orphaned by all and with Id > 1. The default UserAgent's Id = 1.

Services > AnalyticsService:

public async Task<int> GetSessionPurgeCountAsync()
{
    return await _dbSessionSet
        .GroupJoin(_dbPageHitSet, s => s.Id, p => p.SessionId, (s, grouping) => new { s, grouping })
        .SelectMany(p => p.grouping.DefaultIfEmpty(), (sp, p) => new { Session = sp.s, PageHit = p })
        .Where(both => both.PageHit == null && both.Session.Id > 1)
        .CountAsync()
        .ConfigureAwait(false);
}

The project implements a Shrink Database function. After purging the analytic data, you can shrink the database if the SQL user has the db_owner role.

Services > AnalyticsService:

public async Task<int> ShrinkDatabaseAsync()
{
    var dbName = _dbContext.Database.GetDbConnection().Database;
    return await _dbContext.Database.ExecuteSqlInterpolatedAsync($"DBCC SHRINKDATABASE({dbName})");
}

After purging and shrinking, Hangfire displays Succeeded Jobs.