Merge remote-tracking branch 'origin/main' into blazor

.github/workflows/tests.yaml

@@ -7,7 +7,7 @@ on:
     branches: [ main ]
 
 env:
-  VERSION: 3.0.0.0-beta6
+  VERSION: 3.0.0.0-beta7
 
 jobs:
   build:

README.md

@@ -116,6 +116,14 @@ You will likely make use of SKSE/F4SE/OBSE or other Script Extender if you are m
 
 The user then only has to copy the files from the `Game Folder Files` directory to their game folder after installation. This also works for other files that need to be installed directly to the game folder like ENB or ReShade.
 
+#### Stock Game
+
+As an alternative to Game Folder Files, you may instead utilize the [Stock Game](https://github.com/wabbajack-tools/wabbajack/wiki/Keeping-The-Game-Folder-Clean-(via-local-game-installs)) feature. Stock Game is a folder within the MO2 instance folder that contains the base game files necessary to run a list, including any files you need to add to it such as `skse64_loader.exe`. Stock Game eliminates the need for the user to copy `Game Folder Files` into their game's installation directory and keeps the final installed list separate from the game's installation directory. Additionally, this allows a list author fine control over what files are included by default in the game's installation, such as SKSE, ENB, and so on. 
+
+Note that Stock Game has one downside: When using Stock Game, Wabbajack will expect the game to come from a single source and *only* a single source (Steam, GOG, etc.). As an example: Morrowind is available on GOG, Steam, and Beth.net. When utilizing Stock Game with a copy of Morrowind from Steam, Wabbajack will expect all users who install the list to have the game through Steam, effectively excluding users who have purchased the game from GOG or Beth.net. Because of this, it is recommended that Stock Game only be utilized with games that have a single source, such as Skyrim SE which is only available through Steam.
+
+Detailed setup information for Stock Game on Skyrim SE can be found [here.](https://github.com/LivelyDismay/Learn-To-Mod/blob/main/lessons/Setting%20up%20Stock%20Game%20for%20Skyrim%20SE.md)
+
 #### Special Flags
 
 There are some special cases where you want to change the default Wabbajack behavior for a specific mod. You can place the following flags in the notes or comments section of a mod to change how Wabbajack handles that mod.
@@ -152,7 +160,11 @@ You don't even have to tell Wabbajack that a specific file was modified, that wo
 
 #### BSA Decomposition
 
-Wabbajack is able to analyse `.bsa` and `.ba2` files and can also create those. This means that you can compress files into a BSA and Wabbajack will index that BSA, match the files against the downloads and re-create the BSA during installation for the user.
+Wabbajack is able to analyse `.bsa` and `.ba2` files and can also create those. This means that any changes made to a BSA/BA2 are copied on the end-user's installation, including the creation of new BSAs or the modification of existing ones. Unpacking, modifying, and then repacking a BSA is possible and those modifications will carry over to the final install. Similarly, creation of new BSAs will be replicated. Wabbajack does not analyze the BSA as a whole - instead it will look inside the BSA, hash the files within, and compare those to their original sources. It will then construct the BSA if new or reconstruct it if modified as necessary. Note that with particularly large BSAs this building and rebuilding process can take a significant amount of time on the user's machine depending on the user's hardware specifications.
+
+#### Merges
+
+Similar to BSAs, Wabbajack is able to analyze and create merged mods. Wabbajack will source the necessary files to create the merge from aforementioned sources of mods and then rebuild the merge on the user's end. Generally it is recommended to tag merges with the `WABBAJACK_NOMATCH_INCLUDE` flag to ensure that any files generated by the merging process are also carried over to the user. Wabbajack does not by default include the files and information necessary to build or rebuild the merge in the program used to build the merge originally, only the merge itself will be installed. 
 
 #### Multiple MO2 Profiles
 
@@ -174,7 +186,9 @@ We already talked about how Wabbajack will _match_ files to determine where they
 
 ### Compilation
 
-Compilation itself is very straight forward and similar to Installation: you let Wabbajack run and hope it finishes successfully. The key for making it run successfully is having a "Wabbajack-compliant" MO2 setup which you can get by following the advices from the previous sections.
+Compilation itself is very straight forward and similar to Installation: you let Wabbajack run and hope it finishes successfully. The key for making it run successfully is having a "Wabbajack-compliant" MO2 setup which you can get by following the advice from the previous sections.
+
+For a more detailed guide on executing the compilation process, refer to [this document.](https://github.com/LivelyDismay/Learn-To-Mod/blob/main/lessons/Compiling%20a%20Modlist%20for%20Wabbajack.md)
 
 #### Wabbajack Compilation Configuration
 
@@ -250,7 +264,7 @@ modID=3863
 fileID=1000172397
 ```
 
-This is the basic configuration for an archive that game from Nexus Mods. You can get these types of `.meta` files through MO2 by using the _Query Info_ option in the Downloads tab.
+This is the basic configuration for an archive that came from Nexus Mods. You can get these types of `.meta` files through MO2 by using the _Query Info_ option in the Downloads tab.
 
 Mods can also be hosted somewhere else, eg on other modding sites like LoversLab, ModDB or normal file hosting services like GDrive and MEGA. Below is a table for all supported sites that Wabbajack can download from.
 

Wabbajack.CLI/Program.cs

@@ -67,6 +67,7 @@ internal class Program
                 services.AddSingleton<IVerb, SteamLogin>();
                 services.AddSingleton<IVerb, SteamAppDumpInfo>();
                 services.AddSingleton<IVerb, SteamDownloadFile>();
+                services.AddSingleton<IVerb, UploadToNexus>();
 
                 services.AddSingleton<IUserInterventionHandler, UserInterventionHandler>();
             }).Build();

Wabbajack.CLI/Verbs/UploadToNexus.cs

@@ -0,0 +1,47 @@
+using System.CommandLine;
+using System.CommandLine.Invocation;
+using System.Text.Json;
+using System.Threading.Tasks;
+using Microsoft.Extensions.Logging;
+using Wabbajack.Common;
+using Wabbajack.DTOs.JsonConverters;
+using Wabbajack.Networking.NexusApi;
+using Wabbajack.Networking.NexusApi.DTOs;
+using Wabbajack.Paths;
+using Wabbajack.Paths.IO;
+
+
+namespace Wabbajack.CLI.Verbs;
+
+public class UploadToNexus : IVerb
+{
+    private readonly ILogger<UploadToNexus> _logger;
+    private readonly NexusApi _client;
+    private readonly DTOSerializer _dtos;
+
+    public UploadToNexus(ILogger<UploadToNexus> logger, NexusApi wjClient, DTOSerializer dtos)
+    {
+        _logger = logger;
+        _client = wjClient;
+        _dtos = dtos;
+    }
+    public Command MakeCommand()
+    {
+        var command = new Command("upload-to-nexus");
+        command.Add(new Option<AbsolutePath>(new[] {"-d", "-definition"}, "Definition JSON file"));
+        command.Description = "Uploads a file to the Nexus defined by the given .json definition file";
+        command.Handler = CommandHandler.Create(Run);
+        return command;
+    }
+
+    public async Task<int> Run(AbsolutePath definition)
+    {
+        var d = await definition.FromJson<UploadDefinition>(_dtos);
+
+        await _client.UploadFile(d);
+        
+
+        return 0;
+    }
+    
+}

Wabbajack.Networking.NexusApi/DTOs/ModInfo.cs

@@ -16,7 +16,7 @@ public class Endorsement
 {
     [JsonPropertyName("endorse_status")] public string EndorseStatus { get; set; } = "";
 
-    [JsonPropertyName("timestamp")] public int Timestamp { get; set; }
+    [JsonPropertyName("timestamp")] public string? Timestamp { get; set; }
 
     [JsonPropertyName("version")] public string Version { get; set; } = "";
 }

Wabbajack.Networking.NexusApi/DTOs/UploadDefinition.cs

@@ -0,0 +1,120 @@
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Text.Json.Serialization;
+using System.Web;
+using Wabbajack.DTOs;
+using Wabbajack.Paths;
+using Wabbajack.Paths.IO;
+
+namespace Wabbajack.Networking.NexusApi.DTOs;
+
+public enum Category : int
+{
+    Main = 1,
+    Updates = 2,
+    Optional = 3,
+    Old = 4,
+    Misc = 5,
+    Archives = 7
+}
+
+public enum ChunkStatus
+{
+    NoContent,
+    Waiting,
+    Done
+}
+
+public class UploadDefinition
+{
+    public const long ChunkSize = 5242880; // 5MB chunks
+    public Game Game { get; set; }
+    
+    [JsonIgnore]
+    public long GameId => Game.MetaData().NexusGameId;
+
+    public string Name { get; set; }
+    public AbsolutePath Path { get; set; }
+    
+    public string Version { get; set; }
+    
+    public string Category { get; set; }
+    
+    public bool NewExisting { get; set; }
+    
+    public long OldFileId { get; set; }
+    
+    public bool RemoveOldVersion { get; set; }
+    
+    public string BriefOverview { get; set; }
+
+    public string FileUUID { get; set; } = "";
+
+    public long FileSize => Path.Size();
+    public long ModId { get; set; }
+
+    public long TotalChunks => (long) Math.Ceiling(FileSize / (double) ChunkSize);
+    public string ResumableIdentifier => FileSize + "-" + Path.FileName.ToString().Replace(".", "").Replace(" ", "");
+    public string ResumableRelativePath => HttpUtility.UrlEncode(Path.FileName.ToString());
+    public bool SetAsMain { get; set; }
+
+    public IEnumerable<Chunk> Chunks()
+    {
+
+        var size = FileSize;
+
+        if (size <= ChunkSize)
+        {
+
+            yield return new Chunk
+            {
+                Index = 0,
+                Offset = 0,
+                Size = size
+            };
+            yield break;
+        }
+        
+        for (long block = 0; block * ChunkSize < size; block++) {
+            yield return new Chunk
+            {
+                Index = block,
+                Size = Math.Min(ChunkSize, size - block * ChunkSize),
+                Offset = block * ChunkSize
+            };
+        }
+    }
+}
+
+public class Chunk
+{
+    public long Index { get; set; }
+    public long Size { get; set; }
+    public long Offset { get; set; }
+}
+
+public class ChunkStatusResult
+{
+    [JsonPropertyName("filename")]
+    public string Filename { get; set; }
+
+    [JsonPropertyName("status")]
+    public bool Status { get; set; }
+    
+    [JsonPropertyName("uuid")]
+    public string UUID { get; set; }
+}
+
+public class FileStatusResult
+{
+    [JsonPropertyName("file_chunks_reassembled")]
+    public bool FileChunksAssembled { get; set; }
+    
+    [JsonPropertyName("s3_upload_complete")]
+    public bool S3UploadComplete { get; set; }
+    
+    [JsonPropertyName("virus_total_result")]
+    public int VirusTotalStatus { get; set; }
+
+}

Wabbajack.Networking.NexusApi/NexusApi.cs

@@ -1,16 +1,22 @@
 using System;
+using System.IO;
 using System.Linq;
+using System.Net;
 using System.Net.Http;
 using System.Net.Http.Headers;
+using System.Net.Http.Json;
 using System.Text.Json;
 using System.Threading;
 using System.Threading.Tasks;
 using Microsoft.Extensions.Logging;
+using Wabbajack.Common;
 using Wabbajack.DTOs;
 using Wabbajack.DTOs.Logins;
 using Wabbajack.Networking.Http;
 using Wabbajack.Networking.Http.Interfaces;
 using Wabbajack.Networking.NexusApi.DTOs;
+using Wabbajack.Paths;
+using Wabbajack.Paths.IO;
 using Wabbajack.RateLimiter;
 using Wabbajack.Server.DTOs;
 
@@ -165,4 +171,143 @@ public class NexusApi
         var msg = await GenerateMessage(HttpMethod.Get, Endpoints.Updates, game.MetaData().NexusName, "1m");
         return await Send<UpdateEntry[]>(msg, token);
     }
+
+    public async Task<ChunkStatus> ChunkStatus(UploadDefinition definition, Chunk chunk)
+    {
+        var msg = new HttpRequestMessage();
+        msg.Method = HttpMethod.Get;
+
+        var query =
+            $"resumableChunkNumber={chunk.Index + 1}&resumableCurrentChunkSize={chunk.Size}&resumableTotalSize={definition.FileSize}"
+            + $"&resumableType=&resumableIdentifier={definition.ResumableIdentifier}&resumableFilename={definition.ResumableRelativePath}"
+            + $"&resumableRelativePath={definition.ResumableRelativePath}&resumableTotalChunks={definition.Chunks().Count()}";
+
+        msg.RequestUri = new Uri($"https://upload.nexusmods.com/uploads/chunk?{query}");
+
+        using var result = await _client.SendAsync(msg);
+        if (!result.IsSuccessStatusCode)
+            throw new HttpException(result);
+        if (result.StatusCode == HttpStatusCode.NoContent)
+            return DTOs.ChunkStatus.NoContent;
+
+        var status = await result.Content.ReadFromJsonAsync<ChunkStatusResult>();
+        return status?.Status ?? false ? DTOs.ChunkStatus.Done : DTOs.ChunkStatus.Waiting;
+    }
+
+    public async Task<ChunkStatusResult> UploadChunk(UploadDefinition d, Chunk chunk)
+    {
+        var form = new MultipartFormDataContent();
+        form.Add(new StringContent((chunk.Index+1).ToString()), "resumableChunkNumber");
+        form.Add(new StringContent(UploadDefinition.ChunkSize.ToString()), "resumableChunkSize");
+        form.Add(new StringContent(chunk.Size.ToString()), "resumableCurrentChunkSize");
+        form.Add(new StringContent(d.FileSize.ToString()), "resumableTotalSize");
+        form.Add(new StringContent(""), "resumableType");
+        form.Add(new StringContent(d.ResumableIdentifier), "resumableIdentifier");
+        form.Add(new StringContent(d.ResumableRelativePath), "resumableFilename");
+        form.Add(new StringContent(d.ResumableRelativePath), "resumableRelativePath");
+        form.Add(new StringContent(d.Chunks().Count().ToString()), "resumableTotalChunks");
+
+        await using var ms = new MemoryStream();
+        await using var fs = d.Path.Open(FileMode.Open, FileAccess.Read, FileShare.Read);
+        fs.Position = chunk.Offset;
+        await fs.CopyToLimitAsync(ms, (int)chunk.Size, CancellationToken.None);
+        ms.Position = 0;
+        
+        form.Add(new StreamContent(ms), "file", "blob");
+
+        var msg = new HttpRequestMessage(HttpMethod.Post,  "https://upload.nexusmods.com/uploads/chunk");
+        msg.Content = form;
+
+        var result = await _client.SendAsync(msg);
+        if (result.StatusCode != HttpStatusCode.OK)
+            throw new HttpException(result);
+
+        var response = await result.Content.ReadFromJsonAsync<ChunkStatusResult>(_jsonOptions);
+        return response;
+    }
+    public async Task UploadFile(UploadDefinition d)
+    {
+        _logger.LogInformation("Checking Access");
+        await CheckAccess();
+        
+        _logger.LogInformation("Checking chunk status");
+        
+        var numberOfChunks = d.Chunks().Count();
+        var chunkStatus = new ChunkStatusResult();
+        foreach (var chunk in d.Chunks())
+        {
+            var status = await ChunkStatus(d, chunk);
+            _logger.LogInformation("({Index}/{MaxChunks}) Chunk status: {Status}", chunk.Index, numberOfChunks, status);
+            if (status == DTOs.ChunkStatus.NoContent)
+            {
+                _logger.LogInformation("({Index}/{MaxChunks}) Uploading", chunk.Index, numberOfChunks);
+                chunkStatus = await UploadChunk(d, chunk);
+            }
+        }
+
+        await WaitForFileStatus(chunkStatus);
+
+        await AddFile(d, chunkStatus);
+
+    }
+
+    private async Task CheckAccess()
+    {
+        var msg = new HttpRequestMessage(HttpMethod.Get, "https://www.nexusmods.com/users/myaccount");
+        msg.AddCookies((await ApiKey.Get())!.Cookies);
+        using var response = await _client.SendAsync(msg);
+        var body = await response.Content.ReadAsStringAsync();
+
+        if (body.Contains("You are not allowed to access this area!"))
+            throw new HttpException(403, "Nexus Cookies are incorrect");
+    }
+
+    private async Task AddFile(UploadDefinition d, ChunkStatusResult status)
+    {
+        _logger.LogInformation("Saving file update {Name} to {Game}:{ModId}", d.Path.FileName, d.Game, d.ModId);
+        
+        var msg = new HttpRequestMessage(HttpMethod.Post,
+            "https://www.nexusmods.com/Core/Libs/Common/Managers/Mods?AddFile");
+        msg.Headers.Referrer =
+            new Uri(
+                $"https://www.nexusmods.com/{d.Game.MetaData().NexusName}/mods/edit/?id={d.ModId}&game_id={d.GameId}&step=files");
+        
+        msg.AddCookies((await ApiKey.Get())!.Cookies);
+        var form = new MultipartFormDataContent();
+        form.Add(new StringContent(d.GameId.ToString()), "game_id");
+        form.Add(new StringContent(d.Name), "name");
+        form.Add(new StringContent(d.Version), "file-version");
+        form.Add(new StringContent((d.RemoveOldVersion ? 1 : 0).ToString()), "update-version");
+        form.Add(new StringContent(((int)Enum.Parse<Category>(d.Category, true)).ToString()), "category");
+        form.Add(new StringContent((d.NewExisting ? 1 : 0).ToString()), "new-existing");
+        form.Add(new StringContent(d.OldFileId.ToString()), "old_file_id");
+        form.Add(new StringContent((d.RemoveOldVersion ? 1 : 0).ToString()), "remove-old-version");
+        form.Add(new StringContent(d.BriefOverview), "brief-overview");
+        form.Add(new StringContent((d.SetAsMain ? 1 : 0).ToString()), "set_as_main_nmm");
+        form.Add(new StringContent(status.UUID), "file_uuid");
+        form.Add(new StringContent(d.FileSize.ToString()), "file_size");
+        form.Add(new StringContent(d.ModId.ToString()), "mod_id");
+        form.Add(new StringContent(d.ModId.ToString()), "id");
+        form.Add(new StringContent("save"), "action");
+        form.Add(new StringContent(status.Filename), "uploaded_file");
+        form.Add(new StringContent(d.Path.FileName.ToString()), "original_file");
+        msg.Content = form;
+        
+        using var result = await _client.SendAsync(msg);
+        if (!result.IsSuccessStatusCode)
+            throw new HttpException(result);
+    }
+
+    private async Task<FileStatusResult> WaitForFileStatus(ChunkStatusResult chunkStatus)
+    {
+        while (true)
+        {
+            _logger.LogInformation("Checking file status of {Uuid}", chunkStatus.UUID);
+            var data = await _client.GetFromJsonAsync<FileStatusResult>(
+                $"https://upload.nexusmods.com/uploads/check_status?id={chunkStatus.UUID}");
+            if (data.FileChunksAssembled)
+                return data;
+            await Task.Delay(TimeSpan.FromSeconds(5));
+        }
+    }
 }

Wabbajack.Server/AppSettings.cs

@@ -22,6 +22,7 @@ public class AppSettings
     
     public string PatchesFilesFolder { get; set; }
     public string MirrorFilesFolder { get; set; }
+    public string NexusCacheFolder { get; set; }
     public string MetricsFolder { get; set; } = "";
     public string TarLogPath { get; set; }
     public string GitHubKey { get; set; } = "";

Wabbajack.Server/Controllers/NexusCache.cs

@@ -1,12 +1,18 @@
 using System;
 using System.Net.Http;
+using System.Text.Json;
 using System.Threading.Tasks;
 using Microsoft.AspNetCore.Authorization;
 using Microsoft.AspNetCore.Http;
 using Microsoft.AspNetCore.Mvc;
 using Microsoft.Extensions.Logging;
+using Wabbajack.Common;
+using Wabbajack.DTOs.JsonConverters;
 using Wabbajack.Networking.NexusApi;
 using Wabbajack.Networking.NexusApi.DTOs;
+using Wabbajack.Paths;
+using Wabbajack.Paths.IO;
+using Wabbajack.Server.Services;
 
 namespace Wabbajack.BuildServer.Controllers;
 
@@ -20,12 +26,17 @@ public class NexusCache : ControllerBase
     private readonly ILogger<NexusCache> _logger;
     private AppSettings _settings;
     private readonly HttpClient _client;
+    private readonly AbsolutePath _cacheFolder;
+    private readonly DTOSerializer _dtos;
+    private readonly NexusCacheManager _cache;
 
-    public NexusCache(ILogger<NexusCache> logger,AppSettings settings, HttpClient client)
+    public NexusCache(ILogger<NexusCache> logger,AppSettings settings, HttpClient client, NexusCacheManager cache, DTOSerializer dtos)
     {
         _settings = settings;
         _logger = logger;
         _client = client;
+        _cache = cache;
+        _dtos = dtos;
     }
 
     private async Task ForwardToNexus(HttpRequest src)
@@ -40,6 +51,16 @@ public class NexusCache : ControllerBase
         await response.Content.CopyToAsync(Response.Body);
     }
 
+    private async Task<T> ForwardRequest<T>(HttpRequest src, CancellationToken token)
+    {
+        _logger.LogInformation("Nexus Cache Forwarding: {path}", src.Path);
+        var request = new HttpRequestMessage(HttpMethod.Get, (Uri?)new Uri("https://api.nexusmods.com/" + src.Path));
+        request.Headers.Add("apikey", (string?)src.Headers["apikey"]);
+        request.Headers.Add("User-Agent", (string?)src.Headers.UserAgent);
+        using var response = await _client.SendAsync(request, token);
+        return (await JsonSerializer.DeserializeAsync<T>(await response.Content.ReadAsStreamAsync(token), _dtos.Options, token))!;
+    }
+
     /// <summary>
     ///     Looks up the mod details for a given Gamename/ModId pair. If the entry is not found in the cache it will
     ///     be requested from the server (using the caller's Nexus API key if provided).
@@ -50,22 +71,41 @@ public class NexusCache : ControllerBase
     /// <returns>A Mod Info result</returns>
     [HttpGet]
     [Route("{GameName}/mods/{ModId}.json")]
-    public async Task GetModInfo(string GameName, long ModId)
+    public async Task GetModInfo(string GameName, long ModId, CancellationToken token)
     {
-        await ForwardToNexus(Request);
+        var key = $"modinfo_{GameName}_{ModId}";
+        await ReturnCachedResult<ModInfo>(key, token);
+    }
+
+    private async Task ReturnCachedResult<T>(string key, CancellationToken token)
+    {
+        key = key.ToLowerInvariant();
+        var cached = await _cache.GetCache<T>(key, token);
+        if (cached == null)
+        {
+            var returned = await ForwardRequest<T>(Request, token);
+            await _cache.SaveCache(key, returned, token);
+            Response.StatusCode = 200;
+            Response.ContentType = "application/json";
+            await JsonSerializer.SerializeAsync(Response.Body, returned, _dtos.Options, cancellationToken: token);
+        }
+
+        await JsonSerializer.SerializeAsync(Response.Body, cached, _dtos.Options, cancellationToken: token);
     }
 
     [HttpGet]
     [Route("{GameName}/mods/{ModId}/files.json")]
-    public async Task GetModFiles(string GameName, long ModId)
+    public async Task GetModFiles(string GameName, long ModId, CancellationToken token)
     {
-        await ForwardToNexus(Request);
+        var key = $"modfiles_{GameName}_{ModId}";
+        await ReturnCachedResult<ModFiles>(key, token);
     }
 
     [HttpGet]
     [Route("{GameName}/mods/{ModId}/files/{FileId}.json")]
-    public async Task GetModFile(string GameName, long ModId, long FileId)
+    public async Task GetModFile(string GameName, long ModId, long FileId, CancellationToken token)
     {
-        await ForwardToNexus(Request);
+        var key = $"modfile_{GameName}_{ModId}_{FileId}";
+        await ReturnCachedResult<ModFile>(key, token);
     }
 }

Wabbajack.Server/Services/NexusCacheManager.cs

@@ -0,0 +1,148 @@
+using System.Text.Json;
+using K4os.Compression.LZ4.Internal;
+using Microsoft.Extensions.Logging;
+using Wabbajack.BuildServer;
+using Wabbajack.Common;
+using Wabbajack.DTOs;
+using Wabbajack.DTOs.JsonConverters;
+using Wabbajack.Networking.NexusApi;
+using Wabbajack.Networking.NexusApi.DTOs;
+using Wabbajack.Paths;
+using Wabbajack.Paths.IO;
+using Wabbajack.Server.DTOs;
+
+namespace Wabbajack.Server.Services;
+
+public class NexusCacheManager
+{
+    private readonly ILogger<NexusCacheManager> _loggger;
+    private readonly DTOSerializer _dtos;
+    private readonly AppSettings _configuration;
+    private readonly AbsolutePath _cacheFolder;
+    private readonly SemaphoreSlim _lockObject;
+    private readonly NexusApi _nexusAPI;
+    private readonly Timer _timer;
+    private readonly DiscordWebHook _discord;
+
+    public NexusCacheManager(ILogger<NexusCacheManager> logger, DTOSerializer dtos, AppSettings configuration, NexusApi nexusApi, DiscordWebHook discord)
+    {
+        _loggger = logger;
+        _dtos = dtos;
+        _configuration = configuration;
+        _cacheFolder = configuration.NexusCacheFolder.ToAbsolutePath();
+        _lockObject = new SemaphoreSlim(1);
+        _nexusAPI = nexusApi;
+        _discord = discord;
+
+        _timer = new Timer(_ => UpdateNexusCacheAPI().FireAndForget(), null, TimeSpan.FromSeconds(2),
+            TimeSpan.FromHours(4));
+    }
+    
+    
+    private AbsolutePath CacheFile(string key)
+    {
+        return _cacheFolder.Combine(key).WithExtension(Ext.Json);
+    }
+
+        
+    private bool HaveCache(string key)
+    {
+        return CacheFile(key).FileExists();
+    }
+
+    public async Task SaveCache<T>(string key, T value, CancellationToken token)
+    {
+        var ms = new MemoryStream();
+        await JsonSerializer.SerializeAsync(ms, value, _dtos.Options, token);
+        await ms.FlushAsync(token);
+        var data = ms.ToArray();
+        await _lockObject.WaitAsync(token);
+        try
+        {
+            await CacheFile(key).WriteAllBytesAsync(data, token: token);
+        }
+        finally
+        {
+            _lockObject.Release();
+        }
+    }
+
+    public async Task<T?> GetCache<T>(string key, CancellationToken token)
+    {
+        if (!HaveCache(key)) return default;
+        
+        var file = CacheFile(key);
+        await _lockObject.WaitAsync(token);
+        byte[] data;
+        try
+        {
+            data = await file.ReadAllBytesAsync(token);
+        }
+        catch (FileNotFoundException ex)
+        {
+            return default;
+        }
+        finally
+        {
+            _lockObject.Release();
+        }
+        return await JsonSerializer.DeserializeAsync<T>(new MemoryStream(data), _dtos.Options, token);
+    }
+
+    public async Task UpdateNexusCacheAPI()
+    {
+        var gameTasks = GameRegistry.Games.Values
+            .Where(g => g.NexusName != null)
+            .SelectAsync(async game =>
+            {
+                    var mods = await _nexusAPI.GetUpdates(game.Game, CancellationToken.None);
+                    return (game, mods); 
+            });
+
+
+        var purgeList = new List<(string Key, DateTime Date)>();
+        
+        await foreach (var (game, mods) in gameTasks)
+        {
+            foreach (var mod in mods.Item1)
+            {
+                var date = Math.Max(mod.LastestModActivity, mod.LatestFileUpdate).AsUnixTime();
+                purgeList.Add(($"_{game.Game.MetaData().NexusName!.ToLowerInvariant()}_{mod.ModId}_", date));
+            }
+        }
+
+        // This is O(m * n) where n and m are 15,000 items, we really should improve this
+        var files = (from file in _cacheFolder.EnumerateFiles().AsParallel()
+            from entry in purgeList
+            where file.FileName.ToString().Contains(entry.Key)
+            where file.LastModifiedUtc() < entry.Date
+            select file).ToHashSet();
+
+        foreach (var file in files)
+        {
+            await PurgeCacheEntry(file);
+        }
+        
+        await _discord.Send(Channel.Ham, new DiscordMessage
+        {
+            Content = $"Cleared {files.Count} Nexus cache entries due to updates"
+        });
+    }
+
+    private async Task PurgeCacheEntry(AbsolutePath file)
+    {
+        await _lockObject.WaitAsync();
+        try
+        {
+            if (file.FileExists()) file.Delete();
+        }
+        catch (FileNotFoundException)
+        {
+            return;
+        }
+        finally
+        {
+            _lockObject.Release();
+        }
+    }
+}

Wabbajack.Server/Startup.cs

@@ -1,6 +1,7 @@
 using System.Diagnostics;
 using System.IO;
 using System.Net.Http;
+using System.Runtime.InteropServices;
 using System.Threading.Tasks;
 using Microsoft.AspNetCore.Builder;
 using Microsoft.AspNetCore.Hosting;
@@ -16,11 +17,18 @@ using Nettle.Compiler;
 using Newtonsoft.Json;
 using Octokit;
 using Wabbajack.BuildServer;
+using Wabbajack.DTOs;
 using Wabbajack.DTOs.JsonConverters;
+using Wabbajack.DTOs.Logins;
 using Wabbajack.Networking.GitHub;
+using Wabbajack.Networking.Http.Interfaces;
+using Wabbajack.Networking.NexusApi;
+using Wabbajack.Paths;
+using Wabbajack.RateLimiter;
 using Wabbajack.Server.DataModels;
 using Wabbajack.Server.Extensions;
 using Wabbajack.Server.Services;
+using Wabbajack.Services.OSIntegrated.TokenProviders;
 
 namespace Wabbajack.Server;
 
@@ -66,6 +74,27 @@ public class Startup
         services.AddSingleton<AuthorFiles>();
         services.AddSingleton<AuthorKeys>();
         services.AddSingleton<Client>();
+        services.AddSingleton<NexusCacheManager>();
+        services.AddSingleton<NexusApi>();
+        services.AddAllSingleton<ITokenProvider<NexusApiState>, NexusApiTokenProvider>();
+        services.AddAllSingleton<IResource, IResource<HttpClient>>(s => new Resource<HttpClient>("Web Requests", 12));
+        // Application Info
+        
+        var version =
+            $"{ThisAssembly.Git.SemVer.Major}.{ThisAssembly.Git.SemVer.Major}.{ThisAssembly.Git.SemVer.Patch}{ThisAssembly.Git.SemVer.DashLabel}";
+        services.AddSingleton(s => new ApplicationInfo
+        {
+            ApplicationSlug = "Wabbajack",
+            ApplicationName = Environment.ProcessPath?.ToAbsolutePath().FileName.ToString() ?? "Wabbajack",
+            ApplicationSha = ThisAssembly.Git.Sha,
+            Platform = RuntimeInformation.ProcessArchitecture.ToString(),
+            OperatingSystemDescription = RuntimeInformation.OSDescription,
+            RuntimeIdentifier = RuntimeInformation.RuntimeIdentifier,
+            OSVersion = Environment.OSVersion.VersionString,
+            Version = version
+        });
+
+        
         services.AddResponseCaching();
         services.AddSingleton(s =>
         {
@@ -147,5 +176,8 @@ public class Startup
         });
 
         app.UseEndpoints(endpoints => { endpoints.MapControllers(); });
+
+        // Trigger the internal update code
+        var _ = app.ApplicationServices.GetRequiredService<NexusCacheManager>();
     }
 }

Wabbajack.Server/Wabbajack.Server.csproj

@@ -12,6 +12,7 @@
         <PackageReference Include="Dapper" Version="2.0.123" />
         <PackageReference Include="Discord.Net.WebSocket" Version="2.4.0" />
         <PackageReference Include="FluentFTP" Version="35.0.5" />
+        <PackageReference Include="GitInfo" Version="2.2.0" />
         <PackageReference Include="Microsoft.AspNetCore.Authentication.Core" Version="2.2.0" />
         <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="6.0.0" />
         <PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="2.2.0" />
@@ -61,6 +62,7 @@
         <ProjectReference Include="..\Wabbajack.Networking.NexusApi\Wabbajack.Networking.NexusApi.csproj" />
         <ProjectReference Include="..\Wabbajack.Paths.IO\Wabbajack.Paths.IO.csproj" />
         <ProjectReference Include="..\Wabbajack.Paths\Wabbajack.Paths.csproj" />
+        <ProjectReference Include="..\Wabbajack.Services.OSIntegrated\Wabbajack.Services.OSIntegrated.csproj" />
     </ItemGroup>
 
 

Wabbajack.Server/appsettings.json

@@ -13,6 +13,7 @@
     "AuthorAPIKeyFile": "c:\\tmp\\author_keys.txt",
     "PatchesFilesFolder": "c:\\tmp\\patches",
     "MirrorFilesFolder": "c:\\tmp\\mirrors",
+    "NexusCacheFolder": "c:\\tmp\\nexus-cache",
     "GitHubKey": ""
   },
   "AllowedHosts": "*"