Merge pull request #87 from wabbajack-tools/rework-download-integration

Rework download integration

CHANGELOG.md

@@ -15,6 +15,7 @@
 * Add BA2 support
 * Fix Downloads folder being incorrectly detected in some cases
 * Fix validation error on selecting an installation directory in Install mode
+* Reworked download code to be more extensible and stable
 
 #### Version 0.9.4 - 10/2/2019
 * Point github icon to https://github.com/wabbajack-tools/wabbajack

NexusPage.html

@@ -276,3 +276,107 @@
 <h3 id="faq">FAQ</h3>
 
 <p><strong>How do I get Wabbjack to handle mods from <code>X</code></strong></p>
+
+<p>Look at the [<code>RECIPES.md</code>] file, we keep a knowledgebase of how to deal with given types of mods in that file.</p>
+
+<p><strong>How do I contribute to Wabbajack?</strong></p>
+
+<p>Look at the <a href="https://github.com/halgari/wabbajack/blob/master/CONTRIBUTING.md"><code>CONTRIBUTING.md</code></a> file for detailed guidelines.</p>
+
+<p><strong>Why does each modlist install another copy of Mod Organizer 2?</strong></p>
+
+<p>
+    Self-contained folders are a cleaner abstraction than dumping tons of modlists into the same set of folders. It's easy to uninstall a modlist (simply delete the folder),
+    and MO2 really isn't designed to support lots of disparate modlists. For example if two modlists both wanted a given texture mod, but different options they would
+    somehow have to keep the names of their mods separate. MO2 isn't that big of an app, so there's really no reason not to install a new copy for each modlist.
+</p>
+
+<p><strong>Why don't I see any mods when I open Mod Organizer 2 after install?</strong></p>
+
+<p>
+    Make sure you selected the "Portable" mode when starting MO2 for the first time. In addition, make sure you haven't installed MO2 in a non-portable way on the same box.
+    Really, always use "Portable Mode" it's cleaner and there really isn't a reason not too do so. Make the data self-contained. It's cleaner that way.
+</p>
+
+<p><strong>Will Wabbajack ever support Vortex/other mod managers?</strong></p>
+
+<p>
+    I'll be honest, I don't use anything but MO2, so I probably won't write the code. If someone were to write a patch for the functionality
+    I wouldn't throw away the code, but it would have to be done in a way that was relatively seamless for users. Since Wabbajack treats all files in the same way
+    it doesn't know what mod manager a user is using. This means that if the modlist creator used Vortex all users of the modlist would have to use Vortex. This doesn't seem
+    optimal. It's possible perhaps, but it's at the bottom of the priority list.
+</p>
+
+<p><strong>How does Wabbajack differ from Automaton?</strong></p>
+
+<p>
+    I (halgari) used to be a developer working on Automaton. Sadly development was moving a bit too slowly for my liking, and I realized that a complete rewrite would allow the
+    implementation of some really nice features (like BSA packing). As such I made the decision to strike out on my own and make an app that worked first, and then make it pretty.
+    The end result is an app with a ton of features, and a less than professional UI. But that's my motto when coding "make it work, then make it pretty".
+</p>
+
+<h2 id="thanksto">Thanks to</h2>
+
+<p>Our tester and Discord members who encourage development and help test the builds.</p>
+
+<h3 id="patreonsupporters">Patreon Supporters</h3>
+
+<h4 id="daedralevelpatreonsupporters">Daedra level Patreon Supporters</h4>
+
+<ul>
+    <li>Ancalgon</li>
+
+    <li>Theo</li>
+
+    <li>Dascede</li>
+
+    <li>Kristina Poňuchálková</li>
+
+    <li>metherul</li>
+
+    <li>Decopauge123</li>
+</ul>
+
+<h4 id="patreonsupporters-1">Patreon Supporters</h4>
+
+<ul>
+    <li>Druwski</li>
+
+    <li>Soothsayre</li>
+
+    <li>krageon</li>
+
+    <li>Scumbag</li>
+
+    <li>Burt Wheeler</li>
+
+    <li>Jesse Earl Rockwell</li>
+
+    <li>Mike Gray</li>
+
+    <li>Theryl</li>
+
+    <li>Daniel Gardner</li>
+
+    <li>Dapper</li>
+
+    <li>Corapol</li>
+
+    <li>HQM</li>
+
+    <li>Argos</li>
+
+    <li>sorrydaijin</li>
+
+    <li>William Chudziak</li>
+
+    <li>N Kalim</li>
+</ul>
+
+<h3 id="licensecopyright">License &amp; Copyright</h3>
+
+<p>
+    All original code in Wabbajack is given freely via the GPL3 license. Parts of Wabbajack use libraries that carry their own Open Sources licenses, those parts
+    retain their original copyrights. Note: Wabbajack installers contain code from Wabbajack. Therefore, selling of modlist files is strictly forbidden. As is hosting
+    the files behind any sort of paywall. You recieved this tool free of charge, respect this by giving freely as you were given.
+</p>

README.md

@@ -1,4 +1,4 @@
-## Wabbajack - An automated modlist installer for TES/Fallout games
+## Wabbajack - An automated modlist installer for TES/Fallout games
 
 [![Build Status](https://dev.azure.com/tbaldridge/tbaldridge/_apis/build/status/halgari.wabbajack?branchName=master)](https://dev.azure.com/tbaldridge/tbaldridge/_build/latest?definitionId=1&branchName=master)
 
@@ -209,6 +209,11 @@ Our tester and Discord members who encourage development and help test the build
 #### Daedra level Patreon Supporters
 
 - Ancalgon
+- Theo
+- Dascede
+- Kristina Poňuchálková
+- metherul
+- Decopauge123
 
 #### Patreon Supporters
 
@@ -216,6 +221,18 @@ Our tester and Discord members who encourage development and help test the build
 - Soothsayre
 - krageon
 - Scumbag
+- Burt Wheeler
+- Jesse Earl Rockwell
+- Mike Gray
+- Theryl
+- Daniel Gardner
+- Dapper
+- Corapol
+- HQM
+- Argos
+- sorrydaijin
+- William Chudziak
+- N Kalim
 
 ### License & Copyright
 

Wabbajack.Common/Consts.cs

@@ -9,7 +9,7 @@ namespace Wabbajack.Common
     {
         public static bool TestMode { get; set; } = false;
 
-        public static string ModlistExtension = ".modlist_v1";
+        public static string ModlistExtension = ".modlist_v2";
         public static string GameFolderFilesDir = "Game Folder Files";
         public static string LOOTFolderFilesDir = "LOOT Config Files";
         public static string BSACreationDir = "TEMP_BSA_FILES";
@@ -57,7 +57,7 @@ namespace Wabbajack.Common
             {
                 var platformType = Environment.Is64BitOperatingSystem ? "x64" : "x86";
                 var headerString =
-                    $"{AppName}/{Assembly.GetEntryAssembly().GetName().Version} ({Environment.OSVersion.VersionString}; {platformType}) {RuntimeInformation.FrameworkDescription}";
+                    $"{AppName}/{Assembly.GetEntryAssembly()?.GetName()?.Version ?? new Version(0, 1)} ({Environment.OSVersion.VersionString}; {platformType}) {RuntimeInformation.FrameworkDescription}";
                 return headerString;
             }
         }

Wabbajack.Common/Utils.cs

@@ -167,6 +167,16 @@ namespace Wabbajack.Common
             return new DynamicIniData(new FileIniDataParser().ReadFile(file));
         }
 
+        /// <summary>
+        /// Loads a INI from the given string
+        /// </summary>
+        /// <param name="file"></param>
+        /// <returns></returns>
+        public static dynamic LoadIniString(this string file)
+        {
+            return new DynamicIniData(new FileIniDataParser().ReadData(new StreamReader(new MemoryStream(Encoding.UTF8.GetBytes(file)))));
+        }
+
         public static void ToJSON<T>(this T obj, string filename)
         {
             File.WriteAllText(filename,
@@ -194,7 +204,7 @@ namespace Wabbajack.Common
         public static string ToJSON<T>(this T obj)
         {
             return JsonConvert.SerializeObject(obj, Formatting.Indented,
-                new JsonSerializerSettings {TypeNameHandling = TypeNameHandling.Auto});
+                new JsonSerializerSettings {TypeNameHandling = TypeNameHandling.All});
         }
 
         public static T FromJSON<T>(this string filename)
@@ -529,5 +539,23 @@ namespace Wabbajack.Common
             }
             return outarr;
         }
+
+        /// <summary>
+        /// Roundtrips the value throught the JSON routines
+        /// </summary>
+        /// <typeparam name="TV"></typeparam>
+        /// <typeparam name="TR"></typeparam>
+        /// <param name="tv"></param>
+        /// <returns></returns>
+        public static T ViaJSON<T>(this T tv)
+        {
+            return tv.ToJSON().FromJSONString<T>();
+        }
+
+        public static void Error(string msg)
+        {
+            Log(msg);
+            throw new Exception(msg);
+        }
     }
 }

Wabbajack.Test/ContentRightsManagementTests.cs

@@ -4,6 +4,7 @@ using System.Linq;
 using System.Security.Policy;
 using Microsoft.VisualStudio.TestTools.UnitTesting;
 using Wabbajack.Common;
+using Wabbajack.Downloaders;
 using Wabbajack.Validation;
 using Game = Wabbajack.Common.Game;
 
@@ -56,7 +57,7 @@ namespace Wabbajack.Test
         [TestMethod]
         public void TestRightsFallthrough()
         {
-            var permissions = validate.FilePermissions(new NexusMod()
+            var permissions = validate.FilePermissions(new NexusDownloader.State
             {
                 Author = "bill",
                 GameName = "Skyrim",
@@ -69,7 +70,7 @@ namespace Wabbajack.Test
             permissions.CanModifyAssets.AssertIsFalse();
             permissions.CanUseInOtherGames.AssertIsFalse();
 
-            permissions = validate.FilePermissions(new NexusMod()
+            permissions = validate.FilePermissions(new NexusDownloader.State
             {
                 Author = "bob",
                 GameName = "Skyrim",
@@ -82,7 +83,7 @@ namespace Wabbajack.Test
             permissions.CanModifyAssets.AssertIsTrue();
             permissions.CanUseInOtherGames.AssertIsTrue();
 
-            permissions = validate.FilePermissions(new NexusMod()
+            permissions = validate.FilePermissions(new NexusDownloader.State
             {
                 Author = "bill",
                 GameName = "Fallout4",
@@ -95,7 +96,7 @@ namespace Wabbajack.Test
             permissions.CanModifyAssets.AssertIsTrue();
             permissions.CanUseInOtherGames.AssertIsTrue();
 
-            permissions = validate.FilePermissions(new NexusMod()
+            permissions = validate.FilePermissions(new NexusDownloader.State
             {
                 Author = "bill",
                 GameName = "Skyrim",
@@ -108,7 +109,7 @@ namespace Wabbajack.Test
             permissions.CanModifyAssets.AssertIsTrue();
             permissions.CanUseInOtherGames.AssertIsTrue();
 
-            permissions = validate.FilePermissions(new NexusMod()
+            permissions = validate.FilePermissions(new NexusDownloader.State
             {
                 Author = "bill",
                 GameName = "Skyrim",
@@ -131,14 +132,18 @@ namespace Wabbajack.Test
                 GameType = Game.Skyrim,
                 Archives = new List<Archive>
                 {
-                    new NexusMod
+                    new Archive
+                    {
+                        State = new NexusDownloader.State                    
                         {
                             GameName = "Skyrim",
                             Author = "bill",
                             ModID = "42",
                             FileID = "33",
+                        },
                         Hash = "DEADBEEF"
                     }
+
                 },
                 Directives = new List<Directive>
                 {
@@ -196,44 +201,44 @@ namespace Wabbajack.Test
 
             // Error due to file downloaded from 3rd party
             modlist.GameType = Game.Skyrim;
-            modlist.Archives[0] = new DirectURLArchive()
+            modlist.Archives[0] = new Archive()
             {
-                URL = "https://somebadplace.com",
+                State = new HTTPDownloader.State() { Url = "https://somebadplace.com" },
                 Hash = "DEADBEEF"
             };
             errors = validate.Validate(modlist);
-            Assert.AreEqual(errors.Count(), 1);
+            Assert.AreEqual(1, errors.Count());
 
             // Ok due to file downloaded from whitelisted 3rd party
             modlist.GameType = Game.Skyrim;
-            modlist.Archives[0] = new DirectURLArchive()
+            modlist.Archives[0] = new Archive
             {
-                URL = "https://somegoodplace.com/myfile",
+                State = new HTTPDownloader.State { Url = "https://somegoodplace.com/baz.7z" },
                 Hash = "DEADBEEF"
             };
             errors = validate.Validate(modlist);
-            Assert.AreEqual(errors.Count(), 0);
+            Assert.AreEqual(0, errors.Count());
 
 
             // Error due to file downloaded from bad 3rd party
             modlist.GameType = Game.Skyrim;
-            modlist.Archives[0] = new GoogleDriveMod()
+            modlist.Archives[0] = new Archive
             {
-                Id = "bleg",
+                State = new GoogleDriveDownloader.State { Id = "bleg"},
                 Hash = "DEADBEEF"
             };
             errors = validate.Validate(modlist);
             Assert.AreEqual(errors.Count(), 1);
 
-            // Error due to file downloaded from good 3rd party
+            // Ok due to file downloaded from good google site
             modlist.GameType = Game.Skyrim;
-            modlist.Archives[0] = new GoogleDriveMod()
+            modlist.Archives[0] = new Archive
             {
-                Id = "googleDEADBEEF",
+                State = new GoogleDriveDownloader.State { Id = "googleDEADBEEF" },
                 Hash = "DEADBEEF"
             };
             errors = validate.Validate(modlist);
-            Assert.AreEqual(errors.Count(), 0);
+            Assert.AreEqual(0, errors.Count());
 
         }
 

Wabbajack.Test/DownloaderTests.cs

@@ -0,0 +1,158 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Microsoft.VisualStudio.TestTools.UnitTesting;
+using Wabbajack.Common;
+using Wabbajack.Downloaders;
+using Wabbajack.Validation;
+using File = Alphaleonis.Win32.Filesystem.File;
+
+namespace Wabbajack.Test
+{
+    [TestClass]
+    public class DownloaderTests
+    {
+        [TestMethod]
+        public void MegaDownload()
+        {
+            var ini = @"[General]
+                        directURL=https://mega.nz/#!CsMSFaaJ!-uziC4mbJPRy2e4pPk8Gjb3oDT_38Be9fzZ6Ld4NL-k";
+
+            var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
+
+            Assert.IsNotNull(state);
+
+            var converted = state.ViaJSON();
+            Assert.IsTrue(converted.Verify());
+            var filename = Guid.NewGuid().ToString();
+
+            Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist {AllowedPrefixes = new List<string>{"https://mega.nz/#!CsMSFaaJ!-uziC4mbJPRy2e4pPk8Gjb3oDT_38Be9fzZ6Ld4NL-k" } }));
+            Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string>{ "blerg" }}));
+
+            converted.Download(new Archive {Name = "MEGA Test.txt"}, filename);
+
+            Assert.AreEqual("Lb1iTsz3iyZeHGs3e94TVmOhf22sqtHLhqkCdXbjiyc=", Utils.FileSHA256(filename));
+
+            Assert.AreEqual(File.ReadAllText(filename), "Cheese for Everyone!");
+        }
+
+        [TestMethod]
+        public void DropboxTests()
+        {
+            var ini = @"[General]
+                        directURL=https://www.dropbox.com/s/5hov3m2pboppoc2/WABBAJACK_TEST_FILE.txt?dl=0";
+
+            var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
+
+            Assert.IsNotNull(state);
+
+            var converted = state.ViaJSON();
+            Assert.IsTrue(converted.Verify());
+            var filename = Guid.NewGuid().ToString();
+
+            Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> { "https://www.dropbox.com/s/5hov3m2pboppoc2/WABBAJACK_TEST_FILE.txt?" } }));
+            Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> { "blerg" } }));
+
+            converted.Download(new Archive { Name = "MEGA Test.txt" }, filename);
+
+            Assert.AreEqual("Lb1iTsz3iyZeHGs3e94TVmOhf22sqtHLhqkCdXbjiyc=", Utils.FileSHA256(filename));
+
+            Assert.AreEqual(File.ReadAllText(filename), "Cheese for Everyone!");
+        }
+
+        [TestMethod]
+        public void GoogleDriveTests()
+        {
+            var ini = @"[General]
+                        directURL=https://drive.google.com/file/d/1grLRTrpHxlg7VPxATTFNfq2OkU_Plvh_/view?usp=sharing";
+
+            var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
+
+            Assert.IsNotNull(state);
+
+            var converted = state.ViaJSON();
+            Assert.IsTrue(converted.Verify());
+            var filename = Guid.NewGuid().ToString();
+
+            Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { GoogleIDs = new List<string> { "1grLRTrpHxlg7VPxATTFNfq2OkU_Plvh_" } }));
+            Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { GoogleIDs = new List<string>()}));
+
+            converted.Download(new Archive { Name = "MEGA Test.txt" }, filename);
+
+            Assert.AreEqual("Lb1iTsz3iyZeHGs3e94TVmOhf22sqtHLhqkCdXbjiyc=", Utils.FileSHA256(filename));
+
+            Assert.AreEqual(File.ReadAllText(filename), "Cheese for Everyone!");
+        }
+
+        [TestMethod]
+        public void HttpDownload()
+        {
+            var ini = @"[General]
+                        directURL=https://raw.githubusercontent.com/wabbajack-tools/opt-out-lists/master/ServerWhitelist.yml";
+
+            var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
+
+            Assert.IsNotNull(state);
+
+            var converted = state.ViaJSON();
+            Assert.IsTrue(converted.Verify());
+            var filename = Guid.NewGuid().ToString();
+
+            Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> { "https://raw.githubusercontent.com/" } }));
+            Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string>() }));
+
+            converted.Download(new Archive { Name = "Github Test Test.txt" }, filename);
+
+            Assert.IsTrue(File.ReadAllText(filename).StartsWith("# Server whitelist for Wabbajack"));
+        }
+
+        [TestMethod]
+        public void NexusDownload()
+        {
+            var ini = @"[General]
+                        gameName=SkyrimSE
+                        modID = 12604
+                        fileID=35407";
+
+            var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
+
+            Assert.IsNotNull(state);
+
+            var converted = state.ViaJSON();
+            Assert.IsTrue(converted.Verify());
+            var filename = Guid.NewGuid().ToString();
+
+            Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> () }));
+
+            converted.Download(new Archive { Name = "SkyUI.7z" }, filename);
+
+            Assert.AreEqual(filename.FileSHA256(), "U3Xg6RBR9XrUY9/jQSu6WKu5dfhHmpaN2dTl0ylDFmI=");
+        }
+
+        [TestMethod]
+        public void ModDbTests()
+        {
+            var ini = @"[General]
+                        directURL=https://www.moddb.com/downloads/start/124908?referer=https%3A%2F%2Fwww.moddb.com%2Fmods%2Fautopause";
+
+            var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
+
+            Assert.IsNotNull(state);
+
+            var converted = state.ViaJSON();
+            Assert.IsTrue(converted.Verify());
+            var filename = Guid.NewGuid().ToString();
+
+            Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string>() }));
+
+            converted.Download(new Archive { Name = "moddbtest.7z" }, filename);
+
+            Assert.AreEqual("lUvpEjqxfyidBONSHcDy6EnZIPpAD2K4rkJ5ejCXc2k=", filename.FileSHA256());
+        }
+    }
+
+
+
+}

Wabbajack.Test/Wabbajack.Test.csproj

@@ -61,6 +61,7 @@
     <Reference Include="AlphaFS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=4d31a58f7d7ad5c9, processorArchitecture=MSIL">
       <HintPath>..\packages\AlphaFS.2.2.6\lib\net452\AlphaFS.dll</HintPath>
     </Reference>
+    <Reference Include="Microsoft.CSharp" />
     <Reference Include="Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
       <HintPath>..\packages\MSTest.TestFramework.1.3.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll</HintPath>
     </Reference>
@@ -73,6 +74,7 @@
     <Reference Include="System.Transactions" />
   </ItemGroup>
   <ItemGroup>
+    <Compile Include="DownloaderTests.cs" />
     <Compile Include="Extensions.cs" />
     <Compile Include="TestUtils.cs" />
     <Compile Include="SanityTests.cs" />

Wabbajack/AppState.cs

@@ -15,6 +15,7 @@ using System.Windows.Input;
 using System.Windows.Media.Imaging;
 using System.Windows.Threading;
 using Wabbajack.Common;
+using Wabbajack.Downloaders;
 using Wabbajack.NexusApi;
 using Wabbajack.UI;
 
@@ -484,7 +485,10 @@ namespace Wabbajack
 
             ApplyModlistProperties();
 
-            _slideShow.SlideShowElements = modlist.Archives.OfType<NexusMod>().Select(m => 
+            _slideShow.SlideShowElements = modlist.Archives
+                .Select(m => m.State)
+                .OfType<NexusDownloader.State>()
+                .Select(m => 
                 new Slide(NexusApiUtils.FixupSummary(m.ModName),m.ModID,
                     NexusApiUtils.FixupSummary(m.Summary), NexusApiUtils.FixupSummary(m.Author),
                     m.Adult,m.NexusURL,m.SlideShowPic)).ToList();

Wabbajack/Compiler.cs

@@ -13,6 +13,7 @@ using System.Text.RegularExpressions;
 using System.Web;
 using VFS;
 using Wabbajack.Common;
+using Wabbajack.Downloaders;
 using Wabbajack.NexusApi;
 using Wabbajack.Validation;
 using Directory = Alphaleonis.Win32.Filesystem.Directory;
@@ -469,123 +470,21 @@ namespace Wabbajack
             {
                 if (found.IniData == null)
                     Error($"No download metadata found for {found.Name}, please use MO2 to query info or add a .meta file and try again.");
-                var general = found.IniData.General;
-                if (general == null)
-                    Error($"No General section in mod metadata found for {found.Name}, please use MO2 to query info or add the info and try again.");
 
-                Archive result;
+                var result = new Archive();
+                result.State = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(found.IniData);
 
-                if (general.directURL != null && general.directURL.StartsWith("https://drive.google.com"))
+                if (result.State == null)
-                {
+                    Error($"{found.Name} could not be handled by any of the downloaders");
-                    var regex = new Regex("((?<=id=)[a-zA-Z0-9_-]*)|(?<=\\/file\\/d\\/)[a-zA-Z0-9_-]*");
-                    var match = regex.Match(general.directURL);
-                    result = new GoogleDriveMod
-                    {
-                        Id = match.ToString()
-                    };
-                }
-                else if (general.directURL != null && general.directURL.StartsWith(Consts.MegaPrefix))
-                {
-                    result = new MEGAArchive
-                    {
-                        URL = general.directURL
-                    };
-                }
-                else if (general.directURL != null && general.directURL.StartsWith("https://www.dropbox.com/"))
-                {
-                    var uri = new UriBuilder((string)general.directURL);
-                    var query = HttpUtility.ParseQueryString(uri.Query);
-
-                    if (query.GetValues("dl").Count() > 0)
-                        query.Remove("dl");
-
-                    query.Set("dl", "1");
-
-                    uri.Query = query.ToString();
-
-                    result = new DirectURLArchive
-                    {
-                        URL = uri.ToString()
-                    };
-                }
-                else if (general.directURL != null &&
-                         general.directURL.StartsWith("https://www.moddb.com/downloads/start"))
-                {
-                    result = new MODDBArchive
-                    {
-                        URL = general.directURL
-                    };
-                }
-                else if (general.directURL != null && general.directURL.StartsWith("http://www.mediafire.com/file/"))
-                {
-                    Error("MediaFire links are not currently supported");
-                    return null;
-                    /*result = new MediaFireArchive()
-                    {
-                        URL = general.directURL
-                    };*/
-                }
-                else if (general.directURL != null)
-                {
-                    var tmp = new DirectURLArchive
-                    {
-                        URL = general.directURL
-                    };
-                    if (general.directURLHeaders != null)
-                    {
-                        tmp.Headers = new List<string>();
-                        tmp.Headers.AddRange(general.directURLHeaders.Split('|'));
-                    }
-
-                    result = tmp;
-                }
-                else if (general.modID != null && general.fileID != null && general.gameName != null)
-                {
-                    var nm = new NexusMod
-                    {
-                        GameName = general.gameName,
-                        FileID = general.fileID,
-                        ModID = general.modID,
-                        Version = general.version ?? "0.0.0.0"
-                    };
-                    var info = new NexusApiClient().GetModInfo(nm);
-                    nm.Author = info.author;
-                    nm.UploadedBy = info.uploaded_by;
-                    nm.UploaderProfile = info.uploaded_users_profile_url;
-                    nm.ModName = info.name;
-                    nm.SlideShowPic = info.picture_url;
-                    nm.NexusURL = NexusApiUtils.GetModURL(info.game_name, info.mod_id);
-                    nm.Summary = info.summary;
-                    nm.Adult = info.contains_adult_content;
-
-                    result = nm;
-                }
-                else if (general.manualURL != null)
-                {
-                    result = new ManualArchive
-                    {
-                        URL = general.manualURL,
-                        Notes = general.manualNotes,
-                    };
-                }
-                else
-                {
-                    Error($"No way to handle archive {found.Name} but it's required by the modlist");
-                    return null;
-                }
 
                 result.Name = found.Name;
                 result.Hash = found.File.Hash;
                 result.Meta = found.Meta;
                 result.Size = found.File.Size;
 
-                if (result is ManualArchive) return result;
-
                 Info($"Checking link for {found.Name}");
 
-                var installer = new Installer("", null, "");
+                if (!result.State.Verify())
-
-                if (!installer.DownloadArchive(result, false))
                     Error(
                         $"Unable to resolve link for {found.Name}. If this is hosted on the Nexus the file may have been removed.");
 

Wabbajack/Data.cs

@@ -4,6 +4,7 @@ using System.Collections.Generic;
 using Compression.BSA;
 using VFS;
 using Wabbajack.Common;
+using Wabbajack.Downloaders;
 
 namespace Wabbajack
 {
@@ -218,81 +219,7 @@ namespace Wabbajack
         public string Name;
 
         public long Size;
-    }
+        public AbstractDownloadState State { get; set; }
-
-    [Serializable]
-    public class NexusMod : Archive
-    {
-        public string Author;
-        public string FileID;
-        public string GameName;
-        public string ModID;
-        public string UploadedBy;
-        public string UploaderProfile;
-        public string Version;
-        public string SlideShowPic;
-        public string ModName;
-        public string NexusURL;
-        public string Summary;
-        public bool Adult;
-    }
-
-    [Serializable]
-    public class ManualArchive : Archive
-    {
-        public string URL;
-        public string Notes;
-    }
-
-    [Serializable]
-    public class GoogleDriveMod : Archive
-    {
-        public string Id;
-    }
-
-    /// <summary>
-    ///     URL that can be downloaded directly without any additional options
-    /// </summary>
-    [Serializable]
-    public class DirectURLArchive : Archive
-    {
-        [JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
-        public List<string> Headers;
-
-        public string URL;
-    }
-
-    /// <summary>
-    ///     An archive that requires additional HTTP headers.
-    /// </summary>
-    [Serializable]
-    public class DirectURLArchiveEx : DirectURLArchive
-    {
-        public Dictionary<string, string> Headers;
-    }
-
-    /// <summary>
-    ///     Archive that comes from MEGA
-    /// </summary>
-    [Serializable]
-    public class MEGAArchive : DirectURLArchive
-    {
-    }
-
-    /// <summary>
-    ///     Archive that comes from MODDB
-    /// </summary>
-    [Serializable]
-    public class MODDBArchive : DirectURLArchive
-    {
-    }
-
-    /// <summary>
-    ///     Archive that comes from MediaFire
-    /// </summary>
-    [Serializable]
-    public class MediaFireArchive : DirectURLArchive
-    {
     }
 
     [Serializable]

Wabbajack/Downloaders/AbstractDownloadState.cs

@@ -0,0 +1,38 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Wabbajack.Validation;
+
+namespace Wabbajack.Downloaders
+{
+    /// <summary>
+    /// Base for all abstract downloaders
+    /// </summary>
+    public abstract class AbstractDownloadState
+    {
+        /// <summary>
+        /// Returns true if this file is allowed to be downloaded via whitelist
+        /// </summary>
+        /// <param name="whitelist"></param>
+        /// <returns></returns>
+        public abstract bool IsWhitelisted(ServerWhitelist whitelist);
+
+        /// <summary>
+        /// Downloads this file to the given destination location
+        /// </summary>
+        /// <param name="destination"></param>
+        public abstract void Download(Archive a, string destination);
+
+        /// <summary>
+        /// Returns true if this link is still valid
+        /// </summary>
+        /// <returns></returns>
+        public abstract bool Verify();
+
+        public abstract IDownloader GetDownloader();
+
+        public abstract string GetReportEntry(Archive a);
+    }
+}

Wabbajack/Downloaders/DownloadDispatcher.cs

@@ -0,0 +1,41 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Security.Policy;
+using System.Text;
+using System.Threading.Tasks;
+
+namespace Wabbajack.Downloaders
+{
+    public static class DownloadDispatcher
+    {
+        private static readonly List<IDownloader> Downloaders = new List<IDownloader>()
+        {
+            new MegaDownloader(),
+            new DropboxDownloader(),
+            new GoogleDriveDownloader(),
+            new ModDBDownloader(),
+            new NexusDownloader(),
+            new ManualDownloader(),
+            new HTTPDownloader()
+        };
+
+        private static readonly Dictionary<Type, IDownloader> IndexedDownloaders;
+
+        static DownloadDispatcher()
+        {
+            IndexedDownloaders = Downloaders.ToDictionary(d => d.GetType());
+        }
+
+        public static T GetInstance<T>()
+        {
+            return (T)IndexedDownloaders[typeof(T)];
+        }
+
+        public static AbstractDownloadState ResolveArchive(dynamic ini)
+        {
+            return Downloaders.Select(d => d.GetDownloaderState(ini)).FirstOrDefault(result => result != null);
+        }
+
+    }
+}

Wabbajack/Downloaders/DropboxDownloader.cs

@@ -0,0 +1,37 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using System.Web;
+
+namespace Wabbajack.Downloaders
+{
+    public class DropboxDownloader : IDownloader
+    {
+        public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
+        {
+            var urlstring = archive_ini?.General?.directURL;
+            if (urlstring == null) return null;
+            var uri = new UriBuilder((string)urlstring);
+            if (uri.Host != "www.dropbox.com") return null;
+            var query = HttpUtility.ParseQueryString(uri.Query);
+
+            if (query.GetValues("dl").Length > 0)
+                query.Remove("dl");
+
+            query.Set("dl", "1");
+
+            uri.Query = query.ToString();
+
+            return new HTTPDownloader.State()
+            {
+                Url = uri.ToString().Replace("dropbox.com:443/", "dropbox.com/")
+            };
+        }
+
+        public void Prepare()
+        {
+        }
+    }
+}

Wabbajack/Downloaders/GoogleDriveDownloader.cs

@@ -0,0 +1,77 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Net.Http;
+using System.Security.Cryptography.X509Certificates;
+using System.Text;
+using System.Text.RegularExpressions;
+using System.Threading.Tasks;
+using Wabbajack.Common;
+using Wabbajack.Validation;
+
+namespace Wabbajack.Downloaders
+{
+    public class GoogleDriveDownloader : IDownloader
+    {
+        public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
+        {
+            var url = archive_ini?.General?.directURL;
+            var regex = new Regex("((?<=id=)[a-zA-Z0-9_-]*)|(?<=\\/file\\/d\\/)[a-zA-Z0-9_-]*");
+            if (url != null && url.StartsWith("https://drive.google.com"))
+            {
+                var match = regex.Match(url);
+                return new State
+                {
+                    Id = match.ToString()
+                };
+            }
+
+            return null;
+        }
+
+        public void Prepare()
+        {
+        }
+
+        public class State : AbstractDownloadState
+        {
+            public string Id { get; set; }
+            public override bool IsWhitelisted(ServerWhitelist whitelist)
+            {
+                return whitelist.GoogleIDs.Contains(Id);
+            }
+
+            public override void Download(Archive a, string destination)
+            {
+                ToHttpState().Download(a, destination);
+            }
+
+            private HTTPDownloader.State ToHttpState()
+            {
+                var initial_url = $"https://drive.google.com/uc?id={Id}&export=download";
+                var client = new HttpClient();
+                var result = client.GetStringSync(initial_url);
+                var regex = new Regex("(?<=/uc\\?export=download&amp;confirm=).*(?=;id=)");
+                var confirm = regex.Match(result);
+                var url = $"https://drive.google.com/uc?export=download&confirm={confirm}&id={Id}";
+                var http_state = new HTTPDownloader.State {Url = url};
+                return http_state;
+            }
+
+            public override bool Verify()
+            {
+                return ToHttpState().Verify();
+            }
+
+            public override IDownloader GetDownloader()
+            {
+                return DownloadDispatcher.GetInstance<GoogleDriveDownloader>();
+            }
+
+            public override string GetReportEntry(Archive a)
+            {
+                return $"* GoogleDrive - [{a.Name}](https://drive.google.com/uc?id={Id}&export=download)";
+            }
+        }
+    }
+}

Wabbajack/Downloaders/HTTPDownloader.cs

@@ -0,0 +1,138 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Net.Http;
+using System.ServiceModel.Configuration;
+using System.Text;
+using System.Threading.Tasks;
+using Newtonsoft.Json;
+using Wabbajack.Common;
+using Wabbajack.Validation;
+using File = Alphaleonis.Win32.Filesystem.File;
+
+namespace Wabbajack.Downloaders
+{
+    public class HTTPDownloader : IDownloader
+    {
+
+        public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
+        {
+            var url = archive_ini?.General?.directURL;
+
+            if (url != null)
+            {
+                var tmp = new State
+                {
+                    Url = url
+                };
+                if (archive_ini?.General?.directURLHeaders != null)
+                {
+                    tmp.Headers = new List<string>();
+                    tmp.Headers.AddRange(archive_ini?.General.directURLHeaders.Split('|'));
+                }
+                return tmp;
+            }
+
+            return null;
+        }
+
+        public void Prepare()
+        {
+        }
+
+        public class State : AbstractDownloadState
+        {
+            public string Url { get; set; }
+
+            [JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
+            public List<string> Headers { get; set; }
+
+            public override bool IsWhitelisted(ServerWhitelist whitelist)
+            {
+                return whitelist.AllowedPrefixes.Any(p => Url.StartsWith(p));
+            }
+
+            public override void Download(Archive a, string destination)
+            {
+                DoDownload(a, destination, true);
+            }
+
+            public bool DoDownload(Archive a, string destination, bool download)
+            {
+                var client = new HttpClient();
+                client.DefaultRequestHeaders.Add("User-Agent", Consts.UserAgent);
+
+                if (Headers != null)
+                    foreach (var header in Headers)
+                    {
+                        var idx = header.IndexOf(':');
+                        var k = header.Substring(0, idx);
+                        var v = header.Substring(idx + 1);
+                        client.DefaultRequestHeaders.Add(k, v);
+                    }
+
+                long total_read = 0;
+                var buffer_size = 1024 * 32;
+
+                var response = client.GetSync(Url);
+                var stream = response.Content.ReadAsStreamAsync();
+                try
+                {
+                    stream.Wait();
+                }
+                catch (Exception ex)
+                {
+                }
+
+                ;
+                if (stream.IsFaulted)
+                {
+                    Utils.Log($"While downloading {Url} - {stream.Exception.ExceptionToString()}");
+                    return false;
+                }
+
+                if (!download)
+                    return true;
+
+                var header_var = "1";
+                if (response.Content.Headers.Contains("Content-Length"))
+                    header_var = response.Content.Headers.GetValues("Content-Length").FirstOrDefault();
+
+                var content_size = header_var != null ? long.Parse(header_var) : 1;
+
+
+                using (var webs = stream.Result)
+                using (var fs = File.OpenWrite(destination))
+                {
+                    var buffer = new byte[buffer_size];
+                    while (true)
+                    {
+                        var read = webs.Read(buffer, 0, buffer_size);
+                        if (read == 0) break;
+                        Utils.Status($"Downloading {a.Name}", (int)(total_read * 100 / content_size));
+
+                        fs.Write(buffer, 0, read);
+                        total_read += read;
+                    }
+                }
+
+                return true;
+            }
+
+            public override bool Verify()
+            {
+                return DoDownload(new Archive {Name = ""}, "", false);
+            }
+
+            public override IDownloader GetDownloader()
+            {
+                return DownloadDispatcher.GetInstance<HTTPDownloader>();
+            }
+
+            public override string GetReportEntry(Archive a)
+            {
+                return $"* [{a.Name} - {Url}]({Url})";
+            }
+        }
+    }
+}

Wabbajack/Downloaders/IDownloader.cs

@@ -0,0 +1,19 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Wabbajack.Validation;
+
+namespace Wabbajack.Downloaders
+{
+    public interface IDownloader
+    {
+        AbstractDownloadState GetDownloaderState(dynamic archive_ini);
+        
+        /// <summary>
+        /// Called before any downloads are inacted by the installer;
+        /// </summary>
+        void Prepare();
+    }
+}

Wabbajack/Downloaders/MEGADownloader.cs

@@ -0,0 +1,62 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Linq.Expressions;
+using System.Runtime.InteropServices.WindowsRuntime;
+using System.Security.Policy;
+using System.Text;
+using System.Threading.Tasks;
+using Alphaleonis.Win32.Filesystem;
+using CG.Web.MegaApiClient;
+using Wabbajack.Common;
+using Wabbajack.Validation;
+
+namespace Wabbajack.Downloaders
+{
+    public class MegaDownloader : IDownloader
+    {
+
+        public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
+        {
+            var url = archive_ini?.General?.directURL;
+            if (url != null && url.StartsWith(Consts.MegaPrefix))
+                return new State {Url = url};
+            return null;
+        }
+
+        public void Prepare()
+        {
+        }
+
+        public class State : HTTPDownloader.State
+        {
+            public override void Download(Archive a, string destination)
+            {
+                var client = new MegaApiClient();
+                Utils.Status("Logging into MEGA (as anonymous)");
+                client.LoginAnonymous();
+                var file_link = new Uri(Url);
+                var node = client.GetNodeFromLink(file_link);
+                Utils.Status($"Downloading MEGA file: {a.Name}");
+                client.DownloadFile(file_link, destination);
+            }
+
+            public override bool Verify()
+            {
+                var client = new MegaApiClient();
+                Utils.Status("Logging into MEGA (as anonymous)");
+                client.LoginAnonymous();
+                var file_link = new Uri(Url);
+                try
+                {
+                    var node = client.GetNodeFromLink(file_link);
+                    return true;
+                }
+                catch (Exception)
+                {
+                    return false;
+                }
+            }
+        }
+    }
+}

Wabbajack/Downloaders/ManualDownloader.cs

@@ -0,0 +1,52 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Wabbajack.Common;
+using Wabbajack.Validation;
+
+namespace Wabbajack.Downloaders
+{
+    class ManualDownloader : IDownloader
+    {
+        public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
+        {
+            var url = archive_ini?.General?.manualURL;
+            return url != null ? new State { Url = url} : null;
+        }
+
+        public void Prepare()
+        {
+        }
+
+        public class State : AbstractDownloadState
+        {
+            public string Url { get; set; }
+            public override bool IsWhitelisted(ServerWhitelist whitelist)
+            {
+                return true;
+            }
+
+            public override void Download(Archive a, string destination)
+            {
+                Utils.Log($"You must manually visit {Url} and download {a.Name} file by hand.");
+            }
+
+            public override bool Verify()
+            {
+                return true;
+            }
+
+            public override IDownloader GetDownloader()
+            {
+                return DownloadDispatcher.GetInstance<ManualDownloader>();
+            }
+
+            public override string GetReportEntry(Archive a)
+            {
+                return $"* Manual Download - [{a.Name} - {Url}]({Url})";
+            }
+        }
+    }
+}

Wabbajack/Downloaders/ModDBDownloader.cs

@@ -0,0 +1,77 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Net.Http;
+using System.Text;
+using System.Text.RegularExpressions;
+using System.Threading.Tasks;
+using Wabbajack.Common;
+using Wabbajack.Validation;
+
+namespace Wabbajack.Downloaders
+{
+    public class ModDBDownloader : IDownloader
+    {
+        public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
+        {
+            var url = archive_ini?.General?.directURL;
+
+            if (url != null && url.StartsWith("https://www.moddb.com/downloads/start"))
+            {
+                return new State
+                {
+                    Url = url
+                };
+            }
+
+            return null;
+        }
+
+        public void Prepare()
+        {
+            throw new NotImplementedException();
+        }
+
+        public class State : AbstractDownloadState
+        {
+            public string Url { get; set; }
+            public override bool IsWhitelisted(ServerWhitelist whitelist)
+            {
+                // Everything from Moddb is whitelisted
+                return true;
+            }
+
+            public override void Download(Archive a, string destination)
+            {
+                var new_url = GetDownloadUrl();
+                new HTTPDownloader.State {Url = new_url}.Download(a, destination);
+            }
+
+            private string GetDownloadUrl()
+            {
+                var client = new HttpClient();
+                var result = client.GetStringSync(Url);
+                var regex = new Regex("https:\\/\\/www\\.moddb\\.com\\/downloads\\/mirror\\/.*(?=\\\")");
+                var match = regex.Match(result);
+                var new_url = match.Value;
+                return new_url;
+            }
+
+            public override bool Verify()
+            {
+                var new_url = GetDownloadUrl();
+                return new HTTPDownloader.State { Url = new_url }.Verify();
+            }
+
+            public override IDownloader GetDownloader()
+            {
+                return DownloadDispatcher.GetInstance<ModDBDownloader>();
+            }
+
+            public override string GetReportEntry(Archive a)
+            {
+                return $"* ModDB - [{a.Name}]({Url})";
+            }
+        }
+    }
+}

Wabbajack/Downloaders/NexusDownloader.cs

@@ -0,0 +1,134 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Wabbajack.Common;
+using Wabbajack.NexusApi;
+using Wabbajack.Validation;
+
+namespace Wabbajack.Downloaders
+{
+    public class NexusDownloader : IDownloader
+    {
+        public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
+        {
+            var general = archive_ini?.General;
+
+            if (general.modID != null && general.fileID != null && general.gameName != null)
+            {
+                var info = new NexusApiClient().GetModInfo(general.gameName, general.modID);
+                return new State
+                {
+                    GameName = general.gameName,
+                    FileID = general.fileID,
+                    ModID = general.modID,
+                    Version = general.version ?? "0.0.0.0",
+                    Author = info.author,
+                    UploadedBy = info.uploaded_by,
+                    UploaderProfile = info.uploaded_users_profile_url,
+                    ModName = info.name,
+                    SlideShowPic = info.picture_url,
+                    NexusURL = NexusApiUtils.GetModURL(info.game_name, info.mod_id),
+                    Summary = info.summary,
+                    Adult = info.contains_adult_content
+
+                };
+            }
+
+            return null;
+        }
+
+        public void Prepare()
+        {
+            var client = new NexusApiClient();
+            var status = client.GetUserStatus();
+            if (!client.IsAuthenticated)
+            {
+                Utils.Error($"Authenticating for the Nexus failed. A nexus account is required to automatically download mods.");
+                return;
+            }
+
+            if (!status.is_premium)
+            {
+                Utils.Error($"Automated installs with Wabbajack requires a premium nexus account. {client.Username} is not a premium account.");
+                return;
+            }
+        }
+
+        public class State : AbstractDownloadState
+        {
+            public string Author;
+            public string FileID;
+            public string GameName;
+            public string ModID;
+            public string UploadedBy;
+            public string UploaderProfile;
+            public string Version;
+            public string SlideShowPic;
+            public string ModName;
+            public string NexusURL;
+            public string Summary;
+            public bool Adult;
+
+            public override bool IsWhitelisted(ServerWhitelist whitelist)
+            {
+                // Nexus files are always whitelisted
+                return true;
+            }
+
+            public override void Download(Archive a, string destination)
+            {
+                string url;
+                try
+                {
+                    url = new NexusApiClient().GetNexusDownloadLink(this, false);
+                }
+                catch (Exception ex)
+                {
+                    Utils.Log($"{a.Name} - Error Getting Nexus Download URL - {ex.Message}");
+                    return;
+                }
+
+                Utils.Log($"Downloading Nexus Archive - {a.Name} - {GameName} - {ModID} - {FileID}");
+
+                new HTTPDownloader.State
+                {
+                    Url = url
+                }.Download(a, destination);
+
+            }
+
+            public override bool Verify()
+            {
+                try
+                {
+                    new NexusApiClient().GetNexusDownloadLink(this, true);
+                    return true;
+                }
+                catch (Exception ex)
+                {
+                    Utils.Log($"{ModName} - {GameName} - {ModID} - {FileID} - Error Getting Nexus Download URL - {ex.Message}");
+                    return false;
+                }
+
+            }
+
+            public override IDownloader GetDownloader()
+            {
+                return DownloadDispatcher.GetInstance<NexusDownloader>();
+            }
+
+            public override string GetReportEntry(Archive a)
+            {
+                var profile = UploaderProfile.Replace("/games/",
+                    "/" + NexusApiUtils.ConvertGameName(GameName).ToLower() + "/");
+
+                return string.Join("\n", 
+                    $"* [{a.Name}](http://nexusmods.com/{NexusApiUtils.ConvertGameName(GameName)}/mods/{ModID})", 
+                    $"    * Author : [{UploadedBy}]({profile})", 
+                    $"    * Version : {Version}");
+            }
+        }
+    }
+}

Wabbajack/Installer.cs

@@ -11,6 +11,7 @@ using System.Text.RegularExpressions;
 using System.Windows;
 using VFS;
 using Wabbajack.Common;
+using Wabbajack.Downloaders;
 using Wabbajack.NexusApi;
 using Wabbajack.Validation;
 using Directory = Alphaleonis.Win32.Filesystem.Directory;
@@ -162,7 +163,8 @@ namespace Wabbajack
         private void AskToEndorse()
         {
             var mods = ModList.Archives
-                .OfType<NexusMod>()
+                .Select(m => m.State)
+                .OfType<NexusDownloader.State>()
                 .GroupBy(f => (f.GameName, f.ModID))
                 .Select(mod => mod.First())
                 .ToArray();
@@ -419,24 +421,11 @@ namespace Wabbajack
             Info($"Missing {missing.Count} archives");
 
             Info("Getting Nexus API Key, if a browser appears, please accept");
-            if (ModList.Archives.OfType<NexusMod>().Any())
-            {
-                var client = new NexusApiClient();
-                var status = client.GetUserStatus();
-                if (!client.IsAuthenticated)
-                {
-                    Error(
-                        $"Authenticating for the Nexus failed. A nexus account is required to automatically download mods.");
-                    return;
-                }
 
-                if (!status.is_premium)
+            var dispatchers = ModList.Archives.Select(m => m.State.GetDownloader()).Distinct();
-                {
+
-                    Error(
+            foreach (var dispatcher in dispatchers)
-                        $"Automated installs with Wabbajack requires a premium nexus account. {client.Username} is not a premium account.");
+                dispatcher.Prepare();
-                    return;
-                }
-            }
 
             DownloadMissingArchives(missing);
         }
@@ -460,36 +449,7 @@ namespace Wabbajack
         {
             try
             {
-                switch (archive)
+                archive.State.Download(archive, Path.Combine(DownloadFolder, archive.Name));
-                {
-                    case NexusMod a:
-                        string url;
-                        try
-                        {
-                            url = new NexusApiClient().GetNexusDownloadLink(a, !download);
-                            if (!download) return true;
-                        }
-                        catch (Exception ex)
-                        {
-                            Info($"{a.Name} - Error Getting Nexus Download URL - {ex.Message}");
-                            return false;
-                        }
-
-                        Info($"Downloading Nexus Archive - {archive.Name} - {a.GameName} - {a.ModID} - {a.FileID}");
-                        DownloadURLDirect(archive, url);
-                        return true;
-                    case MEGAArchive a:
-                        return DownloadMegaArchive(a, download);
-                    case GoogleDriveMod a:
-                        return DownloadGoogleDriveArchive(a, download);
-                    case MODDBArchive a:
-                        return DownloadModDBArchive(archive, (archive as MODDBArchive).URL, download);
-                    case MediaFireArchive a:
-                        return false;
-                    //return DownloadMediaFireArchive(archive, a.URL, download);
-                    case DirectURLArchive a:
-                        return DownloadURLDirect(archive, a.URL, headers: a.Headers, download: download);
-                }
             }
             catch (Exception ex)
             {
@@ -501,128 +461,6 @@ namespace Wabbajack
             return false;
         }
 
-        private void DownloadMediaFireArchive(Archive a, string url)
-        {
-            var client = new HttpClient();
-            var result = client.GetStringSync(url);
-            var regex = new Regex("(?<= href =\\\").*\\.mediafire\\.com.*(?=\\\")");
-            var confirm = regex.Match(result);
-            DownloadURLDirect(a, confirm.ToString(), client);
-        }
-
-        private bool DownloadMegaArchive(MEGAArchive m, bool download)
-        {
-            var client = new MegaApiClient();
-            Status("Logging into MEGA (as anonymous)");
-            client.LoginAnonymous();
-            var file_link = new Uri(m.URL);
-            var node = client.GetNodeFromLink(file_link);
-            if (!download) return true;
-            Status($"Downloading MEGA file: {m.Name}");
-
-            var output_path = Path.Combine(DownloadFolder, m.Name);
-            client.DownloadFile(file_link, output_path);
-            return true;
-        }
-
-        private bool DownloadGoogleDriveArchive(GoogleDriveMod a, bool download)
-        {
-            var initial_url = $"https://drive.google.com/uc?id={a.Id}&export=download";
-            var client = new HttpClient();
-            var result = client.GetStringSync(initial_url);
-            var regex = new Regex("(?<=/uc\\?export=download&amp;confirm=).*(?=;id=)");
-            var confirm = regex.Match(result);
-            return DownloadURLDirect(a, $"https://drive.google.com/uc?export=download&confirm={confirm}&id={a.Id}",
-                client, download);
-        }
-
-        private bool DownloadModDBArchive(Archive archive, string url, bool download)
-        {
-            var client = new HttpClient();
-            var result = client.GetStringSync(url);
-            var regex = new Regex("https:\\/\\/www\\.moddb\\.com\\/downloads\\/mirror\\/.*(?=\\\")");
-            var match = regex.Match(result);
-            return DownloadURLDirect(archive, match.Value, download: download);
-        }
-
-        private bool DownloadURLDirect(Archive archive, string url, HttpClient client = null, bool download = true,
-            List<string> headers = null)
-        {
-            try
-            {
-                if (client == null)
-                {
-                    client = new HttpClient();
-                    client.DefaultRequestHeaders.Add("User-Agent", Consts.UserAgent);
-                }
-
-                if (headers != null)
-                    foreach (var header in headers)
-                    {
-                        var idx = header.IndexOf(':');
-                        var k = header.Substring(0, idx);
-                        var v = header.Substring(idx + 1);
-                        client.DefaultRequestHeaders.Add(k, v);
-                    }
-
-                long total_read = 0;
-                var buffer_size = 1024 * 32;
-
-                var response = client.GetSync(url);
-                var stream = response.Content.ReadAsStreamAsync();
-                try
-                {
-                    stream.Wait();
-                }
-                catch (Exception ex)
-                {
-                }
-
-                ;
-                if (stream.IsFaulted)
-                {
-                    Info($"While downloading {url} - {stream.Exception.ExceptionToString()}");
-                    return false;
-                }
-
-                if (!download)
-                    return true;
-
-                var header_var = "1";
-                if (response.Content.Headers.Contains("Content-Length"))
-                    header_var = response.Content.Headers.GetValues("Content-Length").FirstOrDefault();
-
-                var content_size = header_var != null ? long.Parse(header_var) : 1;
-
-                var output_path = Path.Combine(DownloadFolder, archive.Name);
-                ;
-
-                using (var webs = stream.Result)
-                using (var fs = File.OpenWrite(output_path))
-                {
-                    var buffer = new byte[buffer_size];
-                    while (true)
-                    {
-                        var read = webs.Read(buffer, 0, buffer_size);
-                        if (read == 0) break;
-                        Status($"Downloading {archive.Name}", (int)(total_read * 100 / content_size));
-
-                        fs.Write(buffer, 0, read);
-                        total_read += read;
-                    }
-                }
-
-                Status($"Hashing {archive.Name}");
-                HashArchive(output_path);
-                return true;
-            }
-            catch (Exception ex)
-            {
-                Info($"{archive.Name} - Error downloading from: {url}");
-                return false;
-            }
-        }
-
         private void HashArchives()
         {
             HashedArchives = Directory.EnumerateFiles(DownloadFolder)

Wabbajack/NexusApi/NexusApi.cs

@@ -11,6 +11,7 @@ using System.Reflection;
 using System.Security.Authentication;
 using System.Threading.Tasks;
 using Wabbajack.Common;
+using Wabbajack.Downloaders;
 using WebSocketSharp;
 using static Wabbajack.NexusApi.NexusApiUtils;
 
@@ -61,6 +62,12 @@ namespace Wabbajack.NexusApi
                     return File.ReadAllText(API_KEY_CACHE_FILE);
                 }
 
+                var env_key = Environment.GetEnvironmentVariable("NEXUSAPIKEY");
+                if (env_key != null)
+                {
+                    return env_key;
+                }
+
                 // open a web socket to receive the api key
                 var guid = Guid.NewGuid();
                 var _websocket = new WebSocket("wss://sso.nexusmods.com")
@@ -154,7 +161,7 @@ namespace Wabbajack.NexusApi
             headers.Add("apikey", _apiKey);
             headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
             headers.Add("Application-Name", Consts.AppName);
-            headers.Add("Application-Version", $"{Assembly.GetEntryAssembly().GetName().Version}");
+            headers.Add("Application-Version", $"{Assembly.GetEntryAssembly()?.GetName()?.Version ?? new Version(0, 1)}");
         }
 
         private T Get<T>(string url)
@@ -172,7 +179,7 @@ namespace Wabbajack.NexusApi
         }
 
 
-        public string GetNexusDownloadLink(NexusMod archive, bool cache = false)
+        public string GetNexusDownloadLink(NexusDownloader.State archive, bool cache = false)
         {
             if (cache && TryGetCachedLink(archive, out var result))
                 return result;
@@ -183,7 +190,7 @@ namespace Wabbajack.NexusApi
             return Get<List<DownloadLink>>(url).First().URI;
         }
 
-        private bool TryGetCachedLink(NexusMod archive, out string result)
+        private bool TryGetCachedLink(NexusDownloader.State archive, out string result)
         {
             if (!Directory.Exists(Consts.NexusCacheDirectory))
                 Directory.CreateDirectory(Consts.NexusCacheDirectory);
@@ -201,20 +208,20 @@ namespace Wabbajack.NexusApi
             return true;
         }
 
-        public NexusFileInfo GetFileInfo(NexusMod mod)
+        public NexusFileInfo GetFileInfo(NexusDownloader.State mod)
         {
             var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(mod.GameName)}/mods/{mod.ModID}/files/{mod.FileID}.json";
             return Get<NexusFileInfo>(url);
         }
 
-        public ModInfo GetModInfo(NexusMod archive)
+        public ModInfo GetModInfo(string gameName, string modId)
         {
             if (!Directory.Exists(Consts.NexusCacheDirectory))
                 Directory.CreateDirectory(Consts.NexusCacheDirectory);
 
             ModInfo result = null;
         TOP:
-            var path = Path.Combine(Consts.NexusCacheDirectory, $"mod-info-{archive.GameName}-{archive.ModID}.json");
+            var path = Path.Combine(Consts.NexusCacheDirectory, $"mod-info-{gameName}-{modId}.json");
             try
             {
                 if (File.Exists(path))
@@ -234,17 +241,17 @@ namespace Wabbajack.NexusApi
                 File.Delete(path);
             }
 
-            var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(archive.GameName)}/mods/{archive.ModID}.json";
+            var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(gameName)}/mods/{modId}.json";
             result = Get<ModInfo>(url);
 
-            result.game_name = archive.GameName;
+            result.game_name = gameName;
-            result.mod_id = archive.ModID;
+            result.mod_id = modId;
             result._internal_version = CACHED_VERSION_NUMBER;
             result.ToJSON(path);
             return result;
         }
 
-        public EndorsementResponse EndorseMod(NexusMod mod)
+        public EndorsementResponse EndorseMod(NexusDownloader.State mod)
         {
             Utils.Status($"Endorsing ${mod.GameName} - ${mod.ModID}");
             var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(mod.GameName)}/mods/{mod.ModID}/endorse.json";

Wabbajack/ReportBuilder.cs

@@ -61,31 +61,7 @@ namespace Wabbajack
             foreach (var archive in SortArchives(lst.Archives))
             {
                 var hash = archive.Hash.FromBase64().ToHEX();
-                switch (archive)
+                NoWrapText(archive.State.GetReportEntry(archive));
-                {
-                    case NexusMod m:
-                        var profile = m.UploaderProfile.Replace("/games/",
-                            "/" + NexusApiUtils.ConvertGameName(m.GameName).ToLower() + "/");
-                        NoWrapText(
-                            $"* [{m.Name}](http://nexusmods.com/{NexusApiUtils.ConvertGameName(m.GameName)}/mods/{m.ModID})");
-                        NoWrapText($"    * Author : [{m.UploadedBy}]({profile})");
-                        NoWrapText($"    * Version : {m.Version}");
-                        break;
-                    case MODDBArchive m:
-                        NoWrapText($"* MODDB - [{m.Name}]({m.URL})");
-                        break;
-                    case MEGAArchive m:
-                        NoWrapText($"* MEGA - [{m.Name}]({m.URL})");
-                        break;
-                    case GoogleDriveMod m:
-                        NoWrapText(
-                            $"* GoogleDrive - [{m.Name}](https://drive.google.com/uc?id={m.Id}&export=download)");
-                        break;
-                    case DirectURLArchive m:
-                        NoWrapText($"* URL - [{m.Name} - {m.URL}]({m.URL})");
-                        break;
-                }
-
                 NoWrapText($"    * Size : {archive.Size.ToFileSizeString()}");
                 NoWrapText($"    * SHA256 : [{hash}](https://www.virustotal.com/gui/file/{hash})");
             }
@@ -147,8 +123,7 @@ namespace Wabbajack
 
         private IEnumerable<Archive> SortArchives(List<Archive> lstArchives)
         {
-            var lst = lstArchives.OfType<NexusMod>().OrderBy(m => m.Author).ThenBy(m => m.Name);
+            return lstArchives.OrderByDescending(a => a.Size);
-            return lst.Concat(lstArchives.Where(m => !(m is NexusMod)).OrderBy(m => m.Name));
         }
     }
 }

Wabbajack/Validation/ValidateModlist.cs

@@ -5,6 +5,7 @@ using System.IO;
 using System.Linq;
 using System.Net.Http;
 using Wabbajack.Common;
+using Wabbajack.Downloaders;
 using YamlDotNet.Serialization;
 using YamlDotNet.Serialization.NamingConventions;
 using Path = Alphaleonis.Win32.Filesystem.Path;
@@ -84,7 +85,7 @@ namespace Wabbajack.Validation
         /// </summary>
         /// <param name="mod"></param>
         /// <returns></returns>
-        public Permissions FilePermissions(NexusMod mod)
+        public Permissions FilePermissions(NexusDownloader.State mod)
         {
             var author_permissions = AuthorPermissions.GetOrDefault(mod.Author)?.Permissions;
             var game_permissions = AuthorPermissions.GetOrDefault(mod.Author)?.Games.GetOrDefault(mod.GameName)?.Permissions;
@@ -111,8 +112,8 @@ namespace Wabbajack.Validation
             ConcurrentStack<string> ValidationErrors = new ConcurrentStack<string>();
             
             var nexus_mod_permissions = modlist.Archives
-                .OfType<NexusMod>()
+                .Where(a => a.State is NexusDownloader.State)
-                .PMap(a => (a.Hash, FilePermissions(a), a))
+                .PMap(a => (a.Hash, FilePermissions((NexusDownloader.State)a.State), a))
                 .ToDictionary(a => a.Hash, a => new { permissions = a.Item2, archive = a.a });
 
             modlist.Directives
@@ -122,13 +123,14 @@ namespace Wabbajack.Validation
                     if (nexus_mod_permissions.TryGetValue(p.ArchiveHashPath[0], out var archive))
                     {
                         var ext = Path.GetExtension(p.ArchiveHashPath.Last());
+                        var url = (archive.archive.State as NexusDownloader.State).NexusURL;
                         if (Consts.AssetFileExtensions.Contains(ext) && !(archive.permissions.CanModifyAssets ?? true))
                         {
-                            ValidationErrors.Push($"{p.To} from {archive.archive.NexusURL} is set to disallow asset modification");
+                            ValidationErrors.Push($"{p.To} from {url} is set to disallow asset modification");
                         }
                         else if (Consts.ESPFileExtensions.Contains(ext) && !(archive.permissions.CanModifyESPs ?? true))
                         {
-                            ValidationErrors.Push($"{p.To} from {archive.archive.NexusURL} is set to disallow asset ESP modification");
+                            ValidationErrors.Push($"{p.To} from {url} is set to disallow asset ESP modification");
                         }
                     }
                 });
@@ -139,10 +141,11 @@ namespace Wabbajack.Validation
                 {
                     if (nexus_mod_permissions.TryGetValue(p.ArchiveHashPath[0], out var archive))
                     {
+                        var url = (archive.archive.State as NexusDownloader.State).NexusURL;
                         if (!(archive.permissions.CanExtractBSAs ?? true) &&
                             p.ArchiveHashPath.Skip(1).ButLast().Any(a => Consts.SupportedBSAs.Contains(Path.GetExtension(a).ToLower())))
                         {
-                            ValidationErrors.Push($"{p.To} from {archive.archive.NexusURL} is set to disallow BSA Extraction");
+                            ValidationErrors.Push($"{p.To} from {url} is set to disallow BSA Extraction");
                         }
                     }
                 });
@@ -150,32 +153,23 @@ namespace Wabbajack.Validation
             var nexus = NexusApi.NexusApiUtils.ConvertGameName(GameRegistry.Games[modlist.GameType].NexusName);
 
             modlist.Archives
-                   .OfType<NexusMod>()
+                   .Where(a => a.State is NexusDownloader.State)
-                   .Where(m => NexusApi.NexusApiUtils.ConvertGameName(m.GameName) != nexus)
+                   .Where(m => NexusApi.NexusApiUtils.ConvertGameName(((NexusDownloader.State)m.State).GameName) != nexus)
                    .Do(m =>
                    {
-                       var permissions = FilePermissions(m);
+                       var permissions = FilePermissions((NexusDownloader.State)m.State);
                        if (!(permissions.CanUseInOtherGames ?? true))
                        {
                            ValidationErrors.Push(
-                               $"The modlist is for {nexus} but {m.Name} is for game type {m.GameName} and is not allowed to be converted to other game types");
+                               $"The modlist is for {nexus} but {m.Name} is for game type {((NexusDownloader.State)m.State).GameName} and is not allowed to be converted to other game types");
                        }
                    });
 
             modlist.Archives
-                   .OfType<GoogleDriveMod>()
+                .Where(m => !m.State.IsWhitelisted(ServerWhitelist))
-                   .PMap(m =>
+                .Do(m =>
                 {
-                if (!ServerWhitelist.GoogleIDs.Contains(m.Id))
+                    ValidationErrors.Push($"{m.Name} is not a whitelisted download");
-                    ValidationErrors.Push($"{m.Name} uses Google Drive id {m.Id} but that id is not in the file whitelist.");
-            });
-
-            modlist.Archives
-                .OfType<DirectURLArchive>()
-                .PMap(m =>
-                {
-                    if (!ServerWhitelist.AllowedPrefixes.Any(prefix => m.URL.StartsWith(prefix)))
-                        ValidationErrors.Push($"{m.Name} will be downloaded from {m.URL} but that URL is not in the server whitelist");
                 });
                 
             return ValidationErrors.ToList();

Wabbajack/Wabbajack.csproj

@@ -188,6 +188,16 @@
       <SubType>Designer</SubType>
     </ApplicationDefinition>
     <Compile Include="Data.cs" />
+    <Compile Include="Downloaders\AbstractDownloadState.cs" />
+    <Compile Include="Downloaders\GoogleDriveDownloader.cs" />
+    <Compile Include="Downloaders\HTTPDownloader.cs" />
+    <Compile Include="Downloaders\DownloadDispatcher.cs" />
+    <Compile Include="Downloaders\DropboxDownloader.cs" />
+    <Compile Include="Downloaders\IDownloader.cs" />
+    <Compile Include="Downloaders\ManualDownloader.cs" />
+    <Compile Include="Downloaders\MegaDownloader.cs" />
+    <Compile Include="Downloaders\ModDBDownloader.cs" />
+    <Compile Include="Downloaders\NexusDownloader.cs" />
     <Compile Include="LambdaCommand.cs" />
     <Compile Include="UI\ModlistPropertiesWindow.xaml.cs">
       <DependentUpon>ModlistPropertiesWindow.xaml</DependentUpon>
@@ -327,6 +337,7 @@
     <EmbeddedResource Include="UI\Icons\github_light.png" />
     <EmbeddedResource Include="UI\Icons\patreon_light.png" />
   </ItemGroup>
+  <ItemGroup />
   <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
   <Import Project="..\packages\Fody.5.1.1\build\Fody.targets" Condition="Exists('..\packages\Fody.5.1.1\build\Fody.targets')" />
   <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">