Mirrored_Repos/wabbajack
1
0
Fork 0
mirror of https://github.com/wabbajack-tools/wabbajack.git synced 2024-08-30 18:42:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #87 from wabbajack-tools/rework-download-integration

Browse Source 
Rework download integration
This commit is contained in:
Timothy Baldridge 2019-10-12 21:19:33 -06:00 committed by GitHub
commit f12aee488d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
26 changed files with 1086 additions and 441 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -15,6 +15,7 @@
* Add BA2 support
* Fix Downloads folder being incorrectly detected in some cases
* Fix validation error on selecting an installation directory in Install mode
* Reworked download code to be more extensible and stable
#### Version 0.9.4 - 10/2/2019
* Point github icon to https://github.com/wabbajack-tools/wabbajack

106
NexusPage.html
View File

@ -275,4 +275,108 @@
<h3 id="faq">FAQ</h3>
<p><strong>How do I get Wabbjack to handle mods from <code>X</code></strong></p>
<p><strong>How do I get Wabbjack to handle mods from <code>X</code></strong></p>
<p>Look at the [<code>RECIPES.md</code>] file, we keep a knowledgebase of how to deal with given types of mods in that file.</p>
<p><strong>How do I contribute to Wabbajack?</strong></p>
<p>Look at the <a href="https://github.com/halgari/wabbajack/blob/master/CONTRIBUTING.md"><code>CONTRIBUTING.md</code></a> file for detailed guidelines.</p>
<p><strong>Why does each modlist install another copy of Mod Organizer 2?</strong></p>
<p>
Self-contained folders are a cleaner abstraction than dumping tons of modlists into the same set of folders. It's easy to uninstall a modlist (simply delete the folder),
and MO2 really isn't designed to support lots of disparate modlists. For example if two modlists both wanted a given texture mod, but different options they would
somehow have to keep the names of their mods separate. MO2 isn't that big of an app, so there's really no reason not to install a new copy for each modlist.
</p>
<p><strong>Why don't I see any mods when I open Mod Organizer 2 after install?</strong></p>
<p>
Make sure you selected the "Portable" mode when starting MO2 for the first time. In addition, make sure you haven't installed MO2 in a non-portable way on the same box.
Really, always use "Portable Mode" it's cleaner and there really isn't a reason not too do so. Make the data self-contained. It's cleaner that way.
</p>
<p><strong>Will Wabbajack ever support Vortex/other mod managers?</strong></p>
<p>
I'll be honest, I don't use anything but MO2, so I probably won't write the code. If someone were to write a patch for the functionality
I wouldn't throw away the code, but it would have to be done in a way that was relatively seamless for users. Since Wabbajack treats all files in the same way
it doesn't know what mod manager a user is using. This means that if the modlist creator used Vortex all users of the modlist would have to use Vortex. This doesn't seem
optimal. It's possible perhaps, but it's at the bottom of the priority list.
</p>
<p><strong>How does Wabbajack differ from Automaton?</strong></p>
<p>
I (halgari) used to be a developer working on Automaton. Sadly development was moving a bit too slowly for my liking, and I realized that a complete rewrite would allow the
implementation of some really nice features (like BSA packing). As such I made the decision to strike out on my own and make an app that worked first, and then make it pretty.
The end result is an app with a ton of features, and a less than professional UI. But that's my motto when coding "make it work, then make it pretty".
</p>
<h2 id="thanksto">Thanks to</h2>
<p>Our tester and Discord members who encourage development and help test the builds.</p>
<h3 id="patreonsupporters">Patreon Supporters</h3>
<h4 id="daedralevelpatreonsupporters">Daedra level Patreon Supporters</h4>
<ul>
<li>Ancalgon</li>
<li>Theo</li>
<li>Dascede</li>
<li>Kristina Poňuchálková</li>
<li>metherul</li>
<li>Decopauge123</li>
</ul>
<h4 id="patreonsupporters-1">Patreon Supporters</h4>
<ul>
<li>Druwski</li>
<li>Soothsayre</li>
<li>krageon</li>
<li>Scumbag</li>
<li>Burt Wheeler</li>
<li>Jesse Earl Rockwell</li>
<li>Mike Gray</li>
<li>Theryl</li>
<li>Daniel Gardner</li>
<li>Dapper</li>
<li>Corapol</li>
<li>HQM</li>
<li>Argos</li>
<li>sorrydaijin</li>
<li>William Chudziak</li>
<li>N Kalim</li>
</ul>
<h3 id="licensecopyright">License &amp; Copyright</h3>
<p>
All original code in Wabbajack is given freely via the GPL3 license. Parts of Wabbajack use libraries that carry their own Open Sources licenses, those parts
retain their original copyrights. Note: Wabbajack installers contain code from Wabbajack. Therefore, selling of modlist files is strictly forbidden. As is hosting
the files behind any sort of paywall. You recieved this tool free of charge, respect this by giving freely as you were given.
</p>

19
README.md
View File

@ -1,4 +1,4 @@
## Wabbajack - An automated modlist installer for TES/Fallout games
## Wabbajack - An automated modlist installer for TES/Fallout games
[![Build Status](https://dev.azure.com/tbaldridge/tbaldridge/_apis/build/status/halgari.wabbajack?branchName=master)](https://dev.azure.com/tbaldridge/tbaldridge/_build/latest?definitionId=1&branchName=master)
@ -209,6 +209,11 @@ Our tester and Discord members who encourage development and help test the build
#### Daedra level Patreon Supporters
- Ancalgon
- Theo
- Dascede
- Kristina Poňuchálková
- metherul
- Decopauge123
#### Patreon Supporters
@ -216,6 +221,18 @@ Our tester and Discord members who encourage development and help test the build
- Soothsayre
- krageon
- Scumbag
- Burt Wheeler
- Jesse Earl Rockwell
- Mike Gray
- Theryl
- Daniel Gardner
- Dapper
- Corapol
- HQM
- Argos
- sorrydaijin
- William Chudziak
- N Kalim
### License & Copyright

4
Wabbajack.Common/Consts.cs
View File

@ -9,7 +9,7 @@ namespace Wabbajack.Common
{
public static bool TestMode { get; set; } = false;
public static string ModlistExtension = ".modlist_v1";
public static string ModlistExtension = ".modlist_v2";
public static string GameFolderFilesDir = "Game Folder Files";
public static string LOOTFolderFilesDir = "LOOT Config Files";
public static string BSACreationDir = "TEMP_BSA_FILES";
@ -57,7 +57,7 @@ namespace Wabbajack.Common
{
var platformType = Environment.Is64BitOperatingSystem ? "x64" : "x86";
var headerString =
$"{AppName}/{Assembly.GetEntryAssembly().GetName().Version} ({Environment.OSVersion.VersionString}; {platformType}) {RuntimeInformation.FrameworkDescription}";
$"{AppName}/{Assembly.GetEntryAssembly()?.GetName()?.Version ?? new Version(0, 1)} ({Environment.OSVersion.VersionString}; {platformType}) {RuntimeInformation.FrameworkDescription}";
return headerString;
}
}

30
Wabbajack.Common/Utils.cs
View File

@ -167,6 +167,16 @@ namespace Wabbajack.Common
return new DynamicIniData(new FileIniDataParser().ReadFile(file));
}
/// <summary>
/// Loads a INI from the given string
/// </summary>
/// <param name="file"></param>
/// <returns></returns>
public static dynamic LoadIniString(this string file)
{
return new DynamicIniData(new FileIniDataParser().ReadData(new StreamReader(new MemoryStream(Encoding.UTF8.GetBytes(file)))));
}
public static void ToJSON<T>(this T obj, string filename)
{
File.WriteAllText(filename,
@ -194,7 +204,7 @@ namespace Wabbajack.Common
public static string ToJSON<T>(this T obj)
{
return JsonConvert.SerializeObject(obj, Formatting.Indented,
new JsonSerializerSettings {TypeNameHandling = TypeNameHandling.Auto});
new JsonSerializerSettings {TypeNameHandling = TypeNameHandling.All});
}
public static T FromJSON<T>(this string filename)
@ -529,5 +539,23 @@ namespace Wabbajack.Common
}
return outarr;
}
/// <summary>
/// Roundtrips the value throught the JSON routines
/// </summary>
/// <typeparam name="TV"></typeparam>
/// <typeparam name="TR"></typeparam>
/// <param name="tv"></param>
/// <returns></returns>
public static T ViaJSON<T>(this T tv)
{
return tv.ToJSON().FromJSONString<T>();
}
public static void Error(string msg)
{
Log(msg);
throw new Exception(msg);
}
}
}

49
Wabbajack.Test/ContentRightsManagementTests.cs
View File

@ -4,6 +4,7 @@ using System.Linq;
using System.Security.Policy;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Wabbajack.Common;
using Wabbajack.Downloaders;
using Wabbajack.Validation;
using Game = Wabbajack.Common.Game;
@ -56,7 +57,7 @@ namespace Wabbajack.Test
[TestMethod]
public void TestRightsFallthrough()
{
var permissions = validate.FilePermissions(new NexusMod()
var permissions = validate.FilePermissions(new NexusDownloader.State
{
Author = "bill",
GameName = "Skyrim",
@ -69,7 +70,7 @@ namespace Wabbajack.Test
permissions.CanModifyAssets.AssertIsFalse();
permissions.CanUseInOtherGames.AssertIsFalse();
permissions = validate.FilePermissions(new NexusMod()
permissions = validate.FilePermissions(new NexusDownloader.State
{
Author = "bob",
GameName = "Skyrim",
@ -82,7 +83,7 @@ namespace Wabbajack.Test
permissions.CanModifyAssets.AssertIsTrue();
permissions.CanUseInOtherGames.AssertIsTrue();
permissions = validate.FilePermissions(new NexusMod()
permissions = validate.FilePermissions(new NexusDownloader.State
{
Author = "bill",
GameName = "Fallout4",
@ -95,7 +96,7 @@ namespace Wabbajack.Test
permissions.CanModifyAssets.AssertIsTrue();
permissions.CanUseInOtherGames.AssertIsTrue();
permissions = validate.FilePermissions(new NexusMod()
permissions = validate.FilePermissions(new NexusDownloader.State
{
Author = "bill",
GameName = "Skyrim",
@ -108,7 +109,7 @@ namespace Wabbajack.Test
permissions.CanModifyAssets.AssertIsTrue();
permissions.CanUseInOtherGames.AssertIsTrue();
permissions = validate.FilePermissions(new NexusMod()
permissions = validate.FilePermissions(new NexusDownloader.State
{
Author = "bill",
GameName = "Skyrim",
@ -131,14 +132,18 @@ namespace Wabbajack.Test
GameType = Game.Skyrim,
Archives = new List<Archive>
{
new NexusMod
new Archive
{
GameName = "Skyrim",
Author = "bill",
ModID = "42",
FileID = "33",
State = new NexusDownloader.State
{
GameName = "Skyrim",
Author = "bill",
ModID = "42",
FileID = "33",
},
Hash = "DEADBEEF"
}
},
Directives = new List<Directive>
{
@ -196,44 +201,44 @@ namespace Wabbajack.Test
// Error due to file downloaded from 3rd party
modlist.GameType = Game.Skyrim;
modlist.Archives[0] = new DirectURLArchive()
modlist.Archives[0] = new Archive()
{
URL = "https://somebadplace.com",
State = new HTTPDownloader.State() { Url = "https://somebadplace.com" },
Hash = "DEADBEEF"
};
errors = validate.Validate(modlist);
Assert.AreEqual(errors.Count(), 1);
Assert.AreEqual(1, errors.Count());
// Ok due to file downloaded from whitelisted 3rd party
modlist.GameType = Game.Skyrim;
modlist.Archives[0] = new DirectURLArchive()
modlist.Archives[0] = new Archive
{
URL = "https://somegoodplace.com/myfile",
State = new HTTPDownloader.State { Url = "https://somegoodplace.com/baz.7z" },
Hash = "DEADBEEF"
};
errors = validate.Validate(modlist);
Assert.AreEqual(errors.Count(), 0);
Assert.AreEqual(0, errors.Count());
// Error due to file downloaded from bad 3rd party
modlist.GameType = Game.Skyrim;
modlist.Archives[0] = new GoogleDriveMod()
modlist.Archives[0] = new Archive
{
Id = "bleg",
State = new GoogleDriveDownloader.State { Id = "bleg"},
Hash = "DEADBEEF"
};
errors = validate.Validate(modlist);
Assert.AreEqual(errors.Count(), 1);
// Error due to file downloaded from good 3rd party
// Ok due to file downloaded from good google site
modlist.GameType = Game.Skyrim;
modlist.Archives[0] = new GoogleDriveMod()
modlist.Archives[0] = new Archive
{
Id = "googleDEADBEEF",
State = new GoogleDriveDownloader.State { Id = "googleDEADBEEF" },
Hash = "DEADBEEF"
};
errors = validate.Validate(modlist);
Assert.AreEqual(errors.Count(), 0);
Assert.AreEqual(0, errors.Count());
}

158
Wabbajack.Test/DownloaderTests.cs Normal file
View File

@ -0,0 +1,158 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Wabbajack.Common;
using Wabbajack.Downloaders;
using Wabbajack.Validation;
using File = Alphaleonis.Win32.Filesystem.File;
namespace Wabbajack.Test
{
[TestClass]
public class DownloaderTests
{
[TestMethod]
public void MegaDownload()
{
var ini = @"[General]
directURL=https://mega.nz/#!CsMSFaaJ!-uziC4mbJPRy2e4pPk8Gjb3oDT_38Be9fzZ6Ld4NL-k";
var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
Assert.IsNotNull(state);
var converted = state.ViaJSON();
Assert.IsTrue(converted.Verify());
var filename = Guid.NewGuid().ToString();
Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist {AllowedPrefixes = new List<string>{"https://mega.nz/#!CsMSFaaJ!-uziC4mbJPRy2e4pPk8Gjb3oDT_38Be9fzZ6Ld4NL-k" } }));
Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string>{ "blerg" }}));
converted.Download(new Archive {Name = "MEGA Test.txt"}, filename);
Assert.AreEqual("Lb1iTsz3iyZeHGs3e94TVmOhf22sqtHLhqkCdXbjiyc=", Utils.FileSHA256(filename));
Assert.AreEqual(File.ReadAllText(filename), "Cheese for Everyone!");
}
[TestMethod]
public void DropboxTests()
{
var ini = @"[General]
directURL=https://www.dropbox.com/s/5hov3m2pboppoc2/WABBAJACK_TEST_FILE.txt?dl=0";
var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
Assert.IsNotNull(state);
var converted = state.ViaJSON();
Assert.IsTrue(converted.Verify());
var filename = Guid.NewGuid().ToString();
Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> { "https://www.dropbox.com/s/5hov3m2pboppoc2/WABBAJACK_TEST_FILE.txt?" } }));
Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> { "blerg" } }));
converted.Download(new Archive { Name = "MEGA Test.txt" }, filename);
Assert.AreEqual("Lb1iTsz3iyZeHGs3e94TVmOhf22sqtHLhqkCdXbjiyc=", Utils.FileSHA256(filename));
Assert.AreEqual(File.ReadAllText(filename), "Cheese for Everyone!");
}
[TestMethod]
public void GoogleDriveTests()
{
var ini = @"[General]
directURL=https://drive.google.com/file/d/1grLRTrpHxlg7VPxATTFNfq2OkU_Plvh_/view?usp=sharing";
var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
Assert.IsNotNull(state);
var converted = state.ViaJSON();
Assert.IsTrue(converted.Verify());
var filename = Guid.NewGuid().ToString();
Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { GoogleIDs = new List<string> { "1grLRTrpHxlg7VPxATTFNfq2OkU_Plvh_" } }));
Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { GoogleIDs = new List<string>()}));
converted.Download(new Archive { Name = "MEGA Test.txt" }, filename);
Assert.AreEqual("Lb1iTsz3iyZeHGs3e94TVmOhf22sqtHLhqkCdXbjiyc=", Utils.FileSHA256(filename));
Assert.AreEqual(File.ReadAllText(filename), "Cheese for Everyone!");
}
[TestMethod]
public void HttpDownload()
{
var ini = @"[General]
directURL=https://raw.githubusercontent.com/wabbajack-tools/opt-out-lists/master/ServerWhitelist.yml";
var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
Assert.IsNotNull(state);
var converted = state.ViaJSON();
Assert.IsTrue(converted.Verify());
var filename = Guid.NewGuid().ToString();
Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> { "https://raw.githubusercontent.com/" } }));
Assert.IsFalse(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string>() }));
converted.Download(new Archive { Name = "Github Test Test.txt" }, filename);
Assert.IsTrue(File.ReadAllText(filename).StartsWith("# Server whitelist for Wabbajack"));
}
[TestMethod]
public void NexusDownload()
{
var ini = @"[General]
gameName=SkyrimSE
modID = 12604
fileID=35407";
var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
Assert.IsNotNull(state);
var converted = state.ViaJSON();
Assert.IsTrue(converted.Verify());
var filename = Guid.NewGuid().ToString();
Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string> () }));
converted.Download(new Archive { Name = "SkyUI.7z" }, filename);
Assert.AreEqual(filename.FileSHA256(), "U3Xg6RBR9XrUY9/jQSu6WKu5dfhHmpaN2dTl0ylDFmI=");
}
[TestMethod]
public void ModDbTests()
{
var ini = @"[General]
directURL=https://www.moddb.com/downloads/start/124908?referer=https%3A%2F%2Fwww.moddb.com%2Fmods%2Fautopause";
var state = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(ini.LoadIniString());
Assert.IsNotNull(state);
var converted = state.ViaJSON();
Assert.IsTrue(converted.Verify());
var filename = Guid.NewGuid().ToString();
Assert.IsTrue(converted.IsWhitelisted(new ServerWhitelist { AllowedPrefixes = new List<string>() }));
converted.Download(new Archive { Name = "moddbtest.7z" }, filename);
Assert.AreEqual("lUvpEjqxfyidBONSHcDy6EnZIPpAD2K4rkJ5ejCXc2k=", filename.FileSHA256());
}
}
}

2
Wabbajack.Test/Wabbajack.Test.csproj
View File

@ -61,6 +61,7 @@
<Reference Include="AlphaFS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=4d31a58f7d7ad5c9, processorArchitecture=MSIL">
<HintPath>..\packages\AlphaFS.2.2.6\lib\net452\AlphaFS.dll</HintPath>
</Reference>
<Reference Include="Microsoft.CSharp" />
<Reference Include="Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
<HintPath>..\packages\MSTest.TestFramework.1.3.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll</HintPath>
</Reference>
@ -73,6 +74,7 @@
<Reference Include="System.Transactions" />
</ItemGroup>
<ItemGroup>
<Compile Include="DownloaderTests.cs" />
<Compile Include="Extensions.cs" />
<Compile Include="TestUtils.cs" />
<Compile Include="SanityTests.cs" />

6
Wabbajack/AppState.cs
View File

@ -15,6 +15,7 @@ using System.Windows.Input;
using System.Windows.Media.Imaging;
using System.Windows.Threading;
using Wabbajack.Common;
using Wabbajack.Downloaders;
using Wabbajack.NexusApi;
using Wabbajack.UI;
@ -484,7 +485,10 @@ namespace Wabbajack
ApplyModlistProperties();
_slideShow.SlideShowElements = modlist.Archives.OfType<NexusMod>().Select(m =>
_slideShow.SlideShowElements = modlist.Archives
.Select(m => m.State)
.OfType<NexusDownloader.State>()
.Select(m =>
new Slide(NexusApiUtils.FixupSummary(m.ModName),m.ModID,
NexusApiUtils.FixupSummary(m.Summary), NexusApiUtils.FixupSummary(m.Author),
m.Adult,m.NexusURL,m.SlideShowPic)).ToList();

113
Wabbajack/Compiler.cs
View File

@ -13,6 +13,7 @@ using System.Text.RegularExpressions;
using System.Web;
using VFS;
using Wabbajack.Common;
using Wabbajack.Downloaders;
using Wabbajack.NexusApi;
using Wabbajack.Validation;
using Directory = Alphaleonis.Win32.Filesystem.Directory;
@ -469,123 +470,21 @@ namespace Wabbajack
{
if (found.IniData == null)
Error($"No download metadata found for {found.Name}, please use MO2 to query info or add a .meta file and try again.");
var general = found.IniData.General;
if (general == null)
Error($"No General section in mod metadata found for {found.Name}, please use MO2 to query info or add the info and try again.");
Archive result;
var result = new Archive();
result.State = (AbstractDownloadState)DownloadDispatcher.ResolveArchive(found.IniData);
if (general.directURL != null && general.directURL.StartsWith("https://drive.google.com"))
{
var regex = new Regex("((?<=id=)[a-zA-Z0-9_-]*)|(?<=\\/file\\/d\\/)[a-zA-Z0-9_-]*");
var match = regex.Match(general.directURL);
result = new GoogleDriveMod
{
Id = match.ToString()
};
}
else if (general.directURL != null && general.directURL.StartsWith(Consts.MegaPrefix))
{
result = new MEGAArchive
{
URL = general.directURL
};
}
else if (general.directURL != null && general.directURL.StartsWith("https://www.dropbox.com/"))
{
var uri = new UriBuilder((string)general.directURL);
var query = HttpUtility.ParseQueryString(uri.Query);
if (query.GetValues("dl").Count() > 0)
query.Remove("dl");
query.Set("dl", "1");
uri.Query = query.ToString();
result = new DirectURLArchive
{
URL = uri.ToString()
};
}
else if (general.directURL != null &&
general.directURL.StartsWith("https://www.moddb.com/downloads/start"))
{
result = new MODDBArchive
{
URL = general.directURL
};
}
else if (general.directURL != null && general.directURL.StartsWith("http://www.mediafire.com/file/"))
{
Error("MediaFire links are not currently supported");
return null;
/*result = new MediaFireArchive()
{
URL = general.directURL
};*/
}
else if (general.directURL != null)
{
var tmp = new DirectURLArchive
{
URL = general.directURL
};
if (general.directURLHeaders != null)
{
tmp.Headers = new List<string>();
tmp.Headers.AddRange(general.directURLHeaders.Split('|'));
}
result = tmp;
}
else if (general.modID != null && general.fileID != null && general.gameName != null)
{
var nm = new NexusMod
{
GameName = general.gameName,
FileID = general.fileID,
ModID = general.modID,
Version = general.version ?? "0.0.0.0"
};
var info = new NexusApiClient().GetModInfo(nm);
nm.Author = info.author;
nm.UploadedBy = info.uploaded_by;
nm.UploaderProfile = info.uploaded_users_profile_url;
nm.ModName = info.name;
nm.SlideShowPic = info.picture_url;
nm.NexusURL = NexusApiUtils.GetModURL(info.game_name, info.mod_id);
nm.Summary = info.summary;
nm.Adult = info.contains_adult_content;
result = nm;
}
else if (general.manualURL != null)
{
result = new ManualArchive
{
URL = general.manualURL,
Notes = general.manualNotes,
};
}
else
{
Error($"No way to handle archive {found.Name} but it's required by the modlist");
return null;
}
if (result.State == null)
Error($"{found.Name} could not be handled by any of the downloaders");
result.Name = found.Name;
result.Hash = found.File.Hash;
result.Meta = found.Meta;
result.Size = found.File.Size;
if (result is ManualArchive) return result;
Info($"Checking link for {found.Name}");
var installer = new Installer("", null, "");
if (!installer.DownloadArchive(result, false))
if (!result.State.Verify())
Error(
$"Unable to resolve link for {found.Name}. If this is hosted on the Nexus the file may have been removed.");

77
Wabbajack/Data.cs
View File

@ -4,6 +4,7 @@ using System.Collections.Generic;
using Compression.BSA;
using VFS;
using Wabbajack.Common;
using Wabbajack.Downloaders;
namespace Wabbajack
{
@ -218,81 +219,7 @@ namespace Wabbajack
public string Name;
public long Size;
}
[Serializable]
public class NexusMod : Archive
{
public string Author;
public string FileID;
public string GameName;
public string ModID;
public string UploadedBy;
public string UploaderProfile;
public string Version;
public string SlideShowPic;
public string ModName;
public string NexusURL;
public string Summary;
public bool Adult;
}
[Serializable]
public class ManualArchive : Archive
{
public string URL;
public string Notes;
}
[Serializable]
public class GoogleDriveMod : Archive
{
public string Id;
}
/// <summary>
/// URL that can be downloaded directly without any additional options
/// </summary>
[Serializable]
public class DirectURLArchive : Archive
{
[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
public List<string> Headers;
public string URL;
}
/// <summary>
/// An archive that requires additional HTTP headers.
/// </summary>
[Serializable]
public class DirectURLArchiveEx : DirectURLArchive
{
public Dictionary<string, string> Headers;
}
/// <summary>
/// Archive that comes from MEGA
/// </summary>
[Serializable]
public class MEGAArchive : DirectURLArchive
{
}
/// <summary>
/// Archive that comes from MODDB
/// </summary>
[Serializable]
public class MODDBArchive : DirectURLArchive
{
}
/// <summary>
/// Archive that comes from MediaFire
/// </summary>
[Serializable]
public class MediaFireArchive : DirectURLArchive
{
public AbstractDownloadState State { get; set; }
}
[Serializable]

38
Wabbajack/Downloaders/AbstractDownloadState.cs Normal file
View File

@ -0,0 +1,38 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Wabbajack.Validation;
namespace Wabbajack.Downloaders
{
/// <summary>
/// Base for all abstract downloaders
/// </summary>
public abstract class AbstractDownloadState
{
/// <summary>
/// Returns true if this file is allowed to be downloaded via whitelist
/// </summary>
/// <param name="whitelist"></param>
/// <returns></returns>
public abstract bool IsWhitelisted(ServerWhitelist whitelist);
/// <summary>
/// Downloads this file to the given destination location
/// </summary>
/// <param name="destination"></param>
public abstract void Download(Archive a, string destination);
/// <summary>
/// Returns true if this link is still valid
/// </summary>
/// <returns></returns>
public abstract bool Verify();
public abstract IDownloader GetDownloader();
public abstract string GetReportEntry(Archive a);
}
}

41
Wabbajack/Downloaders/DownloadDispatcher.cs Normal file
View File

@ -0,0 +1,41 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Policy;
using System.Text;
using System.Threading.Tasks;
namespace Wabbajack.Downloaders
{
public static class DownloadDispatcher
{
private static readonly List<IDownloader> Downloaders = new List<IDownloader>()
{
new MegaDownloader(),
new DropboxDownloader(),
new GoogleDriveDownloader(),
new ModDBDownloader(),
new NexusDownloader(),
new ManualDownloader(),
new HTTPDownloader()
};
private static readonly Dictionary<Type, IDownloader> IndexedDownloaders;
static DownloadDispatcher()
{
IndexedDownloaders = Downloaders.ToDictionary(d => d.GetType());
}
public static T GetInstance<T>()
{
return (T)IndexedDownloaders[typeof(T)];
}
public static AbstractDownloadState ResolveArchive(dynamic ini)
{
return Downloaders.Select(d => d.GetDownloaderState(ini)).FirstOrDefault(result => result != null);
}
}
}

37
Wabbajack/Downloaders/DropboxDownloader.cs Normal file
View File

@ -0,0 +1,37 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Web;
namespace Wabbajack.Downloaders
{
public class DropboxDownloader : IDownloader
{
public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
{
var urlstring = archive_ini?.General?.directURL;
if (urlstring == null) return null;
var uri = new UriBuilder((string)urlstring);
if (uri.Host != "www.dropbox.com") return null;
var query = HttpUtility.ParseQueryString(uri.Query);
if (query.GetValues("dl").Length > 0)
query.Remove("dl");
query.Set("dl", "1");
uri.Query = query.ToString();
return new HTTPDownloader.State()
{
Url = uri.ToString().Replace("dropbox.com:443/", "dropbox.com/")
};
}
public void Prepare()
{
}
}
}

77
Wabbajack/Downloaders/GoogleDriveDownloader.cs Normal file
View File

@ -0,0 +1,77 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using Wabbajack.Common;
using Wabbajack.Validation;
namespace Wabbajack.Downloaders
{
public class GoogleDriveDownloader : IDownloader
{
public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
{
var url = archive_ini?.General?.directURL;
var regex = new Regex("((?<=id=)[a-zA-Z0-9_-]*)|(?<=\\/file\\/d\\/)[a-zA-Z0-9_-]*");
if (url != null && url.StartsWith("https://drive.google.com"))
{
var match = regex.Match(url);
return new State
{
Id = match.ToString()
};
}
return null;
}
public void Prepare()
{
}
public class State : AbstractDownloadState
{
public string Id { get; set; }
public override bool IsWhitelisted(ServerWhitelist whitelist)
{
return whitelist.GoogleIDs.Contains(Id);
}
public override void Download(Archive a, string destination)
{
ToHttpState().Download(a, destination);
}
private HTTPDownloader.State ToHttpState()
{
var initial_url = $"https://drive.google.com/uc?id={Id}&export=download";
var client = new HttpClient();
var result = client.GetStringSync(initial_url);
var regex = new Regex("(?<=/uc\\?export=download&amp;confirm=).*(?=;id=)");
var confirm = regex.Match(result);
var url = $"https://drive.google.com/uc?export=download&confirm={confirm}&id={Id}";
var http_state = new HTTPDownloader.State {Url = url};
return http_state;
}
public override bool Verify()
{
return ToHttpState().Verify();
}
public override IDownloader GetDownloader()
{
return DownloadDispatcher.GetInstance<GoogleDriveDownloader>();
}
public override string GetReportEntry(Archive a)
{
return $"* GoogleDrive - [{a.Name}](https://drive.google.com/uc?id={Id}&export=download)";
}
}
}
}

138
Wabbajack/Downloaders/HTTPDownloader.cs Normal file
View File

@ -0,0 +1,138 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.ServiceModel.Configuration;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;
using Wabbajack.Common;
using Wabbajack.Validation;
using File = Alphaleonis.Win32.Filesystem.File;
namespace Wabbajack.Downloaders
{
public class HTTPDownloader : IDownloader
{
public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
{
var url = archive_ini?.General?.directURL;
if (url != null)
{
var tmp = new State
{
Url = url
};
if (archive_ini?.General?.directURLHeaders != null)
{
tmp.Headers = new List<string>();
tmp.Headers.AddRange(archive_ini?.General.directURLHeaders.Split('|'));
}
return tmp;
}
return null;
}
public void Prepare()
{
}
public class State : AbstractDownloadState
{
public string Url { get; set; }
[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
public List<string> Headers { get; set; }
public override bool IsWhitelisted(ServerWhitelist whitelist)
{
return whitelist.AllowedPrefixes.Any(p => Url.StartsWith(p));
}
public override void Download(Archive a, string destination)
{
DoDownload(a, destination, true);
}
public bool DoDownload(Archive a, string destination, bool download)
{
var client = new HttpClient();
client.DefaultRequestHeaders.Add("User-Agent", Consts.UserAgent);
if (Headers != null)
foreach (var header in Headers)
{
var idx = header.IndexOf(':');
var k = header.Substring(0, idx);
var v = header.Substring(idx + 1);
client.DefaultRequestHeaders.Add(k, v);
}
long total_read = 0;
var buffer_size = 1024 * 32;
var response = client.GetSync(Url);
var stream = response.Content.ReadAsStreamAsync();
try
{
stream.Wait();
}
catch (Exception ex)
{
}
;
if (stream.IsFaulted)
{
Utils.Log($"While downloading {Url} - {stream.Exception.ExceptionToString()}");
return false;
}
if (!download)
return true;
var header_var = "1";
if (response.Content.Headers.Contains("Content-Length"))
header_var = response.Content.Headers.GetValues("Content-Length").FirstOrDefault();
var content_size = header_var != null ? long.Parse(header_var) : 1;
using (var webs = stream.Result)
using (var fs = File.OpenWrite(destination))
{
var buffer = new byte[buffer_size];
while (true)
{
var read = webs.Read(buffer, 0, buffer_size);
if (read == 0) break;
Utils.Status($"Downloading {a.Name}", (int)(total_read * 100 / content_size));
fs.Write(buffer, 0, read);
total_read += read;
}
}
return true;
}
public override bool Verify()
{
return DoDownload(new Archive {Name = ""}, "", false);
}
public override IDownloader GetDownloader()
{
return DownloadDispatcher.GetInstance<HTTPDownloader>();
}
public override string GetReportEntry(Archive a)
{
return $"* [{a.Name} - {Url}]({Url})";
}
}
}
}

19
Wabbajack/Downloaders/IDownloader.cs Normal file
View File

@ -0,0 +1,19 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Wabbajack.Validation;
namespace Wabbajack.Downloaders
{
public interface IDownloader
{
AbstractDownloadState GetDownloaderState(dynamic archive_ini);
/// <summary>
/// Called before any downloads are inacted by the installer;
/// </summary>
void Prepare();
}
}

62
Wabbajack/Downloaders/MEGADownloader.cs Normal file
View File

@ -0,0 +1,62 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Runtime.InteropServices.WindowsRuntime;
using System.Security.Policy;
using System.Text;
using System.Threading.Tasks;
using Alphaleonis.Win32.Filesystem;
using CG.Web.MegaApiClient;
using Wabbajack.Common;
using Wabbajack.Validation;
namespace Wabbajack.Downloaders
{
public class MegaDownloader : IDownloader
{
public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
{
var url = archive_ini?.General?.directURL;
if (url != null && url.StartsWith(Consts.MegaPrefix))
return new State {Url = url};
return null;
}
public void Prepare()
{
}
public class State : HTTPDownloader.State
{
public override void Download(Archive a, string destination)
{
var client = new MegaApiClient();
Utils.Status("Logging into MEGA (as anonymous)");
client.LoginAnonymous();
var file_link = new Uri(Url);
var node = client.GetNodeFromLink(file_link);
Utils.Status($"Downloading MEGA file: {a.Name}");
client.DownloadFile(file_link, destination);
}
public override bool Verify()
{
var client = new MegaApiClient();
Utils.Status("Logging into MEGA (as anonymous)");
client.LoginAnonymous();
var file_link = new Uri(Url);
try
{
var node = client.GetNodeFromLink(file_link);
return true;
}
catch (Exception)
{
return false;
}
}
}
}
}

52
Wabbajack/Downloaders/ManualDownloader.cs Normal file
View File

@ -0,0 +1,52 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Wabbajack.Common;
using Wabbajack.Validation;
namespace Wabbajack.Downloaders
{
class ManualDownloader : IDownloader
{
public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
{
var url = archive_ini?.General?.manualURL;
return url != null ? new State { Url = url} : null;
}
public void Prepare()
{
}
public class State : AbstractDownloadState
{
public string Url { get; set; }
public override bool IsWhitelisted(ServerWhitelist whitelist)
{
return true;
}
public override void Download(Archive a, string destination)
{
Utils.Log($"You must manually visit {Url} and download {a.Name} file by hand.");
}
public override bool Verify()
{
return true;
}
public override IDownloader GetDownloader()
{
return DownloadDispatcher.GetInstance<ManualDownloader>();
}
public override string GetReportEntry(Archive a)
{
return $"* Manual Download - [{a.Name} - {Url}]({Url})";
}
}
}
}

77
Wabbajack/Downloaders/ModDBDownloader.cs Normal file
View File

@ -0,0 +1,77 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using Wabbajack.Common;
using Wabbajack.Validation;
namespace Wabbajack.Downloaders
{
public class ModDBDownloader : IDownloader
{
public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
{
var url = archive_ini?.General?.directURL;
if (url != null && url.StartsWith("https://www.moddb.com/downloads/start"))
{
return new State
{
Url = url
};
}
return null;
}
public void Prepare()
{
throw new NotImplementedException();
}
public class State : AbstractDownloadState
{
public string Url { get; set; }
public override bool IsWhitelisted(ServerWhitelist whitelist)
{
// Everything from Moddb is whitelisted
return true;
}
public override void Download(Archive a, string destination)
{
var new_url = GetDownloadUrl();
new HTTPDownloader.State {Url = new_url}.Download(a, destination);
}
private string GetDownloadUrl()
{
var client = new HttpClient();
var result = client.GetStringSync(Url);
var regex = new Regex("https:\\/\\/www\\.moddb\\.com\\/downloads\\/mirror\\/.*(?=\\\")");
var match = regex.Match(result);
var new_url = match.Value;
return new_url;
}
public override bool Verify()
{
var new_url = GetDownloadUrl();
return new HTTPDownloader.State { Url = new_url }.Verify();
}
public override IDownloader GetDownloader()
{
return DownloadDispatcher.GetInstance<ModDBDownloader>();
}
public override string GetReportEntry(Archive a)
{
return $"* ModDB - [{a.Name}]({Url})";
}
}
}
}

134
Wabbajack/Downloaders/NexusDownloader.cs Normal file
View File

@ -0,0 +1,134 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Wabbajack.Common;
using Wabbajack.NexusApi;
using Wabbajack.Validation;
namespace Wabbajack.Downloaders
{
public class NexusDownloader : IDownloader
{
public AbstractDownloadState GetDownloaderState(dynamic archive_ini)
{
var general = archive_ini?.General;
if (general.modID != null && general.fileID != null && general.gameName != null)
{
var info = new NexusApiClient().GetModInfo(general.gameName, general.modID);
return new State
{
GameName = general.gameName,
FileID = general.fileID,
ModID = general.modID,
Version = general.version ?? "0.0.0.0",
Author = info.author,
UploadedBy = info.uploaded_by,
UploaderProfile = info.uploaded_users_profile_url,
ModName = info.name,
SlideShowPic = info.picture_url,
NexusURL = NexusApiUtils.GetModURL(info.game_name, info.mod_id),
Summary = info.summary,
Adult = info.contains_adult_content
};
}
return null;
}
public void Prepare()
{
var client = new NexusApiClient();
var status = client.GetUserStatus();
if (!client.IsAuthenticated)
{
Utils.Error($"Authenticating for the Nexus failed. A nexus account is required to automatically download mods.");
return;
}
if (!status.is_premium)
{
Utils.Error($"Automated installs with Wabbajack requires a premium nexus account. {client.Username} is not a premium account.");
return;
}
}
public class State : AbstractDownloadState
{
public string Author;
public string FileID;
public string GameName;
public string ModID;
public string UploadedBy;
public string UploaderProfile;
public string Version;
public string SlideShowPic;
public string ModName;
public string NexusURL;
public string Summary;
public bool Adult;
public override bool IsWhitelisted(ServerWhitelist whitelist)
{
// Nexus files are always whitelisted
return true;
}
public override void Download(Archive a, string destination)
{
string url;
try
{
url = new NexusApiClient().GetNexusDownloadLink(this, false);
}
catch (Exception ex)
{
Utils.Log($"{a.Name} - Error Getting Nexus Download URL - {ex.Message}");
return;
}
Utils.Log($"Downloading Nexus Archive - {a.Name} - {GameName} - {ModID} - {FileID}");
new HTTPDownloader.State
{
Url = url
}.Download(a, destination);
}
public override bool Verify()
{
try
{
new NexusApiClient().GetNexusDownloadLink(this, true);
return true;
}
catch (Exception ex)
{
Utils.Log($"{ModName} - {GameName} - {ModID} - {FileID} - Error Getting Nexus Download URL - {ex.Message}");
return false;
}
}
public override IDownloader GetDownloader()
{
return DownloadDispatcher.GetInstance<NexusDownloader>();
}
public override string GetReportEntry(Archive a)
{
var profile = UploaderProfile.Replace("/games/",
"/" + NexusApiUtils.ConvertGameName(GameName).ToLower() + "/");
return string.Join("\n",
$"* [{a.Name}](http://nexusmods.com/{NexusApiUtils.ConvertGameName(GameName)}/mods/{ModID})",
$" * Author : [{UploadedBy}]({profile})",
$" * Version : {Version}");
}
}
}
}

178
Wabbajack/Installer.cs
View File

@ -11,6 +11,7 @@ using System.Text.RegularExpressions;
using System.Windows;
using VFS;
using Wabbajack.Common;
using Wabbajack.Downloaders;
using Wabbajack.NexusApi;
using Wabbajack.Validation;
using Directory = Alphaleonis.Win32.Filesystem.Directory;
@ -162,7 +163,8 @@ namespace Wabbajack
private void AskToEndorse()
{
var mods = ModList.Archives
.OfType<NexusMod>()
.Select(m => m.State)
.OfType<NexusDownloader.State>()
.GroupBy(f => (f.GameName, f.ModID))
.Select(mod => mod.First())
.ToArray();
@ -419,24 +421,11 @@ namespace Wabbajack
Info($"Missing {missing.Count} archives");
Info("Getting Nexus API Key, if a browser appears, please accept");
if (ModList.Archives.OfType<NexusMod>().Any())
{
var client = new NexusApiClient();
var status = client.GetUserStatus();
if (!client.IsAuthenticated)
{
Error(
$"Authenticating for the Nexus failed. A nexus account is required to automatically download mods.");
return;
}
if (!status.is_premium)
{
Error(
$"Automated installs with Wabbajack requires a premium nexus account. {client.Username} is not a premium account.");
return;
}
}
var dispatchers = ModList.Archives.Select(m => m.State.GetDownloader()).Distinct();
foreach (var dispatcher in dispatchers)
dispatcher.Prepare();
DownloadMissingArchives(missing);
}
@ -460,36 +449,7 @@ namespace Wabbajack
{
try
{
switch (archive)
{
case NexusMod a:
string url;
try
{
url = new NexusApiClient().GetNexusDownloadLink(a, !download);
if (!download) return true;
}
catch (Exception ex)
{
Info($"{a.Name} - Error Getting Nexus Download URL - {ex.Message}");
return false;
}
Info($"Downloading Nexus Archive - {archive.Name} - {a.GameName} - {a.ModID} - {a.FileID}");
DownloadURLDirect(archive, url);
return true;
case MEGAArchive a:
return DownloadMegaArchive(a, download);
case GoogleDriveMod a:
return DownloadGoogleDriveArchive(a, download);
case MODDBArchive a:
return DownloadModDBArchive(archive, (archive as MODDBArchive).URL, download);
case MediaFireArchive a:
return false;
//return DownloadMediaFireArchive(archive, a.URL, download);
case DirectURLArchive a:
return DownloadURLDirect(archive, a.URL, headers: a.Headers, download: download);
}
archive.State.Download(archive, Path.Combine(DownloadFolder, archive.Name));
}
catch (Exception ex)
{
@ -501,128 +461,6 @@ namespace Wabbajack
return false;
}
private void DownloadMediaFireArchive(Archive a, string url)
{
var client = new HttpClient();
var result = client.GetStringSync(url);
var regex = new Regex("(?<= href =\\\").*\\.mediafire\\.com.*(?=\\\")");
var confirm = regex.Match(result);
DownloadURLDirect(a, confirm.ToString(), client);
}
private bool DownloadMegaArchive(MEGAArchive m, bool download)
{
var client = new MegaApiClient();
Status("Logging into MEGA (as anonymous)");
client.LoginAnonymous();
var file_link = new Uri(m.URL);
var node = client.GetNodeFromLink(file_link);
if (!download) return true;
Status($"Downloading MEGA file: {m.Name}");
var output_path = Path.Combine(DownloadFolder, m.Name);
client.DownloadFile(file_link, output_path);
return true;
}
private bool DownloadGoogleDriveArchive(GoogleDriveMod a, bool download)
{
var initial_url = $"https://drive.google.com/uc?id={a.Id}&export=download";
var client = new HttpClient();
var result = client.GetStringSync(initial_url);
var regex = new Regex("(?<=/uc\\?export=download&amp;confirm=).*(?=;id=)");
var confirm = regex.Match(result);
return DownloadURLDirect(a, $"https://drive.google.com/uc?export=download&confirm={confirm}&id={a.Id}",
client, download);
}
private bool DownloadModDBArchive(Archive archive, string url, bool download)
{
var client = new HttpClient();
var result = client.GetStringSync(url);
var regex = new Regex("https:\\/\\/www\\.moddb\\.com\\/downloads\\/mirror\\/.*(?=\\\")");
var match = regex.Match(result);
return DownloadURLDirect(archive, match.Value, download: download);
}
private bool DownloadURLDirect(Archive archive, string url, HttpClient client = null, bool download = true,
List<string> headers = null)
{
try
{
if (client == null)
{
client = new HttpClient();
client.DefaultRequestHeaders.Add("User-Agent", Consts.UserAgent);
}
if (headers != null)
foreach (var header in headers)
{
var idx = header.IndexOf(':');
var k = header.Substring(0, idx);
var v = header.Substring(idx + 1);
client.DefaultRequestHeaders.Add(k, v);
}
long total_read = 0;
var buffer_size = 1024 * 32;
var response = client.GetSync(url);
var stream = response.Content.ReadAsStreamAsync();
try
{
stream.Wait();
}
catch (Exception ex)
{
}
;
if (stream.IsFaulted)
{
Info($"While downloading {url} - {stream.Exception.ExceptionToString()}");
return false;
}
if (!download)
return true;
var header_var = "1";
if (response.Content.Headers.Contains("Content-Length"))
header_var = response.Content.Headers.GetValues("Content-Length").FirstOrDefault();
var content_size = header_var != null ? long.Parse(header_var) : 1;
var output_path = Path.Combine(DownloadFolder, archive.Name);
;
using (var webs = stream.Result)
using (var fs = File.OpenWrite(output_path))
{
var buffer = new byte[buffer_size];
while (true)
{
var read = webs.Read(buffer, 0, buffer_size);
if (read == 0) break;
Status($"Downloading {archive.Name}", (int)(total_read * 100 / content_size));
fs.Write(buffer, 0, read);
total_read += read;
}
}
Status($"Hashing {archive.Name}");
HashArchive(output_path);
return true;
}
catch (Exception ex)
{
Info($"{archive.Name} - Error downloading from: {url}");
return false;
}
}
private void HashArchives()
{
HashedArchives = Directory.EnumerateFiles(DownloadFolder)

27
Wabbajack/NexusApi/NexusApi.cs
View File

@ -11,6 +11,7 @@ using System.Reflection;
using System.Security.Authentication;
using System.Threading.Tasks;
using Wabbajack.Common;
using Wabbajack.Downloaders;
using WebSocketSharp;
using static Wabbajack.NexusApi.NexusApiUtils;
@ -61,6 +62,12 @@ namespace Wabbajack.NexusApi
return File.ReadAllText(API_KEY_CACHE_FILE);
}
var env_key = Environment.GetEnvironmentVariable("NEXUSAPIKEY");
if (env_key != null)
{
return env_key;
}
// open a web socket to receive the api key
var guid = Guid.NewGuid();
var _websocket = new WebSocket("wss://sso.nexusmods.com")
@ -154,7 +161,7 @@ namespace Wabbajack.NexusApi
headers.Add("apikey", _apiKey);
headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
headers.Add("Application-Name", Consts.AppName);
headers.Add("Application-Version", $"{Assembly.GetEntryAssembly().GetName().Version}");
headers.Add("Application-Version", $"{Assembly.GetEntryAssembly()?.GetName()?.Version ?? new Version(0, 1)}");
}
private T Get<T>(string url)
@ -172,7 +179,7 @@ namespace Wabbajack.NexusApi
}
public string GetNexusDownloadLink(NexusMod archive, bool cache = false)
public string GetNexusDownloadLink(NexusDownloader.State archive, bool cache = false)
{
if (cache && TryGetCachedLink(archive, out var result))
return result;
@ -183,7 +190,7 @@ namespace Wabbajack.NexusApi
return Get<List<DownloadLink>>(url).First().URI;
}
private bool TryGetCachedLink(NexusMod archive, out string result)
private bool TryGetCachedLink(NexusDownloader.State archive, out string result)
{
if (!Directory.Exists(Consts.NexusCacheDirectory))
Directory.CreateDirectory(Consts.NexusCacheDirectory);
@ -201,20 +208,20 @@ namespace Wabbajack.NexusApi
return true;
}
public NexusFileInfo GetFileInfo(NexusMod mod)
public NexusFileInfo GetFileInfo(NexusDownloader.State mod)
{
var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(mod.GameName)}/mods/{mod.ModID}/files/{mod.FileID}.json";
return Get<NexusFileInfo>(url);
}
public ModInfo GetModInfo(NexusMod archive)
public ModInfo GetModInfo(string gameName, string modId)
{
if (!Directory.Exists(Consts.NexusCacheDirectory))
Directory.CreateDirectory(Consts.NexusCacheDirectory);
ModInfo result = null;
TOP:
var path = Path.Combine(Consts.NexusCacheDirectory, $"mod-info-{archive.GameName}-{archive.ModID}.json");
var path = Path.Combine(Consts.NexusCacheDirectory, $"mod-info-{gameName}-{modId}.json");
try
{
if (File.Exists(path))
@ -234,17 +241,17 @@ namespace Wabbajack.NexusApi
File.Delete(path);
}
var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(archive.GameName)}/mods/{archive.ModID}.json";
var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(gameName)}/mods/{modId}.json";
result = Get<ModInfo>(url);
result.game_name = archive.GameName;
result.mod_id = archive.ModID;
result.game_name = gameName;
result.mod_id = modId;
result._internal_version = CACHED_VERSION_NUMBER;
result.ToJSON(path);
return result;
}
public EndorsementResponse EndorseMod(NexusMod mod)
public EndorsementResponse EndorseMod(NexusDownloader.State mod)
{
Utils.Status($"Endorsing ${mod.GameName} - ${mod.ModID}");
var url = $"https://api.nexusmods.com/v1/games/{ConvertGameName(mod.GameName)}/mods/{mod.ModID}/endorse.json";

29
Wabbajack/ReportBuilder.cs
View File

@ -61,31 +61,7 @@ namespace Wabbajack
foreach (var archive in SortArchives(lst.Archives))
{
var hash = archive.Hash.FromBase64().ToHEX();
switch (archive)
{
case NexusMod m:
var profile = m.UploaderProfile.Replace("/games/",
"/" + NexusApiUtils.ConvertGameName(m.GameName).ToLower() + "/");
NoWrapText(
$"* [{m.Name}](http://nexusmods.com/{NexusApiUtils.ConvertGameName(m.GameName)}/mods/{m.ModID})");
NoWrapText($" * Author : [{m.UploadedBy}]({profile})");
NoWrapText($" * Version : {m.Version}");
break;
case MODDBArchive m:
NoWrapText($"* MODDB - [{m.Name}]({m.URL})");
break;
case MEGAArchive m:
NoWrapText($"* MEGA - [{m.Name}]({m.URL})");
break;
case GoogleDriveMod m:
NoWrapText(
$"* GoogleDrive - [{m.Name}](https://drive.google.com/uc?id={m.Id}&export=download)");
break;
case DirectURLArchive m:
NoWrapText($"* URL - [{m.Name} - {m.URL}]({m.URL})");
break;
}
NoWrapText(archive.State.GetReportEntry(archive));
NoWrapText($" * Size : {archive.Size.ToFileSizeString()}");
NoWrapText($" * SHA256 : [{hash}](https://www.virustotal.com/gui/file/{hash})");
}
@ -147,8 +123,7 @@ namespace Wabbajack
private IEnumerable<Archive> SortArchives(List<Archive> lstArchives)
{
var lst = lstArchives.OfType<NexusMod>().OrderBy(m => m.Author).ThenBy(m => m.Name);
return lst.Concat(lstArchives.Where(m => !(m is NexusMod)).OrderBy(m => m.Name));
return lstArchives.OrderByDescending(a => a.Size);
}
}
}

42
Wabbajack/Validation/ValidateModlist.cs
View File

@ -5,6 +5,7 @@ using System.IO;
using System.Linq;
using System.Net.Http;
using Wabbajack.Common;
using Wabbajack.Downloaders;
using YamlDotNet.Serialization;
using YamlDotNet.Serialization.NamingConventions;
using Path = Alphaleonis.Win32.Filesystem.Path;
@ -84,7 +85,7 @@ namespace Wabbajack.Validation
/// </summary>
/// <param name="mod"></param>
/// <returns></returns>
public Permissions FilePermissions(NexusMod mod)
public Permissions FilePermissions(NexusDownloader.State mod)
{
var author_permissions = AuthorPermissions.GetOrDefault(mod.Author)?.Permissions;
var game_permissions = AuthorPermissions.GetOrDefault(mod.Author)?.Games.GetOrDefault(mod.GameName)?.Permissions;
@ -109,10 +110,10 @@ namespace Wabbajack.Validation
public IEnumerable<string> Validate(ModList modlist)
{
ConcurrentStack<string> ValidationErrors = new ConcurrentStack<string>();
var nexus_mod_permissions = modlist.Archives
.OfType<NexusMod>()
.PMap(a => (a.Hash, FilePermissions(a), a))
.Where(a => a.State is NexusDownloader.State)
.PMap(a => (a.Hash, FilePermissions((NexusDownloader.State)a.State), a))
.ToDictionary(a => a.Hash, a => new { permissions = a.Item2, archive = a.a });
modlist.Directives
@ -122,13 +123,14 @@ namespace Wabbajack.Validation
if (nexus_mod_permissions.TryGetValue(p.ArchiveHashPath[0], out var archive))
{
var ext = Path.GetExtension(p.ArchiveHashPath.Last());
var url = (archive.archive.State as NexusDownloader.State).NexusURL;
if (Consts.AssetFileExtensions.Contains(ext) && !(archive.permissions.CanModifyAssets ?? true))
{
ValidationErrors.Push($"{p.To} from {archive.archive.NexusURL} is set to disallow asset modification");
ValidationErrors.Push($"{p.To} from {url} is set to disallow asset modification");
}
else if (Consts.ESPFileExtensions.Contains(ext) && !(archive.permissions.CanModifyESPs ?? true))
{
ValidationErrors.Push($"{p.To} from {archive.archive.NexusURL} is set to disallow asset ESP modification");
ValidationErrors.Push($"{p.To} from {url} is set to disallow asset ESP modification");
}
}
});
@ -139,10 +141,11 @@ namespace Wabbajack.Validation
{
if (nexus_mod_permissions.TryGetValue(p.ArchiveHashPath[0], out var archive))
{
var url = (archive.archive.State as NexusDownloader.State).NexusURL;
if (!(archive.permissions.CanExtractBSAs ?? true) &&
p.ArchiveHashPath.Skip(1).ButLast().Any(a => Consts.SupportedBSAs.Contains(Path.GetExtension(a).ToLower())))
{
ValidationErrors.Push($"{p.To} from {archive.archive.NexusURL} is set to disallow BSA Extraction");
ValidationErrors.Push($"{p.To} from {url} is set to disallow BSA Extraction");
}
}
});
@ -150,34 +153,25 @@ namespace Wabbajack.Validation
var nexus = NexusApi.NexusApiUtils.ConvertGameName(GameRegistry.Games[modlist.GameType].NexusName);
modlist.Archives
.OfType<NexusMod>()
.Where(m => NexusApi.NexusApiUtils.ConvertGameName(m.GameName) != nexus)
.Where(a => a.State is NexusDownloader.State)
.Where(m => NexusApi.NexusApiUtils.ConvertGameName(((NexusDownloader.State)m.State).GameName) != nexus)
.Do(m =>
{
var permissions = FilePermissions(m);
var permissions = FilePermissions((NexusDownloader.State)m.State);
if (!(permissions.CanUseInOtherGames ?? true))
{
ValidationErrors.Push(
$"The modlist is for {nexus} but {m.Name} is for game type {m.GameName} and is not allowed to be converted to other game types");
$"The modlist is for {nexus} but {m.Name} is for game type {((NexusDownloader.State)m.State).GameName} and is not allowed to be converted to other game types");
}
});
modlist.Archives
.OfType<GoogleDriveMod>()
.PMap(m =>
{
if (!ServerWhitelist.GoogleIDs.Contains(m.Id))
ValidationErrors.Push($"{m.Name} uses Google Drive id {m.Id} but that id is not in the file whitelist.");
});
modlist.Archives
.OfType<DirectURLArchive>()
.PMap(m =>
.Where(m => !m.State.IsWhitelisted(ServerWhitelist))
.Do(m =>
{
if (!ServerWhitelist.AllowedPrefixes.Any(prefix => m.URL.StartsWith(prefix)))
ValidationErrors.Push($"{m.Name} will be downloaded from {m.URL} but that URL is not in the server whitelist");
ValidationErrors.Push($"{m.Name} is not a whitelisted download");
});
return ValidationErrors.ToList();
}
}

11
Wabbajack/Wabbajack.csproj
View File

@ -188,6 +188,16 @@
<SubType>Designer</SubType>
</ApplicationDefinition>
<Compile Include="Data.cs" />
<Compile Include="Downloaders\AbstractDownloadState.cs" />
<Compile Include="Downloaders\GoogleDriveDownloader.cs" />
<Compile Include="Downloaders\HTTPDownloader.cs" />
<Compile Include="Downloaders\DownloadDispatcher.cs" />
<Compile Include="Downloaders\DropboxDownloader.cs" />
<Compile Include="Downloaders\IDownloader.cs" />
<Compile Include="Downloaders\ManualDownloader.cs" />
<Compile Include="Downloaders\MegaDownloader.cs" />
<Compile Include="Downloaders\ModDBDownloader.cs" />
<Compile Include="Downloaders\NexusDownloader.cs" />
<Compile Include="LambdaCommand.cs" />
<Compile Include="UI\ModlistPropertiesWindow.xaml.cs">
<DependentUpon>ModlistPropertiesWindow.xaml</DependentUpon>
@ -327,6 +337,7 @@
<EmbeddedResource Include="UI\Icons\github_light.png" />
<EmbeddedResource Include="UI\Icons\patreon_light.png" />
</ItemGroup>
<ItemGroup />
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<Import Project="..\packages\Fody.5.1.1\build\Fody.targets" Condition="Exists('..\packages\Fody.5.1.1\build\Fody.targets')" />
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">