Backport Docs code links (#7740)

* Docs code links (#7342)

* Update docs

- Add note about permission denied error

* Add macro for generating link to github code

* Implement similar feature for source directory links

* Adds helper function for link checking

* Allow for specification of "raw" file links

* Remove debug statement

* Generate list of available invoke tasks

(cherry picked from commit 797a0c10df)

* auto-detect current branch and use that for links

* remove debug logging

* style fix

* spell fix

---------

Co-authored-by: Oliver <oliver.henry.walters@gmail.com>

docs/.gitignore

@@ -13,6 +13,10 @@ site/
 # Generated API schema files
 docs/api/schema/*.yml
 
+# Temporary cache files
+url_cache.txt
+invoke-commands.txt
+
 # Temp files
 releases.json
 versions.json

docs/docs/develop/contributing.md

@@ -96,7 +96,7 @@ The HEAD of the "stable" branch represents the latest stable release code.
 
 ## API versioning
 
-The [API version](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/InvenTree/api_version.py) needs to be bumped every time when the API is changed.
+The [API version]({{ sourcefile("src/backend/InvenTree/InvenTree/api_version.py") }}) needs to be bumped every time when the API is changed.
 
 ## Environment
 

docs/docs/extend/plugins.md

@@ -16,7 +16,7 @@ For further information, read more about [installing plugins](./plugins/install.
 
 ### Plugin Base Class
 
-Custom plugins must inherit from the [InvenTreePlugin class](https://github.com/inventree/InvenTree/blob/2d1776a151721d65d0ae007049d358085b2fcfd5/InvenTree/plugin/plugin.py#L204). Any plugins installed via the methods outlined above will be "discovered" when the InvenTree server launches.
+Custom plugins must inherit from the [InvenTreePlugin class]({{ sourcefile("src/backend/InvenTree/plugin/plugin.py") }}). Any plugins installed via the methods outlined above will be "discovered" when the InvenTree server launches.
 
 !!! warning "Namechange"
     The name of the base class was changed with `0.7.0` from `IntegrationPluginBase` to `InvenTreePlugin`. While the old name is still available till `0.8.0` we strongly suggest upgrading your plugins. Deprecation warnings are raised if the old name is used.
@@ -28,7 +28,7 @@ Please read all release notes and watch out for warnings - we generally provide
 
 #### Plugins
 
-General classes and mechanisms are provided under the `plugin` [namespaces](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/plugin/__init__.py). These include:
+General classes and mechanisms are provided under the `plugin` [namespaces]({{ sourcefile("src/backend/InvenTree/plugin/__init__.py") }}). These include:
 
 ```python
 # Management objects
@@ -44,7 +44,7 @@ MixinNotImplementedError    # Is raised if a mixin was not implemented (core mec
 
 #### Mixins
 
-Mixins are split up internally to keep the source tree clean and enable better testing separation. All public APIs that should be used are exposed under `plugin.mixins`. These include all built-in mixins and notification methods. An up-to-date reference can be found in the source code (current master can be [found here](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/plugin/mixins/__init__.py)).
+Mixins are split up internally to keep the source tree clean and enable better testing separation. All public APIs that should be used are exposed under `plugin.mixins`. These include all built-in mixins and notification methods. An up-to-date reference can be found in the source code [can be found here]({{ sourcefile("src/backend/InvenTree/plugin/mixins/__init__.py") }}).
 
 #### Models and other internal InvenTree APIs
 
@@ -72,7 +72,7 @@ MIN_VERSION = None  # Lowest InvenTree version number that is supported by the p
 MAX_VERSION = None  # Highest InvenTree version number that is supported by the plugin
 ```
 
-Refer to the [sample plugins](https://github.com/inventree/InvenTree/tree/master/src/backend/InvenTree/plugin/samples) for further examples.
+Refer to the [sample plugins]({{ sourcedir("src/backend/InvenTree/plugin/samples") }}) for further examples.
 
 ### Plugin Config
 

docs/docs/extend/plugins/locate.md

@@ -28,4 +28,4 @@ If a locate plugin is installed and activated, the [InvenTree mobile app](../../
 
 ### Implementation
 
-Refer to the [InvenTree source code](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/plugin/samples/locate/locate_sample.py) for a simple implementation example.
+Refer to the [InvenTree source code]({{ sourcefile("src/backend/InvenTree/plugin/samples/locate/locate_sample.py") }}) for a simple implementation example.

docs/docs/extend/plugins/urls.md

@@ -65,7 +65,7 @@ Additionally, add the following imports after the extended line.
 #### Blocks
 The page_base file is split into multiple sections called blocks. This allows you to implement sections of the webpage while getting many items like navbars, sidebars, and general layout provided for you.
 
-The current default page base can be found [here](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/templates/page_base.html). Look through this file to determine overridable blocks. The [stock app](https://github.com/inventree/InvenTree/tree/master/src/backend/InvenTree/stock) offers a great example of implementing these blocks.
+The current default page base can be found [here]({{ sourcefile("src/backend/InvenTree/templates/page_base.html") }}). Look through this file to determine overridable blocks. The [stock app]({{ sourcedir("src/backend/InvenTree/stock") }}) offers a great example of implementing these blocks.
 
 !!! warning "Sidebar Block"
     You may notice that implementing the `sidebar` block doesn't initially work. Be sure to enable the sidebar using JavaScript. This can be achieved by appending the following code, replacing `label` with a label of your choosing,  to the end of your template file.

docs/docs/extend/plugins/validation.md

@@ -9,7 +9,7 @@ The `ValidationMixin` class enables plugins to perform custom validation of obje
 Any of the methods described below can be implemented in a custom plugin to provide functionality as required.
 
 !!! info "More Info"
-    For more information on any of the methods described below, refer to the InvenTree source code. [A working example is available as a starting point](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/plugin/samples/integration/validation_sample.py).
+    For more information on any of the methods described below, refer to the InvenTree source code. [A working example is available as a starting point]({{ sourcefile("src/backend/InvenTree/plugin/samples/integration/validation_sample.py") }}).
 
 !!! info "Multi Plugin Support"
     It is possible to have multiple plugins loaded simultaneously which support validation methods. For example when validating a field, if one plugin returns a null value (`None`) then the *next* plugin (if available) will be queried.

docs/docs/report/helpers.md

@@ -12,7 +12,7 @@ Some common functions are provided for use in custom report and label templates.
 ```
 
 !!! tip "Use the Source, Luke"
-    To see the full range of available helper functions, refer to the source file [report.py](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/report/templatetags/report.py) where these functions are defined!
+    To see the full range of available helper functions, refer to the source file [report.py]({{ sourcefile("src/backend/InvenTree/report/templatetags/report.py") }}) where these functions are defined!
 
 ## Assigning Variables
 

docs/docs/security.md

@@ -8,9 +8,9 @@ To that end, we have implemented a number of security measures over the years, w
 The InvenTree project is managed by a small team of developers, who are responsible for the ongoing development and maintenance of the software. Two geographically distributed users have administrative access to the InvenTree codebase. Merges are only done by one of these two users, the maintainer Oliver.
 InvenTree is open-source, and we welcome contributions from the community. However, all contributions are reviewed and scrutinised before being merged into the codebase.
 
-We provide a written [Security Policy](https://github.com/inventree/InvenTree/blob/master/SECURITY.md) in our main repo to ensure that all security issues are handled in a timely manner.
+We provide a written [Security Policy]({{ sourcefile("SECURITY.md") }}) in our main repo to ensure that all security issues are handled in a timely manner.
 
-If we become aware of a security issue, we will take immediate action to address the issue, and will provide a public disclosure of the issue once it has been resolved. We support assigning CVEs to security issues where appropriate.  Our past security advisories can be found [here](https://github.com/inventree/InvenTree/security/advisories).
+If we become aware of a security issue, we will take immediate action to address the issue, and will provide a public disclosure of the issue once it has been resolved. We support assigning CVEs to security issues where appropriate.  Our [past security advisories can be found here](https://github.com/inventree/InvenTree/security/advisories).
 
 ## Technical measures
 

docs/docs/start/config.md

@@ -22,7 +22,7 @@ The InvenTree server tries to locate the `config.yaml` configuration file on sta
 !!! tip "Config File Location"
     When the InvenTree server boots, it will report the location where it expects to find the configuration file
 
-The configuration file *template* can be found on [GitHub](https://github.com/inventree/InvenTree/blob/0.15.x/src/backend/InvenTree/config_template.yaml)
+The configuration file *template* can be found on [GitHub]({{ sourcefile("src/backend/InvenTree/config_template.yaml") }})
 
 !!! info "Template File"
     The default configuration file (as defined by the template linked above) will be copied to the specified configuration file location on first run, if a configuration file is not found in that location.

docs/docs/start/docker_install.md

@@ -27,13 +27,13 @@ The following guide provides a streamlined production InvenTree installation, wi
 
 ### Required Files
 
-The following files required for this setup are provided with the InvenTree source, located in the `/contrib/container/` directory of the [InvenTree source code](https://github.com/inventree/InvenTree/tree/master/contrib/container/):
+The following files required for this setup are provided with the InvenTree source, located in the `contrib/container/` directory of the [InvenTree source code]({{ sourcedir("/contrib/container/") }}):
 
 | Filename | Description |
 | --- | --- |
-| [docker-compose.yml](https://raw.githubusercontent.com/inventree/InvenTree/master/contrib/container/docker-compose.yml)| The docker compose script |
+| [docker-compose.yml]({{ sourcefile("contrib/container/docker-compose.yml", raw=True) }}) | The docker compose script |
-| [.env](https://raw.githubusercontent.com/inventree/InvenTree/master/contrib/container/.env) | Environment variables |
+| [.env]({{ sourcefile("contrib/container/.env", raw=True) }}) | Environment variables |
-| [Caddyfile](https://raw.githubusercontent.com/inventree/InvenTree/master/contrib/container/Caddyfile) | Caddy configuration file |
+| [Caddyfile]({{ sourcefile("contrib/container/Caddyfile", raw=True) }}) | Caddy configuration file |
 
 Download these files to a directory on your local machine.
 

docs/docs/start/install.md

@@ -23,7 +23,7 @@ Install required system packages (as superuser):
     The following packages are required on a debian system. A different distribution may require a slightly different set of packages
 
 !!! info "Python Version"
-    InvenTree requires a modern Python version check [here](https://github.com/inventree/InvenTree/blob/master/CONTRIBUTING.md#target-version) for the current minimums.
+    InvenTree requires a modern Python version [check here]({{ sourcefile("CONTRIBUTING.md") }}) for the current minimums.
 
 ```
 sudo apt-get update

docs/docs/start/intro.md

@@ -84,6 +84,12 @@ To display a list of the available InvenTree administration actions, run the fol
 invoke --list
 ```
 
+This provides a list of the available invoke commands - also displayed below:
+
+```
+{{ invoke_commands() }}
+```
+
 ### Virtual Environment
 
 Installing the required Python packages inside a virtual environment allows a local install separate to the system-wide Python installation. While not strictly necessary, using a virtual environment is **highly recommended** as it prevents conflicts between the different Python installations.

docs/main.py

@@ -1,11 +1,171 @@
 """Main entry point for the documentation build process."""
 
 import os
+import subprocess
+
+import requests
+import yaml
+
+
+def get_repo_url(raw=False):
+    """Return the repository URL for the current project."""
+    mkdocs_yml = os.path.join(os.path.dirname(__file__), 'mkdocs.yml')
+
+    with open(mkdocs_yml, 'r') as f:
+        mkdocs_config = yaml.safe_load(f)
+        repo_name = mkdocs_config['repo_name']
+
+    if raw:
+        return f'https://raw.githubusercontent.com/{repo_name}'
+    else:
+        return f'https://github.com/{repo_name}'
+
+
+def check_link(url) -> bool:
+    """Check that a provided URL is valid.
+
+    We allow a number attempts and a lengthy timeout,
+    as we do not want false negatives.
+    """
+    CACHE_FILE = os.path.join(os.path.dirname(__file__), 'url_cache.txt')
+
+    # Keep a local cache file of URLs we have already checked
+    if os.path.exists(CACHE_FILE):
+        with open(CACHE_FILE, 'r') as f:
+            cache = f.read().splitlines()
+
+        if url in cache:
+            return True
+
+    attempts = 5
+
+    while attempts > 0:
+        response = requests.head(url, timeout=5000)
+        if response.status_code == 200:
+            # Update the cache file
+            with open(CACHE_FILE, 'a') as f:
+                f.write(f'{url}\n')
+
+            return True
+
+        attempts -= 1
+
+    return False
+
+
+def get_build_enviroment() -> str:
+    """Returns the branch we are currently building on, based on the environment variables of the various CI platforms."""
+    # Check if we are in ReadTheDocs
+    if os.environ.get('READTHEDOCS') == 'True':
+        return os.environ.get('READTHEDOCS_GIT_IDENTIFIER')
+    # We are in GitHub Actions
+    elif os.environ.get('GITHUB_ACTIONS') == 'true':
+        return os.environ.get('GITHUB_REF')
+    else:
+        return 'master'
 
 
 def define_env(env):
     """Define custom environment variables for the documentation build process."""
 
+    @env.macro
+    def sourcedir(dirname, branch=None):
+        """Return a link to a directory within the source code repository.
+
+        Arguments:
+            - dirname: The name of the directory to link to (relative to the top-level directory)
+
+        Returns:
+            - A fully qualified URL to the source code directory on GitHub
+
+        Raises:
+            - FileNotFoundError: If the directory does not exist, or the generated URL is invalid
+        """
+        if branch == None:
+            branch = get_build_enviroment()
+
+        if dirname.startswith('/'):
+            dirname = dirname[1:]
+
+        # This file exists at ./docs/main.py, so any directory we link to must be relative to the top-level directory
+        here = os.path.dirname(__file__)
+        root = os.path.abspath(os.path.join(here, '..'))
+
+        directory = os.path.join(root, dirname)
+        directory = os.path.abspath(directory)
+
+        if not os.path.exists(directory) or not os.path.isdir(directory):
+            raise FileNotFoundError(f'Source directory {dirname} does not exist.')
+
+        repo_url = get_repo_url()
+
+        url = f'{repo_url}/tree/{branch}/{dirname}'
+
+        # Check that the URL exists before returning it
+        if not check_link(url):
+            raise FileNotFoundError(f'URL {url} does not exist.')
+
+        return url
+
+    @env.macro
+    def sourcefile(filename, branch=None, raw=False):
+        """Return a link to a file within the source code repository.
+
+        Arguments:
+            - filename: The name of the file to link to (relative to the top-level directory)
+
+        Returns:
+            - A fully qualified URL to the source code file on GitHub
+
+        Raises:
+            - FileNotFoundError: If the file does not exist, or the generated URL is invalid
+        """
+        if branch == None:
+            branch = get_build_enviroment()
+
+        if filename.startswith('/'):
+            filename = filename[1:]
+
+        # This file exists at ./docs/main.py, so any file we link to must be relative to the top-level directory
+        here = os.path.dirname(__file__)
+        root = os.path.abspath(os.path.join(here, '..'))
+
+        file_path = os.path.join(root, filename)
+
+        if not os.path.exists(file_path):
+            raise FileNotFoundError(f'Source file {filename} does not exist.')
+
+        repo_url = get_repo_url(raw=raw)
+
+        if raw:
+            url = f'{repo_url}/{branch}/{filename}'
+        else:
+            url = f'{repo_url}/blob/{branch}/{filename}'
+
+        # Check that the URL exists before returning it
+        if not check_link(url):
+            raise FileNotFoundError(f'URL {url} does not exist.')
+
+        return url
+
+    @env.macro
+    def invoke_commands():
+        """Provides an output of the available commands."""
+        here = os.path.dirname(__file__)
+        base = os.path.join(here, '..')
+        base = os.path.abspath(base)
+        tasks = os.path.join(base, 'tasks.py')
+        output = os.path.join(here, 'invoke-commands.txt')
+
+        command = f'invoke -f {tasks} --list > {output}'
+
+        assert subprocess.call(command, shell=True) == 0
+
+        with open(output, 'r') as f:
+            content = f.read()
+
+        return content
+
     @env.macro
     def listimages(subdir):
         """Return a listing of all asset files in the provided subdir."""