merge with main

docs/features/LOGGING.md

@@ -0,0 +1,171 @@
+---
+title: Controlling Logging
+---
+
+# :material-image-off: Controlling Logging
+
+## Controlling How InvokeAI Logs Status Messages
+
+InvokeAI logs status messages using a configurable logging system. You
+can log to the terminal window, to a designated file on the local
+machine, to the syslog facility on a Linux or Mac, or to a properly
+configured web server. You can configure several logs at the same
+time, and control the level of message logged and the logging format
+(to a limited extent).
+
+Three command-line options control logging:
+
+### `--log_handlers <handler1> <handler2> ...`
+
+This option activates one or more log handlers. Options are "console",
+"file", "syslog" and "http". To specify more than one, separate them
+by spaces:
+
+```bash
+invokeai-web --log_handlers console syslog=/dev/log file=C:\Users\fred\invokeai.log
+```
+
+The format of these options is described below.
+
+### `--log_format {plain|color|legacy|syslog}`
+
+This controls the format of log messages written to the console. Only
+the "console" log handler is currently affected by this setting.
+
+* "plain" provides formatted messages like this:
+
+```bash
+
+[2023-05-24 23:18:2[2023-05-24 23:18:50,352]::[InvokeAI]::DEBUG --> this is a debug message
+[2023-05-24 23:18:50,352]::[InvokeAI]::INFO --> this is an informational messages
+[2023-05-24 23:18:50,352]::[InvokeAI]::WARNING --> this is a warning
+[2023-05-24 23:18:50,352]::[InvokeAI]::ERROR --> this is an error
+[2023-05-24 23:18:50,352]::[InvokeAI]::CRITICAL --> this is a critical error
+```
+
+* "color" produces similar output, but the text will be color coded to
+indicate the severity of the message.
+
+* "legacy" produces output similar to InvokeAI versions 2.3 and earlier:
+
+```bash
+### this is a critical error
+*** this is an error
+** this is a warning
+>> this is an informational messages
+   | this is a debug message
+```
+
+* "syslog" produces messages suitable for syslog entries:
+
+```bash
+InvokeAI [2691178] <CRITICAL> this is a critical error
+InvokeAI [2691178] <ERROR> this is an error
+InvokeAI [2691178] <WARNING> this is a warning
+InvokeAI [2691178] <INFO> this is an informational messages
+InvokeAI [2691178] <DEBUG> this is a debug message
+```
+
+(note that the date, time and hostname will be added by the syslog
+system)
+
+### `--log_level {debug|info|warning|error|critical}`
+
+Providing this command-line option will cause only messages at the
+specified level or above to be emitted.
+
+## Console logging
+
+When "console" is provided to `--log_handlers`, messages will be
+written to the command line window in which InvokeAI was launched. By
+default, the color formatter will be used unless overridden by
+`--log_format`.
+
+## File logging
+
+When "file" is provided to `--log_handlers`, entries will be written
+to the file indicated in the path argument. By default, the "plain"
+format will be used:
+
+```bash
+invokeai-web --log_handlers file=/var/log/invokeai.log
+```
+
+## Syslog logging
+
+When "syslog" is requested, entries will be sent to the syslog
+system. There are a variety of ways to control where the log message
+is sent:
+
+* Send to the local machine using the `/dev/log` socket:
+
+```
+invokeai-web --log_handlers syslog=/dev/log
+```
+
+* Send to the local machine using a UDP message:
+
+```
+invokeai-web --log_handlers syslog=localhost
+```
+
+* Send to the local machine using a UDP message on a nonstandard
+  port:
+
+```
+invokeai-web --log_handlers syslog=localhost:512
+```
+
+* Send to a remote machine named "loghost" on the local LAN using
+  facility LOG_USER and UDP packets:
+
+```
+invokeai-web --log_handlers syslog=loghost,facility=LOG_USER,socktype=SOCK_DGRAM
+```
+
+This can be abbreviated `syslog=loghost`, as LOG_USER and SOCK_DGRAM
+are defaults.
+
+* Send to a remote machine named "loghost" using the facility LOCAL0
+  and using a TCP socket:
+
+```
+invokeai-web --log_handlers syslog=loghost,facility=LOG_LOCAL0,socktype=SOCK_STREAM
+```
+
+If no arguments are specified (just a bare "syslog"), then the logging
+system will look for a UNIX socket named `/dev/log`, and if not found
+try to send a UDP message to `localhost`. The Macintosh OS used to
+support logging to a socket named `/var/run/syslog`, but this feature
+has since been disabled.
+
+## Web logging
+
+If you have access to a web server that is configured to log messages
+when a particular URL is requested, you can log using the "http"
+method:
+
+```
+invokeai-web --log_handlers http=http://my.server/path/to/logger,method=POST
+```
+
+The optional [,method=] part can be used to specify whether the URL
+accepts GET (default) or POST messages.
+
+Currently password authentication and SSL are not supported.
+
+## Using the configuration file
+
+You can set and forget logging options by adding a "Logging" section
+to `invokeai.yaml`:
+
+```
+InvokeAI:
+  [... other settings...]
+  Logging:
+    log_handlers:
+       - console
+       - syslog=/dev/log
+    log_level: info
+    log_format: color
+```

docs/features/index.md

@@ -57,6 +57,9 @@ Personalize models by adding your own style or subjects.
 ## * [The NSFW Checker](NSFW.md)
 Prevent InvokeAI from displaying unwanted racy images.
 
+## * [Controlling Logging](LOGGING.md)
+Control how InvokeAI logs status messages.
+
 ## * [Miscellaneous](OTHER.md)
 Run InvokeAI on Google Colab, generate images with repeating patterns,
 batch process a file of prompts, increase the "creativity" of image

invokeai/app/services/config.py

@@ -143,14 +143,13 @@ two configs are kept in separate sections of the config file:
 '''
 import argparse
 import pydoc
-import typing
 import os
 import sys
 from argparse import ArgumentParser
 from omegaconf import OmegaConf, DictConfig
 from pathlib import Path
 from pydantic import BaseSettings, Field, parse_obj_as
-from typing import Any, ClassVar, Dict, List, Literal, Type, Union, get_origin, get_type_hints, get_args
+from typing import ClassVar, Dict, List, Literal, Type, Union, get_origin, get_type_hints, get_args
 
 INIT_FILE = Path('invokeai.yaml')
 LEGACY_INIT_FILE = Path('invokeai.init')
@@ -168,7 +167,7 @@ class InvokeAISettings(BaseSettings):
 
     def parse_args(self, argv: list=sys.argv[1:]):
         parser = self.get_parser()
-        opt, _ = parser.parse_known_args(argv)
+        opt = parser.parse_args(argv)
         for name in self.__fields__:
             if name not in self._excluded():
                 setattr(self, name, getattr(opt,name))
@@ -368,6 +367,11 @@ setting environment variables INVOKEAI_<setting>.
 
     model               : str = Field(default='stable-diffusion-1.5', description='Initial model name', category='Models')
     embeddings          : bool = Field(default=True, description='Load contents of embeddings directory', category='Models')
+
+    log_handlers        : List[str] = Field(default=["console"], description='Log handler. Valid options are "console", "file=<path>", "syslog=path|address:host:port", "http=<url>"', category="Logging")
+    # note - would be better to read the log_format values from logging.py, but this creates circular dependencies issues
+    log_format          : Literal[tuple(['plain','color','syslog','legacy'])] = Field(default="color", description='Log format. Use "plain" for text-only, "color" for colorized output, "legacy" for 2.3-style logging and "syslog" for syslog-style', category="Logging")
+    log_level           : Literal[tuple(["debug","info","warning","error","critical"])] = Field(default="debug", description="Emit logging messages at this level or  higher", category="Logging")
     #fmt: on
 
     def __init__(self, conf: DictConfig = None, argv: List[str]=None, **kwargs):

invokeai/backend/install/invokeai_configure.py

@@ -34,9 +34,12 @@ from transformers import (
     CLIPTextModel,
     CLIPTokenizer,
 )
-
 import invokeai.configs as configs
 
+from invokeai.app.services.config import (
+    get_invokeai_config,
+    InvokeAIAppConfig,
+)
 from invokeai.frontend.install.model_install import addModelsForm, process_and_execute
 from invokeai.frontend.install.widgets import (
     CenteredButtonPress,
@@ -51,10 +54,6 @@ from invokeai.backend.install.model_install_backend import (
     recommended_datasets,
     UserSelections,
 )
-from invokeai.app.services.config import (
-    get_invokeai_config,
-    InvokeAIAppConfig,
-)
 
 warnings.filterwarnings("ignore")
 
@@ -62,7 +61,8 @@ transformers.logging.set_verbosity_error()
 
 
 # --------------------------globals-----------------------
-config = get_invokeai_config()
+
+config = get_invokeai_config(argv=[])
 
 Model_dir = "models"
 Weights_dir = "ldm/stable-diffusion-v1/"
@@ -698,7 +698,7 @@ def write_opts(opts: Namespace, init_file: Path):
     """
 
     # this will load current settings
-    config = InvokeAIAppConfig()
+    config = InvokeAIAppConfig(argv=[])
     for key,value in opts.__dict__.items():
         if hasattr(config,key):
             setattr(config,key,value)
@@ -819,7 +819,7 @@ def main():
         if old_init_file.exists() and not new_init_file.exists():
             print('** Migrating invokeai.init to invokeai.yaml')
             migrate_init_file(old_init_file)
-            config = get_invokeai_config()  # reread defaults
+            config = get_invokeai_config(argv=[])  # reread defaults
 
 
         if not config.model_conf_path.exists():

invokeai/backend/install/model_install_backend.py

@@ -27,7 +27,7 @@ from ..util.logging import InvokeAILogger
 warnings.filterwarnings("ignore")
 
 # --------------------------globals-----------------------
-config = get_invokeai_config()
+config = get_invokeai_config(argv=[])
 Model_dir = "models"
 Weights_dir = "ldm/stable-diffusion-v1/"
 

invokeai/backend/util/__init__.py

@@ -17,3 +17,5 @@ from .util import (
     instantiate_from_config,
     url_attachment_name,
 )
+
+

invokeai/backend/util/logging.py

@@ -30,8 +30,20 @@ import invokeai.backend.util.logging as IAILogger
 IAILogger.debug('this is a debugging message')
 """
 
-import logging
+import logging.handlers
-import sys
+import socket
+import urllib.parse
+
+from abc import abstractmethod
+from pathlib import Path
+
+from invokeai.app.services.config import InvokeAIAppConfig, get_invokeai_config
+
+try:
+    import syslog
+    SYSLOG_AVAILABLE = True
+except:
+    SYSLOG_AVAILABLE = False
 
 # module level functions
 def debug(msg, *args, **kwargs):
@@ -62,11 +74,77 @@ def getLogger(name: str = None) -> logging.Logger:
     return InvokeAILogger.getLogger(name)
 
 
-class InvokeAILogFormatter(logging.Formatter):
+_FACILITY_MAP = dict(
+    LOG_KERN = syslog.LOG_KERN,
+    LOG_USER = syslog.LOG_USER,
+    LOG_MAIL = syslog.LOG_MAIL,
+    LOG_DAEMON = syslog.LOG_DAEMON,
+    LOG_AUTH = syslog.LOG_AUTH,
+    LOG_LPR = syslog.LOG_LPR,
+    LOG_NEWS = syslog.LOG_NEWS,
+    LOG_UUCP = syslog.LOG_UUCP,
+    LOG_CRON = syslog.LOG_CRON,
+    LOG_SYSLOG = syslog.LOG_SYSLOG,
+    LOG_LOCAL0 = syslog.LOG_LOCAL0,
+    LOG_LOCAL1 = syslog.LOG_LOCAL1,
+    LOG_LOCAL2 = syslog.LOG_LOCAL2,
+    LOG_LOCAL3 = syslog.LOG_LOCAL3,
+    LOG_LOCAL4 = syslog.LOG_LOCAL4,
+    LOG_LOCAL5 = syslog.LOG_LOCAL5,
+    LOG_LOCAL6 = syslog.LOG_LOCAL6,
+    LOG_LOCAL7 = syslog.LOG_LOCAL7,
+) if SYSLOG_AVAILABLE else dict()
+
+_SOCK_MAP = dict(
+    SOCK_STREAM = socket.SOCK_STREAM,
+    SOCK_DGRAM = socket.SOCK_DGRAM,
+)
+
+class InvokeAIFormatter(logging.Formatter):
+    '''
+    Base class for logging formatter
+
+    '''
+    def format(self, record):
+        formatter = logging.Formatter(self.log_fmt(record.levelno))
+        return formatter.format(record)
+
+    @abstractmethod
+    def log_fmt(self, levelno: int)->str:
+        pass
+    
+class InvokeAISyslogFormatter(InvokeAIFormatter):
+    '''
+    Formatting for syslog
+    '''
+    def log_fmt(self, levelno: int)->str:
+        return '%(name)s [%(process)d] <%(levelname)s> %(message)s'
+
+class InvokeAILegacyLogFormatter(InvokeAIFormatter):
+    '''
+    Formatting for the InvokeAI Logger (legacy version)
+    '''
+    FORMATS = {
+        logging.DEBUG: "   | %(message)s",
+        logging.INFO: ">> %(message)s",
+        logging.WARNING: "** %(message)s",
+        logging.ERROR: "*** %(message)s",
+        logging.CRITICAL: "### %(message)s",
+    }
+    def log_fmt(self,levelno:int)->str:
+        return self.FORMATS.get(levelno)
+
+class InvokeAIPlainLogFormatter(InvokeAIFormatter):
+    '''
+    Custom Formatting for the InvokeAI Logger (plain version)
+    '''
+    def log_fmt(self, levelno: int)->str:
+        return "[%(asctime)s]::[%(name)s]::%(levelname)s --> %(message)s"
+
+class InvokeAIColorLogFormatter(InvokeAIFormatter):
     '''
     Custom Formatting for the InvokeAI Logger
     '''
-
     # Color Codes
     grey = "\x1b[38;20m"
     yellow = "\x1b[33;20m"
@@ -88,22 +166,109 @@ class InvokeAILogFormatter(logging.Formatter):
         logging.CRITICAL: bold_red + log_format + reset
     }
 
-    def format(self, record):
+    def log_fmt(self, levelno: int)->str:
-        log_fmt = self.FORMATS.get(record.levelno)
+        return self.FORMATS.get(levelno)
-        formatter = logging.Formatter(log_fmt, datefmt="%d-%m-%Y %H:%M:%S")
+
-        return formatter.format(record)
+LOG_FORMATTERS = {
+    'plain': InvokeAIPlainLogFormatter,
+    'color': InvokeAIColorLogFormatter,
+    'syslog': InvokeAISyslogFormatter,
+    'legacy': InvokeAILegacyLogFormatter,
+}
 
 class InvokeAILogger(object):
     loggers = dict()
 
     @classmethod
     def getLogger(cls, name: str = 'InvokeAI') -> logging.Logger:
+        config = get_invokeai_config()
+        
         if name not in cls.loggers:
             logger = logging.getLogger(name)
-            logger.setLevel(logging.DEBUG)
+            logger.setLevel(config.log_level.upper()) # yes, strings work here
-            ch = logging.StreamHandler()
+            for ch in cls.getLoggers(config):
-            fmt = InvokeAILogFormatter()
+                logger.addHandler(ch)
-            ch.setFormatter(fmt)
-            logger.addHandler(ch)
             cls.loggers[name] = logger
         return cls.loggers[name]
+
+    @classmethod
+    def getLoggers(cls, config: InvokeAIAppConfig) -> list[logging.Handler]:
+        handler_strs = config.log_handlers
+        print(f'handler_strs={handler_strs}')
+        handlers = list()
+        for handler in handler_strs:
+            handler_name,*args = handler.split('=',2)
+            args = args[0] if len(args) > 0 else None
+
+            # console is the only handler that gets a custom formatter
+            if handler_name=='console':
+                formatter = LOG_FORMATTERS[config.log_format]
+                ch = logging.StreamHandler()
+                ch.setFormatter(formatter())
+                handlers.append(ch)
+                
+            elif handler_name=='syslog':
+                ch = cls._parse_syslog_args(args)
+                ch.setFormatter(InvokeAISyslogFormatter())
+                handlers.append(ch)
+                
+            elif handler_name=='file':
+                handlers.append(cls._parse_file_args(args))
+                
+            elif handler_name=='http':
+                handlers.append(cls._parse_http_args(args))
+        return handlers
+
+    @staticmethod
+    def _parse_syslog_args(
+            args: str=None
+    )-> logging.Handler:
+        if not SYSLOG_AVAILABLE:
+            raise ValueError("syslog is not available on this system")
+        if not args:
+            args='/dev/log' if Path('/dev/log').exists() else 'address:localhost:514'
+        syslog_args = dict()
+        try:
+            for a in args.split(','):
+                arg_name,*arg_value = a.split(':',2)
+                if arg_name=='address':
+                    host,*port = arg_value
+                    port = 514 if len(port)==0 else int(port[0])
+                    syslog_args['address'] = (host,port)
+                elif arg_name=='facility':
+                    syslog_args['facility'] = _FACILITY_MAP[arg_value[0]]
+                elif arg_name=='socktype':
+                    syslog_args['socktype'] = _SOCK_MAP[arg_value[0]]
+                else:
+                    syslog_args['address'] = arg_name
+        except:
+            raise ValueError(f"{args} is not a value argument list for syslog logging")
+        return logging.handlers.SysLogHandler(**syslog_args)
+            
+    @staticmethod
+    def _parse_file_args(args: str=None)-> logging.Handler:
+        if not args:
+            raise ValueError("please provide filename for file logging using format 'file=/path/to/logfile.txt'")
+        return logging.FileHandler(args)
+
+    @staticmethod
+    def _parse_http_args(args: str=None)-> logging.Handler:
+        if not args:
+            raise ValueError("please provide destination for http logging using format 'http=url'")
+        arg_list = args.split(',')
+        url = urllib.parse.urlparse(arg_list.pop(0))
+        if url.scheme != 'http':
+            raise ValueError(f"the http logging module can only log to HTTP URLs, but {url.scheme} was specified")
+        host = url.hostname
+        path = url.path
+        port = url.port or 80
+        
+        syslog_args = dict()
+        for a in arg_list:
+            arg_name, *arg_value = a.split(':',2)
+            if arg_name=='method':
+                arg_value = arg_value[0] if len(arg_value)>0 else 'GET'
+                syslog_args[arg_name] = arg_value
+            else:  # TODO: Provide support for SSL context and credentials
+                pass
+        return logging.handlers.HTTPHandler(f'{host}:{port}',path,**syslog_args)

invokeai/frontend/install/model_install.py

@@ -56,7 +56,7 @@ from invokeai.app.services.config import get_invokeai_config
 MIN_COLS = 120
 MIN_LINES = 52
 
-config = get_invokeai_config()
+config = get_invokeai_config(argv=[])
 
 class addModelsForm(npyscreen.FormMultiPage):
     # for responsive resizing - disabled

tests/test_config.py

@@ -1,13 +1,17 @@
 import os
 import pytest
+import sys
 
 from omegaconf import OmegaConf
 from pathlib import Path
 
 os.environ['INVOKEAI_ROOT']='/tmp'
+sys.argv = []  # to prevent config from trying to parse pytest arguments
+
 from invokeai.app.services.config import InvokeAIAppConfig, InvokeAISettings
 from invokeai.app.invocations.generate import TextToImageInvocation
 
+
 init1 = OmegaConf.create(
 '''
 InvokeAI: