Oliver 1f6cd9fc54 [WIP] Data importer (#6911) ... * Adds new model for DataImportSession * Add file extension validation Expose to admin interface also * Switch to new 'importer' app * Refactoring to help prevent circular imports * Add serializer registry - Use @register_importer tag for any serializer class * Cleanup migration file - Do not use one-time hard-coded values here * Refactor code into registry.py * Add validation for the uploaded file - Must be importable by tablib * Refactoring * Adds property to retrieve matching serializer class * Update helper functions * Add hook to auto-assign columns on initial creation * Rename field * Enforce initial status value * Add model for individual rows in the data import * Add DataImportRow model * Extract data rows as dict * Update fields - Remove "progress" field (will be calculated) - Added "timestamp" field - Added "complete" field to DataImportRow * Auto-map column names - Provide "sensible" default values * Add API endpoint for DataImportSession * Offload data import operation - For large data files this may take a significant amount of time - Offload it to the background worker process * Refactor data import code * Update models - Add "columns" field to DataImportSession - Add "errors" field to DataImportRow * Move field mapping to a new model type - Simpler validation * Save "valid" status for each data row * Include session defaults when validating row data * Update content_excludes - Ignore importer models in import/export * Remove port from ALLOWED_HOST entries * Skip table events for importer models * Bug fixes * Serializer updates * Add more endpoints - DataImportColumnMappingList - DataImportRowList * further updates: - Add 'get_api_url' method - Handle case where * Expose "available fields" to the DataImportSession serializer Uses the (already available) inventree metadata middleware * Add detail endpoints * Clear existing column mappings * Add endpoint for accepting column mappings * Add API endpoint exposing available importer serializers * Add simple playground area for testing data importer * Adds simple form to start new import session - Needs work, file field does not currently function correctly * data_file is *not* read_only * Add check for file type * Remove debug statements * Refactor column mapping - Generate mapping for each column - Remove "columns" field - Column names are calculated dynamically * Fix uniqueness requirements on mapping table * Admin updates - Prevent deletion of mappings - Prevent addition of mappings * API endpoint updates - Prevent mappings from being deleted - Prevent mappings from being created * Update importer drawer * Add widget for selecting data columns * UI tweaks * Delete import session when closing modal * Allow empty string value * Complete column mapping * Adds ability to remove rows * Adjust drawer specs * Add column 'description' to serializer * Add option to hide labels in API form field * Update column heading * Fix frontend linting errors * Revert drawer position * Return correct type * Fix shadowing * Fix f-string * simplify frontend code * Move importer app * Update API version * Reintroduce export formats * Add new models to RuleSet * typescript cleanup * Typescript cleanup * Improvement for Switch / boolean field * Display original row data on popover * Only display mapped columns * Add DataExportMixin class - Replaces existing APIDownloadMixin - Uses DRF serializers for exporting - *much* more efficient * Create new file: importer.mixins.py * Add new mixin to existing views which support data export * Better error handling * Cleanup: - Remove references to APIDownloadMixin - Remove download_queryset method - All now handled by API-based export functionality * Replace table with InvenTreeTable - Paginate imported rows - Data can be searched, ordered, * Make 'pathstring' fields read-only * Expose list of valid importer types to the API * Exclude read-only fields * Cleanup * Updates for session model - Column is now editable on mapping object - Field is no longer editable - Improve admin integration * Adds new custom hook for controlling data import session * Refactor column mapping widget * Refactor ImportDataSelector * Working on ImportDataSelector component * Adds method for editing fields in import table - Cell edit mode - Row edit mode - Form submission still needs work! * Adds background task for removing old import sessions * Fix api_version.py * Update src/frontend/src/components/importer/ImportDataSelector.tsx Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com> * Update model verbose names * Rename mixin class * Add serializer mixin classes - Will allow for fine-tuning of the import/export proces * @register_importer requires specific mixin * Check subclass for export * Fix typos * Refactor export serializer - Keep operations local to the class * Add shim class to process an individual row before exporting it * Add mixin to existing serializers * Add export functionality for company serializers * Adds placeholder for custom admin class * Update mantine styling * spacing -> gap * Add functionality to pre-process form data before upload * Remove old references to download_queryset * Improvements for data import drawer: - Pin title at top of drawer * Further improvements * Fix column selection input * Formatting improvements * Use a <Stepper> component for better progress display * Cleanup text * Add export-only fields to BuildItem queryset * Expand "export" fields for BuildItem dataset * Skip backup and static steps in CI * Remove hard-coded paths * Fix for "accept_mapping" method * Present required fields first on import session * Add "get_importable_fields" method * Add method for commiting imported row to database * Cleanup * Save "complete" state after row import * Allow prevention of column caching * Remove debug statement * Add basic admin table for import sessions * Fix for table filter functions - New mantine version requires string values * Add filters for import session table * Remove debug message * fix for <FilterItem /> * Create new import session from admin page * Cleanup playground * Re-open an existing import session * Memoize cell value * Update <ImportDataSelector> * Enable download of build line data * Add extra detail fields * Register data importers for the stock app * Enable download of stock item tracking data * Register importerrs for "company" app * Register importers for the "order" app * Add extra fields to purchase order line item serializer * Update verbose names for order models * Cleanup import data table rendering * Pass session information through to cell renderer * add separate 'field_overrides' field * Expose 'field_overrides' to API * Refactor import field selection * Use override data if provided * Fix data extraction - Ignore columns which are not mapped * Fix fields.pop - Provide 'None' argument * Update import data rendering * Handle missing / empty column names when importing data * Bug fixin' * Update hook * Adds button to upload data straight to table * Cache "available_fields" - Reduces API access time by 85% * Fix calculation of completed_row_count * Import individual rows from import session * Allow import of multiple simultaneous records * Improve extraction of metadata - Especially for related fields - Request object no longer required * Implement suspended rendering of model instances * Cleanup * Implement more columns for StockTable * Allow stock filtering by packaging field * Fix "stock_value" column * Improve metadata extraction - Handle read_only_fields in Meta - Handle write_only_fields in Meta * Increase maximum number of importable rows * Force data import to run on background worker * Add export-only fields to StockItemSerializer class * Data conversion when performing initial import * Various tweaks * Fix order of operations for data import * Rename component * Allow import/export of more model types * Fix verbose name * Import rows as a bulk db operation * Enable download for PartCategoryTemplateTable * Update stock item export * Updates for unit tests * Remove xls format for now - Causes some bug in tablib - Surely xlsx is OK? * More unit test updates * Future proof migration * Updates * unit tests * Unit test fix * Remove 'field_overrides' - field_defaults will suffice * Remove 'xls' as download option from frontend * Add simple unit test for data import * PUI tweaks --------- Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com>		2024-07-06 18:29:52 +10:00
..
action.yaml	[WIP] Data importer (#6911)	2024-07-06 18:29:52 +10:00