* Adds new model for DataImportSession
* Add file extension validation
Expose to admin interface also
* Switch to new 'importer' app
* Refactoring to help prevent circular imports
* Add serializer registry
- Use @register_importer tag for any serializer class
* Cleanup migration file
- Do not use one-time hard-coded values here
* Refactor code into registry.py
* Add validation for the uploaded file
- Must be importable by tablib
* Refactoring
* Adds property to retrieve matching serializer class
* Update helper functions
* Add hook to auto-assign columns on initial creation
* Rename field
* Enforce initial status value
* Add model for individual rows in the data import
* Add DataImportRow model
* Extract data rows as dict
* Update fields
- Remove "progress" field (will be calculated)
- Added "timestamp" field
- Added "complete" field to DataImportRow
* Auto-map column names
- Provide "sensible" default values
* Add API endpoint for DataImportSession
* Offload data import operation
- For large data files this may take a significant amount of time
- Offload it to the background worker process
* Refactor data import code
* Update models
- Add "columns" field to DataImportSession
- Add "errors" field to DataImportRow
* Move field mapping to a new model type
- Simpler validation
* Save "valid" status for each data row
* Include session defaults when validating row data
* Update content_excludes
- Ignore importer models in import/export
* Remove port from ALLOWED_HOST entries
* Skip table events for importer models
* Bug fixes
* Serializer updates
* Add more endpoints
- DataImportColumnMappingList
- DataImportRowList
* further updates:
- Add 'get_api_url' method
- Handle case where
* Expose "available fields" to the DataImportSession serializer
Uses the (already available) inventree metadata middleware
* Add detail endpoints
* Clear existing column mappings
* Add endpoint for accepting column mappings
* Add API endpoint exposing available importer serializers
* Add simple playground area for testing data importer
* Adds simple form to start new import session
- Needs work, file field does not currently function correctly
* data_file is *not* read_only
* Add check for file type
* Remove debug statements
* Refactor column mapping
- Generate mapping for each column
- Remove "columns" field
- Column names are calculated dynamically
* Fix uniqueness requirements on mapping table
* Admin updates
- Prevent deletion of mappings
- Prevent addition of mappings
* API endpoint updates
- Prevent mappings from being deleted
- Prevent mappings from being created
* Update importer drawer
* Add widget for selecting data columns
* UI tweaks
* Delete import session when closing modal
* Allow empty string value
* Complete column mapping
* Adds ability to remove rows
* Adjust drawer specs
* Add column 'description' to serializer
* Add option to hide labels in API form field
* Update column heading
* Fix frontend linting errors
* Revert drawer position
* Return correct type
* Fix shadowing
* Fix f-string
* simplify frontend code
* Move importer app
* Update API version
* Reintroduce export formats
* Add new models to RuleSet
* typescript cleanup
* Typescript cleanup
* Improvement for Switch / boolean field
* Display original row data on popover
* Only display mapped columns
* Add DataExportMixin class
- Replaces existing APIDownloadMixin
- Uses DRF serializers for exporting
- *much* more efficient
* Create new file: importer.mixins.py
* Add new mixin to existing views which support data export
* Better error handling
* Cleanup:
- Remove references to APIDownloadMixin
- Remove download_queryset method
- All now handled by API-based export functionality
* Replace table with InvenTreeTable
- Paginate imported rows
- Data can be searched, ordered,
* Make 'pathstring' fields read-only
* Expose list of valid importer types to the API
* Exclude read-only fields
* Cleanup
* Updates for session model
- Column is now editable on mapping object
- Field is no longer editable
- Improve admin integration
* Adds new custom hook for controlling data import session
* Refactor column mapping widget
* Refactor ImportDataSelector
* Working on ImportDataSelector component
* Adds method for editing fields in import table
- Cell edit mode
- Row edit mode
- Form submission still needs work!
* Adds background task for removing old import sessions
* Fix api_version.py
* Update src/frontend/src/components/importer/ImportDataSelector.tsx
Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com>
* Update model verbose names
* Rename mixin class
* Add serializer mixin classes
- Will allow for fine-tuning of the import/export proces
* @register_importer requires specific mixin
* Check subclass for export
* Fix typos
* Refactor export serializer
- Keep operations local to the class
* Add shim class to process an individual row before exporting it
* Add mixin to existing serializers
* Add export functionality for company serializers
* Adds placeholder for custom admin class
* Update mantine styling
* spacing -> gap
* Add functionality to pre-process form data before upload
* Remove old references to download_queryset
* Improvements for data import drawer:
- Pin title at top of drawer
* Further improvements
* Fix column selection input
* Formatting improvements
* Use a <Stepper> component for better progress display
* Cleanup text
* Add export-only fields to BuildItem queryset
* Expand "export" fields for BuildItem dataset
* Skip backup and static steps in CI
* Remove hard-coded paths
* Fix for "accept_mapping" method
* Present required fields first on import session
* Add "get_importable_fields" method
* Add method for commiting imported row to database
* Cleanup
* Save "complete" state after row import
* Allow prevention of column caching
* Remove debug statement
* Add basic admin table for import sessions
* Fix for table filter functions
- New mantine version requires string values
* Add filters for import session table
* Remove debug message
* fix for <FilterItem />
* Create new import session from admin page
* Cleanup playground
* Re-open an existing import session
* Memoize cell value
* Update <ImportDataSelector>
* Enable download of build line data
* Add extra detail fields
* Register data importers for the stock app
* Enable download of stock item tracking data
* Register importerrs for "company" app
* Register importers for the "order" app
* Add extra fields to purchase order line item serializer
* Update verbose names for order models
* Cleanup import data table rendering
* Pass session information through to cell renderer
* add separate 'field_overrides' field
* Expose 'field_overrides' to API
* Refactor import field selection
* Use override data if provided
* Fix data extraction
- Ignore columns which are not mapped
* Fix fields.pop
- Provide 'None' argument
* Update import data rendering
* Handle missing / empty column names when importing data
* Bug fixin'
* Update hook
* Adds button to upload data straight to table
* Cache "available_fields"
- Reduces API access time by 85%
* Fix calculation of completed_row_count
* Import individual rows from import session
* Allow import of multiple simultaneous records
* Improve extraction of metadata
- Especially for related fields
- Request object no longer required
* Implement suspended rendering of model instances
* Cleanup
* Implement more columns for StockTable
* Allow stock filtering by packaging field
* Fix "stock_value" column
* Improve metadata extraction
- Handle read_only_fields in Meta
- Handle write_only_fields in Meta
* Increase maximum number of importable rows
* Force data import to run on background worker
* Add export-only fields to StockItemSerializer class
* Data conversion when performing initial import
* Various tweaks
* Fix order of operations for data import
* Rename component
* Allow import/export of more model types
* Fix verbose name
* Import rows as a bulk db operation
* Enable download for PartCategoryTemplateTable
* Update stock item export
* Updates for unit tests
* Remove xls format for now
- Causes some bug in tablib
- Surely xlsx is OK?
* More unit test updates
* Future proof migration
* Updates
* unit tests
* Unit test fix
* Remove 'field_overrides'
- field_defaults will suffice
* Remove 'xls' as download option from frontend
* Add simple unit test for data import
* PUI tweaks
---------
Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com>
* bump pre-commit tools
* move uv config to pyproject
* style fix
* bump pre-commit deps again
* add config for code spell
* spelling fixes
* bump regex
* bump pre-commit
* bump versions again
* roll back regex version
* reverse uv bump
* compile for 3.12
* use 3.12 for ci
* remove unused env vars
* only cache main version
* set caching dep path
* remove caching distinction
* pip installl regex
* switch back to 3.9 default
* remove install test
* reset versions
* remove tomli
* Update playwright.config.ts
---------
Co-authored-by: Oliver <oliver.henry.walters@gmail.com>
* Add basic model for handling generic attachments
* Refactor migration
* Data migration to convert old files across
* Admin updates
* Increase comment field max_length
* Adjust field name
* Remove legacy serializer classes / endpoints
* Expose new model to API
* Admin site list filters
* Remove legacy attachment models
- Add new mixin class to designate which models can have attachments
* Update data migration
- Ensure other apps are at the correct migration state beforehand
* Add migrations to remove legacy attachment tables
* Fix for "rename_attachment" callback
* Refactor model_type field
- ContentType does not allow easy API serialization
* Set allowed options for admin
* Update model verbose names
* Fix logic for file upload
* Add choices for serializer
* Add API filtering
* Fix for API filter
* Fix for attachment tables in PUI
- Still not solved permission issues
* Bump API version
* Record user when uploading attachment via API
* Refactor <AttachmentTable /> for PUI
* Display 'file_size' in PUI attachment table
* Fix company migrations
* Include permission informtion in roles API endpoint
* Read user permissions in PUI
* Simplify permission checks for <AttachmentTable />
* Automatically clean up old content types
* Cleanup PUI
* Fix typo in data migration
* Add reverse data migration
* Update unit tests
* Use InMemoryStorage for media files in test mode
* Data migration unit test
* Fix "model_type" field
- It is a required field after all
* Add permission check for serializer
* Fix permission check for CUI
* Fix PUI import
* Test python lib against specific branch
- Will be reverted once code is merged
* Revert STORAGES setting
- Might be worth looking into again
* Fix part unit test
* Fix unit test for sales order
* Use 'get_global_setting'
* Use 'get_global_setting'
* Update setting getter
* Unit tests
* Tweaks
* Revert change to settings.py
* More updates for get_global_setting
* Relax API query count requirement
* remove illegal chars and add unit tests
* Fix unit tests
* Fix frontend unit tests
* settings management updates
* Prevent db write under more conditions
* Simplify settings code
* Pop values before creating filters
* Prevent settings write under certain conditions
* Add debug msg
* Clear db on record import
* Refactor permissions checks
- Allows extension / customization of permission checks at a later date
* Unit test updates
* Prevent delete of attachment without correct permissions
* Adjust odcker.yaml
* Cleanup data migrations
* Tweak migration tests for build app
* Update data migration
- Handle case with missing data
* Prevent debug shell in TESTING mode
* Update migration dependencies
- Ensure all apps are "up to date" before removing legacy tables
* add file size test
* Update migration tests
* Revert some settings caching changes
* Fix incorrect logic in migration
* Update unit tests
* Prevent create on CURRENCY_CODES
- Seems to play havoc with bootup sequence
* Fix unit test
* Some refactoring
- Use get_global_setting
* Fix typo
* Revert change
* Add "tags" and "metadata"
* Include "tags" field in API serializer
* add "metadata" endpoint for attachments
* - move reqs file to contrib
- detect previously used python version
- safe extra requirements to INSTALLER_EXTRA
* add missing fi
* move site setting
* Fix sales order shipment / completed buttons
- Template logic needed tweaks
* Adjust logic for completing a sales order
* Tweak UV version
- Trying to get CI to run properly
* Pin version
* Ignore uv
* Cleanup
* Fix another --uv command
* Copy requirements file
* Test more files when building docker image
* Refactor install task
* Raise exception
* Run install task
* Fix typos
- The tests work!
* Remove use of deprecated imp.load_source
The entire `imp` module has been removed from Python 3.12. This patch applies the recommended replacement using `importlib`.
* Fix usage of from importlib.metadata.entry_points to work with newer importlib & Python 3.12
* Update registry.py
Fix order of imports
* Use importlib.util.module_from_spec() instead of deprecated load_module()
* auto-fixed import style (isort)
* enable py 12
* run coverage for lower and upper bound
* fix style error
* make import conditional
* fix?
* fix env
* style fix
* only use new loader on 3.12
* fix order
* fix module loading
* reimplement assertDictContainsSubset
* remove old testing alias
---------
Co-authored-by: Thea Flowers <thea@winterbloom.com>
Co-authored-by: Oliver <oliver.henry.walters@gmail.com>
* use global pin for requests
* unify on yaml for workflo files
* format workflow files
* pin action versions
* fix pinned version
* use system venv
* switch args
* remove uv for now and add setting for pyyaml
* use requirements file
* also switch on docker flow
* generate hashes
* added hashes to reqs
* add hashes for CI too
* add hash checking
* require hashes everywhere possible
* require hashes where possible in docker
* use global pin for requests
* unify on yaml for workflo files
* format workflow files
* pin action versions
* fix pinned version
* use system venv
* switch args
* remove uv for now and add setting for pyyaml
* fix qc args
* bix doc paths
* add coverage to PUI
* fix testing command
* always do all steps
* fix test assumptions
* add test for serving and logging into pui from django / cui tech stack
* fix up coverage
* fix intentation
* remove paralell flags
* remove coverage finish step
* remove flag
* change names
* add full clickdown test
* more tests
* more tests
* sprinkle in some timeouts
* try using admin
* disable page 1 tests for now
* remove additional tests for now
* only build sourcemaps if coverage is enabled
* fix sourcemap assumption
* add more tests
* fix test
* Update package.json
* move tests to seperate file
* rename test files
* move defaults to seperate file
* centralise user creds
* lower migration level
* re-add page 1
* rename and split tests
* split out problematic test
* Add thumbnail building step
* remove unneded steps
* fix filename
* add more logging
* add sourounding table
* add missing tabs
* wait instead of click
* fix test
* split tests out more
* disable test for stock
* tes part sub-pages
* split tests more
* add company tests
* more sales testing
* split parts testing
* Add test for settings + admin center
* append more apps
* set source
* use labels to force full CI suit
* always check for label
* use newer syntax
* remove unneeded carryforward flag
* improve action names
* remove unused upload steps
* enable discovery of locate test
* remove wrong module
* delete all ( :-) )lines for upload
* remove init for now
* add carryfoward back in again
* disable flaky test
* always run finsh step
* deactivate machine test for now
* specify refs explicitly
* add docker exception
* use xml for coverage info
* dummy fix
* remove dummy fix
* dummy change
* always export
* change path to ensure basepath is submitted
* add more paths
* reverst isort change
* remove unused html export
* remove paralell
* add plain InvenTree too
* Update pyproject.toml
* reset paths
* Update qc_checks.yaml
* Update qc_checks.yaml
* fix coverage runner setting
* fix coverage tool path
* use move, not copy
* ignore tmp
* reset isort settings
* use relative files for reporting
* Revert "use relative files for reporting"
This reverts commit 1f662bfb97.
* only run after at least 1 succeeds
* add TODO
* remove coverage subdir run
* also force migrations
* add coverage to migrations
* fix conditional for migration force check
* always upload coverage
* disable flaky test
* fix tests
* wait for migrations
* re-add function arg?
* adjust coverage targets
* add no-cov for fixes of flaky tests
* remove dummy
* revert to "old" upload method"
