Spector Connector - Customer Update
Overview
This document describes the customer update process for the Spector connector, specifically handled by the CustomerUpdateHandler. It details how customer data, including core customer fields, addresses, contact persons, discounts, and dashboards, is retrieved and synchronized with the App4Sales internal database. The process involves initial data validation, GUID resolution (including reactivation logic), price list deduplication, language and country code mapping, handling of customer notes and dashboards, and the processing of extra data fields. Updates are performed in batches to optimize performance.
Data Source Configuration
The primary input for the customer update process is a collection of Customer objects. These objects typically originate from the external ERP system via the Spector connector. Additionally, extra customer-specific data can be supplied through CSV files, accessible via _databaseManager.ExtraCustomerData, which contains CsvCustomer objects. This extra data is identified and merged with the main customer records based on CustomerCode. Language and country mappings are retrieved from the App4Sales database via MappedLanguagesContext and MappedCountriesContext respectively.
Data Mapping Table (Customers)
App4Sales Field | Source Field (API/Excel/DB) | Logic/Notes |
CustomerCode | CustomerCode | Mandatory. Used as a primary identifier. Customers with empty or null codes are skipped. |
CustomerName | CustomerName | Mandatory. Customers with empty or whitespace-only names are skipped. |
CustomerGuid | CustomerGuid / _customerCodeLinks / Portal API / New GUID | If empty: Tries to reuse GUID from previously inactive customers (code starts with '~') in Updater mode. If not found, retrieves from |
Created | Created | If |
Sysmodified | Sysmodified | Truncated to the second to avoid SQL issues. |
PasswordWebshop | PasswordWebshop | Set to |
CustomerDashboard | CustomerDashboard | If empty string or whitespace, set to |
LanguageCode | LanguageCode | Mapped to a standardized App4Sales language code using |
ActionPriceList | ActionPriceList / _administrationSession.ConnectorBaseSettings.DefaultActionPriceListCode | If |
UsesPriceField | UsesPriceField | If Price Deduplication is enabled, IDs are migrated to deduplicated versions. If an old ID is not found, it defaults to price list ID 1. |
PaymentConditionCode | PaymentConditionCode | Empty strings are converted to |
Whitespace is trimmed. | ||
DynamicFreeFields | DynamicFreeFields | If contains non-empty XML, it's deserialized, converted to a standardized |
InternalCode | InternalCode | If not in Updater mode and incoming |
ExtraData | ExtraData | If not in Updater mode and incoming |
VatCode | VatCode | Set to |
VatLiable | VatLiable | Set to |
FreeFields | FreeFieldList / extraDataForCustomer.UnknownElements (starting with "FreeField_") | If |
ItemFilter | ItemFilter / extraDataForCustomer.UnknownElements (starting with "ItemFilter_") | If |
(Other Customer Properties) | CsvCustomer (matching property name) | Properties in the |
Data Mapping Table (Addresses)
App4Sales Field | Source Field (API/Excel/DB) | Logic/Notes |
Unicode unit separator (0x001F) characters are removed. If address is main and its email is empty, it attempts to inherit from customer's email, other customer addresses, or contact persons. | ||
Iso2 | Iso2 / Country | If provided and a mapping exists in |
CustomerGuid | customer.CustomerGuid | Inherited from the parent customer. |
AddressId | AddressId | If empty, a default ID is set. |
Country | Country / Iso2 | If empty, it's set to the |
AddressType | AddressType | If empty, defaults to "Visit". |
AddressLine (Street, HouseNumber, Addition) | AddressLine1 | If |
AddressLine1 | AddressLine (Street, HouseNumber, Addition) | If |
RemoveInvalidXmlChars | Invalid XML characters are removed from the address object. |
Data Mapping Table (Contact Persons)
App4Sales Field | Source Field (API/Excel/DB) | Logic/Notes |
CustomerGuid | customer.CustomerGuid | Inherited from the parent customer. |
PasswordWebshop | PasswordWebshop | Set to |
Whitespace is trimmed. | ||
ContactId | ContactId | If in Updater mode and empty, a |
DynamicFreeFields | DynamicFreeFields | If contains non-empty XML, it's deserialized, converted to a standardized |
FirstName | FirstName / FullName | If |
LastName | LastName / FullName | If |
MiddleName | MiddleName / FullName | If |
FullName | FullName / FirstName, MiddleName, LastName | If |
Initials | FirstName | Set to the first character of |
IsMainContactPerson | MainContactPerson / ContactPersons | If a |
RemoveInvalidXmlChars | Invalid XML characters are removed from the contact person object. |
Data Mapping Table (Item Class Value Discounts)
App4Sales Field | Source Field (API/Excel/DB) | Logic/Notes |
CustomerGuid | customer.CustomerGuid | Inherited from the parent customer. |
discount | discount | Only discounts greater than 0 are processed. |
Special Logic & Filters
Initial Data Validation and Filtering
Incoming customer records are validated for non-empty
CustomerCodeand non-empty/non-whitespaceCustomerName. Invalid records are excluded from further processing, and a warning is logged.
Script Hooks
BeforeUpdateCustomers: A script executed before any customer processing begins. Allows for custom logic to modify the incoming list of customers.UpdateCustomers: A script executed after each batch of customers has been processed (but before database commit). Allows for custom logic on the processed batch.
Customer GUID Management and Reactivation
The system attempts to assign a
CustomerGuidto each customer.If
CustomerGuidis empty and the process is running in "Updater" mode, the system checks for existing GUIDs of previously inactive customers (identified by customer codes starting with '~'). This mechanism reuses GUIDs to preserve historical data for reactivated customers.If no GUID is found through reactivation, the system queries the
PortalServerProviderusingCustomerCodeandCustomerNameto retrieve an existing GUID.If a
CustomerGuidremains empty after these checks, a new GUID is generated, indicating a new customer record. In Updater mode, this new GUID is added to a local cache (_customerCodeLinks).
Price List Deduplication Migration
If the
EnablePriceDeduplicationsetting is active (either globally or connector-specific), theUsesPriceFieldandActionPriceListIDs are migrated based on aKeySettings.GenericPriceListMigrationslookup table.If a price list ID specified in
UsesPriceFielddoes not exist in the system or in the migration lookup, it defaults to price list ID1.If an
ActionPriceListID does not exist or is not found in migration, it is set tonull.
Batch Processing
Customer updates are processed in batches of 100 customers to optimize database performance. Addresses, contacts, item class discounts, and dashboards are collected per batch.
Customer Dashboard Handling
If
CustomerDashboardcontains a non-empty string, it is assumed to be a Base64 encoded string. The string is decoded and stored asDashboardDatain a separateDashboardentity associated with the customer.Any errors during Base64 decoding result in the
CustomerDashboardfield being cleared.The
CustomerDashboardfield on theCustomerobject is set tonullafter processing to prevent its direct persistence.
Customer Notes
If
customer.CustomerNoteis provided, any existing customer notes associated with the "BackOffice" sales representative for that customer are deleted.A new customer note is then inserted into the
CustomerNotestable with "BackOffice" as the sales representative, "Backoffice note" as the subject, and the providedcustomer.CustomerNoteas the message, along with the current date.
VAT Handling
For administrations configured as USA (
AdministrationSession.CurrentSession.Administration.IsUSAis true), theVatCodefield for customers is set tonullandVatLiableis set tofalse.
Invalid XML Characters
Invalid XML characters are removed from both the main customer object and individual contact persons' data to prevent serialization issues.
Address Processing and Normalization
Addresses are filtered; only those that are not empty or have an
ExternalIdare processed.If no valid addresses remain but an ISO2 country code was initially present, a dummy "Visit" address is created to store this ISO2.
The system ensures a pair of "Visit" and "Delivery" addresses exists. If only one is present, a copy is made and its type is flipped. If neither is present but other address types exist, "Visit" and "Delivery" addresses are created from the most relevant existing address (main address first).
Email addresses within addresses are cleaned by removing specific Unicode characters.
ISO2 country codes are mapped using configured
MappedCountriesContext. If unmapped or empty, ISO2 can be derived from theCountryfield, and vice-versa.Default
AddressIdandAddressType(defaults to "Visit") are set if empty.Main addresses with empty email fields attempt to inherit email from the customer or other associated contacts/addresses.
AddressLine1can be parsed into structuredStreet,HouseNumber, andAddition, or vice-versa, depending on which fields are populated.For both "Visit" and "Delivery" addresses, if multiple exist and none is marked as main, the first one encountered is designated as the main address.
Extra Data Processing (from CSV)
Properties from
CsvCustomer(loaded from external CSV files) with matching names will overwrite corresponding properties in the mainCustomerobject.Elements in
CsvCustomer.UnknownElementswith keys prefixed "FreeField_" are parsed and added asCustomerFreeFieldobjects to the customer'sFreeFieldList.If
SyncSettings.CustomerItemFilteris enabled, elements inCsvCustomer.UnknownElementswith keys prefixed "ItemFilter_" are parsed. These are expected to contain item class values separated by '~' (e.g., "ItemFilter_Color~Red~Blue"). These values are validated against an internal item class map and then converted into a query string format, assigned to the customer'sItemFilter.
Contact Person Name Resolution
If
FirstName,LastName, andMiddleNameare all empty/null butFullNameis provided, the individual name components are parsed fromFullName.If
FullNameis empty/null but individual name components (FirstName,MiddleName,LastName) are provided,FullNameis constructed by concatenating them.Initialsare derived from the first character of theFirstName.
Domain Specifics
Customer Core Fields
Customer Identification:
CustomerCodeandCustomerNameare mandatory.CustomerGuidserves as the primary internal identifier, with logic for reusing existing GUIDs for reactivated customers (identified by codes starting with '~' in Updater mode) or generating new ones for new customers.Timestamps:
CreatedandSysmodifieddates are truncated to the second for database compatibility.Webshop Passwords: For security,
PasswordWebshopfields on both customer and contact person objects are set tonullwhen running in Updater mode.Language and Localization:
LanguageCodeis mapped to standardized App4Sales language codes using configured mappings. VAT fields (VatCode,VatLiable) are specifically set tonullandfalsefor USA administrations.Price Lists:
UsesPriceFieldandActionPriceListvalues are subject to migration if theEnablePriceDeduplicationsetting is active. ADefaultActionPriceListCodecan be configured to assign a default action price list to customers who don't have one specified.Payment Conditions: Empty
PaymentConditionCodestrings are converted tonullto prevent potential foreign key constraint issues in the database.Internal Data Preservation (Non-Updater Mode): When not in Updater mode (e.g., updates from App4Sales apps),
InternalCode,ExtraData, andLanguageCodefrom existing customer records in the database are preserved if the incoming customer object has empty values for these fields. This ensures that App4Sales apps do not inadvertently overwrite data managed externally.
Addresses
Address Validation and Filtering: Incoming addresses are filtered; only those that are not empty (contain relevant data) or possess an
ExternalIdare considered valid for processing.Guaranteed Visit/Delivery Addresses: The system actively works to ensure that each customer has at least one "Visit" and one "Delivery" address type. If only one of these types is provided, a copy is created and adjusted to represent the missing type. If neither is present, and other address types exist, "Visit" and "Delivery" addresses are generated from the most relevant existing address (prioritizing main addresses). A dummy "Visit" address can be created if all other addresses are invalid but an ISO2 country code was initially provided, to retain this localization information.
Data Cleaning: Specific Unicode unit separator characters (
0x001F) are removed from email addresses within address records.Country and ISO Mapping:
Iso2country codes are mapped to standardized App4Sales codes using theMappedCountriesContext. If anIso2code is initially missing, it can be derived from theCountryfield, and conversely, a missingCountryfield can be populated from theIso2code.Default Values: If
AddressIdis empty, a default ID is assigned. IfAddressTypeis empty, it defaults to "Visit".Email Inheritance for Main Addresses: If an address is designated as a main address (
IsMainAddressis true) and itsEmailfield is empty, the system attempts to populate it by inheriting an email from: 1) the customer's main email, 2) another address associated with the same customer, or 3) a contact person associated with the customer.Structured Address Handling: The single-line
AddressLine1field can be parsed into its structured components (Street,HouseNumber,Addition) if theAddressLineobject is null. Conversely, if the structuredAddressLineobject is populated, its components are formatted back intoAddressLine1, with formatting adjustments made if the administration is configured for USA.Main Address Designation for Types: For both "Visit" and "Delivery" address types, if multiple addresses of that type exist and none are explicitly marked as the main address, the first address encountered in the list is automatically designated as the main address for that type.
Invalid XML Characters: Invalid XML characters are removed from address objects to prevent serialization issues.
Contacts
Customer Linkage: Each contact person is linked to their parent customer through the
CustomerGuid.Webshop Passwords: The
PasswordWebshopfield for contact persons is set tonullwhen the system is operating in Updater mode.Email Trimming: Email addresses for contact persons are trimmed of leading or trailing whitespace.
Contact ID Generation (Updater Mode): If in Updater mode and a contact's
ContactIdis empty, a unique ID is generated based on the contact person's hash code to ensure identification.Dynamic Free Fields:
DynamicFreeFieldsfor contact persons, if provided as non-empty XML, are deserialized and converted into a standardizedExtraFieldsXML format using custom extra field configurations specifically defined for contact persons.Name Resolution: The system includes logic to populate name components (
FirstName,LastName,MiddleName) from a `FullName` string if the individual components are missing, and to construct `FullName` from individual components if `FullName` is missing.Initialsare derived from theFirstName.Main Contact Person: Logic ensures a single main contact person per customer. If a
MainContactPersonobject is explicitly provided, it is marked as the main contact. If no main contact is specified but other contacts exist, and none are marked as main, the first contact in the list is automatically designated as the main contact.Invalid XML Characters: Invalid XML characters are removed from contact person objects.
Extra Data & Free Fields
Property Overwrites from CSV: Properties within the
CsvCustomerobject (sourced from external CSV files) with values will overwrite identically named properties in the mainCustomerobject. Note that generic `Dictionary` type properties are explicitly skipped.FreeFieldList Population: Elements from
CsvCustomer.UnknownElementswhose keys are prefixed with "FreeField_" (e.g., "FreeField_CustomData") are parsed and added asCustomerFreeFieldobjects to the customer'sFreeFieldList. The portion of the key after the prefix becomes theCaption, and the value becomes theContent.DynamicFreeFields Conversion: XML payloads within
DynamicFreeFields(for both customers and contact persons) are deserialized and converted into a standardizedExtraFieldsXML format. This conversion leverages configured custom extra field definitions for the respective entities.ItemFilter from CSV: If the
SyncSettings.CustomerItemFilteris enabled, elements fromCsvCustomer.UnknownElementswith keys prefixed "ItemFilter_" are parsed to define item filters. These values (expected to be '~' separated, e.g., "ItemFilter_Color~Red~Blue") are validated against an internal item class map and then converted into a query string format, which is assigned to the customer'sItemFilter. This allows for customer-specific item class discounts or filters to be driven by external data.
Discounts & Dashboards
Item Class Value Discounts:
DiscountsPerItemCLassValueobjects are processed only if thediscountvalue is positive. Each discount record is explicitly linked to the customer via itsCustomerGuidbefore persistence.Customer Dashboards: The
CustomerDashboardfield on the incoming customer object is expected to contain a Base64 encoded string representing dashboard layout data. This data is decoded and then stored separately via the_customerDataManager.SaveDashboardsmethod. The originalCustomerDashboardfield on the customer object is cleared after processing to prevent redundant storage. Any errors during Base64 decoding will result in the dashboard data being discarded.
Related Settings & Prerequisites
The following settings and prerequisites influence the behavior of the Customer Update process:
CustomerItemFilter (
AdministrationSession.CurrentSession.SyncSettings.CustomerItemFilter): A boolean setting that, whentrue, enables the processing of item filters from extra customer data (e.g., CSV files) and ensures that existingItemFiltervalues are preserved during updates (when not in Updater mode).EnablePriceDeduplication (
AdministrationSession.CurrentSession.ConnectorBaseSettings.EnablePriceDeduplication): A boolean setting that, whentrue, activates the price list ID migration and deduplication logic, mapping old price list IDs to new, standardized ones.DefaultActionPriceListCode (
AdministrationSession.CurrentSession.ConnectorBaseSettings.DefaultActionPriceListCode): A string setting that, when configured with a price list code, will be used to assign a default action price list to customers who do not have one specified in their incoming data.Administration.IsUSA (
AdministrationSession.CurrentSession.Administration.IsUSA): A boolean flag that, whentrue, triggers specific logic for USA administrations, such as clearing VAT-related fields (VatCode,VatLiable) and adjusting address formatting (e.g., forAddressLine1).KeySettings.GenericPriceListMigrations: A JSON string stored in the
KeySettingsContext. This setting defines the mapping rules for migrating old price list IDs to new, deduplicated ones, which is crucial whenEnablePriceDeduplicationis active._databaseManager.ExtraCustomerData: This internal data source (typically populated from external CSV files via the connector's configuration) provides supplementary customer data that can overwrite or enrich standard customer properties.
CustomExtraFieldsContext: This context provides configuration for custom extra fields for both customers and contact persons, which are essential for the proper deserialization and conversion of
DynamicFreeFieldsXML payloads into a standardized `ExtraFields` format.MappedLanguagesContext: This context contains the configured mappings between ERP language codes and App4Sales' standardized language codes, used for normalizing the
LanguageCodefield on customer objects.MappedCountriesContext: This context holds the configured mappings between ERP country codes and App4Sales' standardized country codes, used for normalizing the
Iso2field on address objects.Script Hooks (
BeforeUpdateCustomers,UpdateCustomers): These are extension points that allow for the injection of custom business logic (via Jint scripts) before the customer update process begins (BeforeUpdateCustomers) and after each batch of customers has been processed (UpdateCustomers), enabling flexible data manipulation or validation.
Known Limitations
Contact Name Parsing (Future Improvement): The
FillContactNamemethod contains a TODO comment indicating that the current logic for parsing contact names fromFullName(and vice-versa) might be temporary or subject to future improvements. This suggests a potential limitation in the current parsing accuracy or flexibility, which may not handle all edge cases of name formats.Missing Mandatory Fields: Customer records with empty or null
CustomerCodeor empty/whitespace-onlyCustomerNameare considered invalid and are skipped from the update process.Customer Dashboard Data Format: The
CustomerDashboardfield strictly expects a Base64 encoded string. Any other format will result in the dashboard data being discarded during processing.Limited Extra Data Overwrite: The
ProcessExtraDataForCustomermethod explicitly skips properties of genericDictionarytypes when attempting to overwrite customer properties fromCsvCustomerdata. This means complex data structures within CSV extra data might not be fully integrated into the main customer object through this mechanism.Unmapped Item Classes in Item Filters: When processing item filters from extra data (e.g.,
ItemFilter_Color), any specified item classes that are not found in the configureditemClassMapwill result in those specific item filters being ignored.