Skip to main content

Spector - Customer sync

Spector Connector - Customer Update Overview This document describes the customer update process for the Spector connector, specific...

Updated over a week ago

Spector Connector - Customer Update

Overview

This document describes the customer update process for the Spector connector, specifically handled by the CustomerUpdateHandler. It details how customer data, including core customer fields, addresses, contact persons, discounts, and dashboards, is retrieved and synchronized with the App4Sales internal database. The process involves initial data validation, GUID resolution (including reactivation logic), price list deduplication, language and country code mapping, handling of customer notes and dashboards, and the processing of extra data fields. Updates are performed in batches to optimize performance.

Data Source Configuration

The primary input for the customer update process is a collection of Customer objects. These objects typically originate from the external ERP system via the Spector connector. Additionally, extra customer-specific data can be supplied through CSV files, accessible via _databaseManager.ExtraCustomerData, which contains CsvCustomer objects. This extra data is identified and merged with the main customer records based on CustomerCode. Language and country mappings are retrieved from the App4Sales database via MappedLanguagesContext and MappedCountriesContext respectively.

Data Mapping Table (Customers)

App4Sales Field

Source Field (API/Excel/DB)

Logic/Notes

CustomerCode

CustomerCode

Mandatory. Used as a primary identifier. Customers with empty or null codes are skipped.

CustomerName

CustomerName

Mandatory. Customers with empty or whitespace-only names are skipped.

CustomerGuid

CustomerGuid / _customerCodeLinks / Portal API / New GUID

If empty: Tries to reuse GUID from previously inactive customers (code starts with '~') in Updater mode. If not found, retrieves from _portalServerProvider using CustomerCode and CustomerName. If still empty, a new GUID is generated.

Created

Created

If DateTime.MinValue, set to DateTime.Now. Truncated to the second to avoid SQL issues.

Sysmodified

Sysmodified

Truncated to the second to avoid SQL issues.

PasswordWebshop

PasswordWebshop

Set to null if in Updater mode.

CustomerDashboard

CustomerDashboard

If empty string or whitespace, set to null. If non-empty, attempts to decode as Base64 string and stores in a separate Dashboard entity. Original field then set to null.

LanguageCode

LanguageCode

Mapped to a standardized App4Sales language code using mappedLanguages configured in MappedLanguagesContext.

ActionPriceList

ActionPriceList / _administrationSession.ConnectorBaseSettings.DefaultActionPriceListCode

If DefaultActionPriceListCode is configured in settings and ActionPriceList is not provided, it's set to the ID of the default action price list. If Price Deduplication is enabled, IDs are migrated.

UsesPriceField

UsesPriceField

If Price Deduplication is enabled, IDs are migrated to deduplicated versions. If an old ID is not found, it defaults to price list ID 1.

PaymentConditionCode

PaymentConditionCode

Empty strings are converted to null to prevent foreign key issues.

Email

Email

Whitespace is trimmed.

DynamicFreeFields

DynamicFreeFields

If contains non-empty XML, it's deserialized, converted to a standardized ExtraFields XML format using customerExtraFields (configured custom extra fields for customers).

InternalCode

InternalCode

If not in Updater mode and incoming InternalCode is empty, it preserves the existing InternalCode from the database. Can be overwritten by ExtraCustomerData.

ExtraData

ExtraData

If not in Updater mode and incoming ExtraData is empty, it preserves the existing ExtraData from the database. Can be overwritten by ExtraCustomerData.

VatCode

VatCode

Set to null if Administration.IsUSA is true. Can be overwritten by ExtraCustomerData.

VatLiable

VatLiable

Set to false if Administration.IsUSA is true. Can be overwritten by ExtraCustomerData.

FreeFields

FreeFieldList / extraDataForCustomer.UnknownElements (starting with "FreeField_")

If FreeFields is empty but FreeFieldList contains items, FreeFieldList is serialized to XML and assigned. Additionally, extra data elements from CsvCustomer.UnknownElements with keys prefixed "FreeField_" are added to FreeFieldList.

ItemFilter

ItemFilter / extraDataForCustomer.UnknownElements (starting with "ItemFilter_")

If SyncSettings.CustomerItemFilter is enabled: if not in Updater mode, the existing ItemFilter from the database is preserved if the incoming is empty. If processing extra data from CSV, item filters are extracted from extraDataForCustomer.UnknownElements (keys starting with "ItemFilter_", e.g., "ItemFilter_Color~Red~Blue") and converted to a query string (e.g., "&Color=Red&Color=Blue"). The ItemClass names are matched against an itemClassMap and item class values are validated against configured values.

(Other Customer Properties)

CsvCustomer (matching property name)

Properties in the CsvCustomer object (from _databaseManager.ExtraCustomerData) with a value will overwrite corresponding properties in the main Customer object if their names match.

Data Mapping Table (Addresses)

App4Sales Field

Source Field (API/Excel/DB)

Logic/Notes

Email

Email

Unicode unit separator (0x001F) characters are removed. If address is main and its email is empty, it attempts to inherit from customer's email, other customer addresses, or contact persons.

Iso2

Iso2 / Country

If provided and a mapping exists in mappedCountryCodes, it's replaced with the mapped CountryCode. Otherwise, trimmed. If still empty, derived from Country using CultureHelper.

CustomerGuid

customer.CustomerGuid

Inherited from the parent customer.

AddressId

AddressId

If empty, a default ID is set.

Country

Country / Iso2

If empty, it's set to the Iso2 value.

AddressType

AddressType

If empty, defaults to "Visit".

AddressLine (Street, HouseNumber, Addition)

AddressLine1

If AddressLine object is null, AddressLine1 (single string) is parsed into Street, HouseNumber, and Addition.

AddressLine1

AddressLine (Street, HouseNumber, Addition)

If AddressLine object exists, Street, HouseNumber, and Addition are formatted back into AddressLine1 string, considering IsUSA administration setting.

RemoveInvalidXmlChars

Invalid XML characters are removed from the address object.

Data Mapping Table (Contact Persons)

App4Sales Field

Source Field (API/Excel/DB)

Logic/Notes

CustomerGuid

customer.CustomerGuid

Inherited from the parent customer.

PasswordWebshop

PasswordWebshop

Set to null if in Updater mode.

Email

Email

Whitespace is trimmed.

ContactId

ContactId

If in Updater mode and empty, a ContactId is generated from the contact person's hash code.

DynamicFreeFields

DynamicFreeFields

If contains non-empty XML, it's deserialized, converted to a standardized ExtraFields XML format using contactPersonExtraFields (configured custom extra fields for contact persons).

FirstName

FirstName / FullName

If FirstName, LastName, MiddleName are null but FullName exists, FirstName is derived from the first part of FullName.

LastName

LastName / FullName

If FirstName, LastName, MiddleName are null but FullName exists, LastName is derived from the last part of FullName.

MiddleName

MiddleName / FullName

If FirstName, LastName, MiddleName are null but FullName exists, MiddleName is derived from the middle parts of FullName.

FullName

FullName / FirstName, MiddleName, LastName

If FullName is null, it's constructed from FirstName, MiddleName, and LastName.

Initials

FirstName

Set to the first character of FirstName if available, otherwise empty.

IsMainContactPerson

MainContactPerson / ContactPersons

If a MainContactPerson is provided, it's marked as main. If no main contact is specified but other contacts exist, the first contact in the list is set as main.

RemoveInvalidXmlChars

Invalid XML characters are removed from the contact person object.

Data Mapping Table (Item Class Value Discounts)

App4Sales Field

Source Field (API/Excel/DB)

Logic/Notes

CustomerGuid

customer.CustomerGuid

Inherited from the parent customer.

discount

discount

Only discounts greater than 0 are processed.

Special Logic & Filters

Initial Data Validation and Filtering

  • Incoming customer records are validated for non-empty CustomerCode and non-empty/non-whitespace CustomerName. Invalid records are excluded from further processing, and a warning is logged.

Script Hooks

  • BeforeUpdateCustomers: A script executed before any customer processing begins. Allows for custom logic to modify the incoming list of customers.

  • UpdateCustomers: A script executed after each batch of customers has been processed (but before database commit). Allows for custom logic on the processed batch.

Customer GUID Management and Reactivation

  • The system attempts to assign a CustomerGuid to each customer.

  • If CustomerGuid is empty and the process is running in "Updater" mode, the system checks for existing GUIDs of previously inactive customers (identified by customer codes starting with '~'). This mechanism reuses GUIDs to preserve historical data for reactivated customers.

  • If no GUID is found through reactivation, the system queries the PortalServerProvider using CustomerCode and CustomerName to retrieve an existing GUID.

  • If a CustomerGuid remains empty after these checks, a new GUID is generated, indicating a new customer record. In Updater mode, this new GUID is added to a local cache (_customerCodeLinks).

Price List Deduplication Migration

  • If the EnablePriceDeduplication setting is active (either globally or connector-specific), the UsesPriceField and ActionPriceList IDs are migrated based on a KeySettings.GenericPriceListMigrations lookup table.

  • If a price list ID specified in UsesPriceField does not exist in the system or in the migration lookup, it defaults to price list ID 1.

  • If an ActionPriceList ID does not exist or is not found in migration, it is set to null.

Batch Processing

  • Customer updates are processed in batches of 100 customers to optimize database performance. Addresses, contacts, item class discounts, and dashboards are collected per batch.

Customer Dashboard Handling

  • If CustomerDashboard contains a non-empty string, it is assumed to be a Base64 encoded string. The string is decoded and stored as DashboardData in a separate Dashboard entity associated with the customer.

  • Any errors during Base64 decoding result in the CustomerDashboard field being cleared.

  • The CustomerDashboard field on the Customer object is set to null after processing to prevent its direct persistence.

Customer Notes

  • If customer.CustomerNote is provided, any existing customer notes associated with the "BackOffice" sales representative for that customer are deleted.

  • A new customer note is then inserted into the CustomerNotes table with "BackOffice" as the sales representative, "Backoffice note" as the subject, and the provided customer.CustomerNote as the message, along with the current date.

VAT Handling

  • For administrations configured as USA (AdministrationSession.CurrentSession.Administration.IsUSA is true), the VatCode field for customers is set to null and VatLiable is set to false.

Invalid XML Characters

  • Invalid XML characters are removed from both the main customer object and individual contact persons' data to prevent serialization issues.

Address Processing and Normalization

  • Addresses are filtered; only those that are not empty or have an ExternalId are processed.

  • If no valid addresses remain but an ISO2 country code was initially present, a dummy "Visit" address is created to store this ISO2.

  • The system ensures a pair of "Visit" and "Delivery" addresses exists. If only one is present, a copy is made and its type is flipped. If neither is present but other address types exist, "Visit" and "Delivery" addresses are created from the most relevant existing address (main address first).

  • Email addresses within addresses are cleaned by removing specific Unicode characters.

  • ISO2 country codes are mapped using configured MappedCountriesContext. If unmapped or empty, ISO2 can be derived from the Country field, and vice-versa.

  • Default AddressId and AddressType (defaults to "Visit") are set if empty.

  • Main addresses with empty email fields attempt to inherit email from the customer or other associated contacts/addresses.

  • AddressLine1 can be parsed into structured Street, HouseNumber, and Addition, or vice-versa, depending on which fields are populated.

  • For both "Visit" and "Delivery" addresses, if multiple exist and none is marked as main, the first one encountered is designated as the main address.

Extra Data Processing (from CSV)

  • Properties from CsvCustomer (loaded from external CSV files) with matching names will overwrite corresponding properties in the main Customer object.

  • Elements in CsvCustomer.UnknownElements with keys prefixed "FreeField_" are parsed and added as CustomerFreeField objects to the customer's FreeFieldList.

  • If SyncSettings.CustomerItemFilter is enabled, elements in CsvCustomer.UnknownElements with keys prefixed "ItemFilter_" are parsed. These are expected to contain item class values separated by '~' (e.g., "ItemFilter_Color~Red~Blue"). These values are validated against an internal item class map and then converted into a query string format, assigned to the customer's ItemFilter.

Contact Person Name Resolution

  • If FirstName, LastName, and MiddleName are all empty/null but FullName is provided, the individual name components are parsed from FullName.

  • If FullName is empty/null but individual name components (FirstName, MiddleName, LastName) are provided, FullName is constructed by concatenating them.

  • Initials are derived from the first character of the FirstName.

Domain Specifics

Customer Core Fields

  • Customer Identification: CustomerCode and CustomerName are mandatory. CustomerGuid serves as the primary internal identifier, with logic for reusing existing GUIDs for reactivated customers (identified by codes starting with '~' in Updater mode) or generating new ones for new customers.

  • Timestamps: Created and Sysmodified dates are truncated to the second for database compatibility.

  • Webshop Passwords: For security, PasswordWebshop fields on both customer and contact person objects are set to null when running in Updater mode.

  • Language and Localization: LanguageCode is mapped to standardized App4Sales language codes using configured mappings. VAT fields (VatCode, VatLiable) are specifically set to null and false for USA administrations.

  • Price Lists: UsesPriceField and ActionPriceList values are subject to migration if the EnablePriceDeduplication setting is active. A DefaultActionPriceListCode can be configured to assign a default action price list to customers who don't have one specified.

  • Payment Conditions: Empty PaymentConditionCode strings are converted to null to prevent potential foreign key constraint issues in the database.

  • Internal Data Preservation (Non-Updater Mode): When not in Updater mode (e.g., updates from App4Sales apps), InternalCode, ExtraData, and LanguageCode from existing customer records in the database are preserved if the incoming customer object has empty values for these fields. This ensures that App4Sales apps do not inadvertently overwrite data managed externally.

Addresses

  • Address Validation and Filtering: Incoming addresses are filtered; only those that are not empty (contain relevant data) or possess an ExternalId are considered valid for processing.

  • Guaranteed Visit/Delivery Addresses: The system actively works to ensure that each customer has at least one "Visit" and one "Delivery" address type. If only one of these types is provided, a copy is created and adjusted to represent the missing type. If neither is present, and other address types exist, "Visit" and "Delivery" addresses are generated from the most relevant existing address (prioritizing main addresses). A dummy "Visit" address can be created if all other addresses are invalid but an ISO2 country code was initially provided, to retain this localization information.

  • Data Cleaning: Specific Unicode unit separator characters (0x001F) are removed from email addresses within address records.

  • Country and ISO Mapping: Iso2 country codes are mapped to standardized App4Sales codes using the MappedCountriesContext. If an Iso2 code is initially missing, it can be derived from the Country field, and conversely, a missing Country field can be populated from the Iso2 code.

  • Default Values: If AddressId is empty, a default ID is assigned. If AddressType is empty, it defaults to "Visit".

  • Email Inheritance for Main Addresses: If an address is designated as a main address (IsMainAddress is true) and its Email field is empty, the system attempts to populate it by inheriting an email from: 1) the customer's main email, 2) another address associated with the same customer, or 3) a contact person associated with the customer.

  • Structured Address Handling: The single-line AddressLine1 field can be parsed into its structured components (Street, HouseNumber, Addition) if the AddressLine object is null. Conversely, if the structured AddressLine object is populated, its components are formatted back into AddressLine1, with formatting adjustments made if the administration is configured for USA.

  • Main Address Designation for Types: For both "Visit" and "Delivery" address types, if multiple addresses of that type exist and none are explicitly marked as the main address, the first address encountered in the list is automatically designated as the main address for that type.

  • Invalid XML Characters: Invalid XML characters are removed from address objects to prevent serialization issues.

Contacts

  • Customer Linkage: Each contact person is linked to their parent customer through the CustomerGuid.

  • Webshop Passwords: The PasswordWebshop field for contact persons is set to null when the system is operating in Updater mode.

  • Email Trimming: Email addresses for contact persons are trimmed of leading or trailing whitespace.

  • Contact ID Generation (Updater Mode): If in Updater mode and a contact's ContactId is empty, a unique ID is generated based on the contact person's hash code to ensure identification.

  • Dynamic Free Fields: DynamicFreeFields for contact persons, if provided as non-empty XML, are deserialized and converted into a standardized ExtraFields XML format using custom extra field configurations specifically defined for contact persons.

  • Name Resolution: The system includes logic to populate name components (FirstName, LastName, MiddleName) from a `FullName` string if the individual components are missing, and to construct `FullName` from individual components if `FullName` is missing. Initials are derived from the FirstName.

  • Main Contact Person: Logic ensures a single main contact person per customer. If a MainContactPerson object is explicitly provided, it is marked as the main contact. If no main contact is specified but other contacts exist, and none are marked as main, the first contact in the list is automatically designated as the main contact.

  • Invalid XML Characters: Invalid XML characters are removed from contact person objects.

Extra Data & Free Fields

  • Property Overwrites from CSV: Properties within the CsvCustomer object (sourced from external CSV files) with values will overwrite identically named properties in the main Customer object. Note that generic `Dictionary` type properties are explicitly skipped.

  • FreeFieldList Population: Elements from CsvCustomer.UnknownElements whose keys are prefixed with "FreeField_" (e.g., "FreeField_CustomData") are parsed and added as CustomerFreeField objects to the customer's FreeFieldList. The portion of the key after the prefix becomes the Caption, and the value becomes the Content.

  • DynamicFreeFields Conversion: XML payloads within DynamicFreeFields (for both customers and contact persons) are deserialized and converted into a standardized ExtraFields XML format. This conversion leverages configured custom extra field definitions for the respective entities.

  • ItemFilter from CSV: If the SyncSettings.CustomerItemFilter is enabled, elements from CsvCustomer.UnknownElements with keys prefixed "ItemFilter_" are parsed to define item filters. These values (expected to be '~' separated, e.g., "ItemFilter_Color~Red~Blue") are validated against an internal item class map and then converted into a query string format, which is assigned to the customer's ItemFilter. This allows for customer-specific item class discounts or filters to be driven by external data.

Discounts & Dashboards

  • Item Class Value Discounts: DiscountsPerItemCLassValue objects are processed only if the discount value is positive. Each discount record is explicitly linked to the customer via its CustomerGuid before persistence.

  • Customer Dashboards: The CustomerDashboard field on the incoming customer object is expected to contain a Base64 encoded string representing dashboard layout data. This data is decoded and then stored separately via the _customerDataManager.SaveDashboards method. The original CustomerDashboard field on the customer object is cleared after processing to prevent redundant storage. Any errors during Base64 decoding will result in the dashboard data being discarded.

Related Settings & Prerequisites

The following settings and prerequisites influence the behavior of the Customer Update process:

  • CustomerItemFilter (AdministrationSession.CurrentSession.SyncSettings.CustomerItemFilter): A boolean setting that, when true, enables the processing of item filters from extra customer data (e.g., CSV files) and ensures that existing ItemFilter values are preserved during updates (when not in Updater mode).

  • EnablePriceDeduplication (AdministrationSession.CurrentSession.ConnectorBaseSettings.EnablePriceDeduplication): A boolean setting that, when true, activates the price list ID migration and deduplication logic, mapping old price list IDs to new, standardized ones.

  • DefaultActionPriceListCode (AdministrationSession.CurrentSession.ConnectorBaseSettings.DefaultActionPriceListCode): A string setting that, when configured with a price list code, will be used to assign a default action price list to customers who do not have one specified in their incoming data.

  • Administration.IsUSA (AdministrationSession.CurrentSession.Administration.IsUSA): A boolean flag that, when true, triggers specific logic for USA administrations, such as clearing VAT-related fields (VatCode, VatLiable) and adjusting address formatting (e.g., for AddressLine1).

  • KeySettings.GenericPriceListMigrations: A JSON string stored in the KeySettingsContext. This setting defines the mapping rules for migrating old price list IDs to new, deduplicated ones, which is crucial when EnablePriceDeduplication is active.

  • _databaseManager.ExtraCustomerData: This internal data source (typically populated from external CSV files via the connector's configuration) provides supplementary customer data that can overwrite or enrich standard customer properties.

  • CustomExtraFieldsContext: This context provides configuration for custom extra fields for both customers and contact persons, which are essential for the proper deserialization and conversion of DynamicFreeFields XML payloads into a standardized `ExtraFields` format.

  • MappedLanguagesContext: This context contains the configured mappings between ERP language codes and App4Sales' standardized language codes, used for normalizing the LanguageCode field on customer objects.

  • MappedCountriesContext: This context holds the configured mappings between ERP country codes and App4Sales' standardized country codes, used for normalizing the Iso2 field on address objects.

  • Script Hooks (BeforeUpdateCustomers, UpdateCustomers): These are extension points that allow for the injection of custom business logic (via Jint scripts) before the customer update process begins (BeforeUpdateCustomers) and after each batch of customers has been processed (UpdateCustomers), enabling flexible data manipulation or validation.

Known Limitations

  • Contact Name Parsing (Future Improvement): The FillContactName method contains a TODO comment indicating that the current logic for parsing contact names from FullName (and vice-versa) might be temporary or subject to future improvements. This suggests a potential limitation in the current parsing accuracy or flexibility, which may not handle all edge cases of name formats.

  • Missing Mandatory Fields: Customer records with empty or null CustomerCode or empty/whitespace-only CustomerName are considered invalid and are skipped from the update process.

  • Customer Dashboard Data Format: The CustomerDashboard field strictly expects a Base64 encoded string. Any other format will result in the dashboard data being discarded during processing.

  • Limited Extra Data Overwrite: The ProcessExtraDataForCustomer method explicitly skips properties of generic Dictionary types when attempting to overwrite customer properties from CsvCustomer data. This means complex data structures within CSV extra data might not be fully integrated into the main customer object through this mechanism.

  • Unmapped Item Classes in Item Filters: When processing item filters from extra data (e.g., ItemFilter_Color), any specified item classes that are not found in the configured itemClassMap will result in those specific item filters being ignored.

Did this answer your question?