NewBaseV4 Connector - CustomerUpdate

This document details the CustomerUpdate pipeline for the NewBaseV4 connector, explaining how customer data from the NewBaseV4 ERP system is synchronized and transformed into various App4Sales internal data models, including Customers, Addresses, Contact Persons, Item Class Value Discounts, and Dashboards. This process is crucial for maintaining up-to-date customer information within the App4Sales platform for Application Administrators and Implementation Consultants.

Data Source Configuration

The NewBaseV4 connector retrieves customer-related data primarily through its REST API. It fetches information about organisations, persons, contacts, account managers, and debtors from the NewBaseV4 system. Additional customer data can also be imported from CSV files. The connector pulls data from NewBaseV4 and pushes updates to the App4Sales internal database.

API Calls and Entities:

Organisations: Retrieved via _organisationProvider.GetOrganisations().
Persons: Retrieved via NewBaseRepository<NewBasePerson>.
Contacts: Retrieved via _contactsProvider.GetContacts().
Account Managers: Retrieved via NewBaseRepository<NewBaseAccountManager>.
Debtors: Retrieved via NewBaseRepository<NewBaseDebtor>, filtered to include only those with a valid CustomerCode.
Notes: Retrieved via NewBaseRepository<NewbaseNote> for existing notes, and inserted via API calls for new notes.
Addresses: Retrieved as part of NewBaseOrganisation data.

The base URL and specific endpoints are defined in NewBaseV4Constants.Routes (e.g., https://webservice.newbase.nl/servoy-service/rest_ws/newbase_ws/).

Data Mapping Table

Customer Core Fields

This section details how core customer fields in App4Sales are populated from NewBaseV4.

App4Sales Field	Source Field (API/Excel/DB)	Logic/Notes
`CustomerCode`	`NewBaseDebtor.CustomerCode` or `NewBaseOrganisation.OrganisationSerial`	Directly mapped from the NewBaseV4 Debtor's customer code. If the `AddOrganisationsThatAreLinkedToDebtorsToTheCustomers` setting is enabled, this is mapped from `NewBaseOrganisation.OrganisationSerial`.
`CustomerName`	`NewBaseDebtor.Name` or `NewBaseOrganisation.Name`	Directly mapped from the NewBaseV4 Debtor's name. If the `AddOrganisationsThatAreLinkedToDebtorsToTheCustomers` setting is enabled, this is mapped from `NewBaseOrganisation.Name`.
`CustomerEnabled`	`NewBaseDebtor.Stopped`	Set to `true` if `NewBaseDebtor.Stopped` is `false` or `null`. Always `true` for newly created App4Sales Customers.
`VatCode`	`NewBaseDebtor.VatNumber`	Directly mapped from the NewBaseV4 Debtor's VAT number. For USA administrations, this is always `null`.
`ChamberOfCommerceCode`	`NewBaseDebtor.ChamberOfCommerce`	Directly mapped from the NewBaseV4 Debtor's Chamber of Commerce registration.
`VatLiable`	Derived from customer's country and `CountryCodeForVatLiable` setting	Evaluated based on the customer's country. If the country code matches the `CountryCodeForVatLiable` setting or if `CountryCodeForVatLiable` is empty, `VatLiable` is set to `true`. Otherwise, it's `false`. For USA administrations, `VatLiable` is always `false`.
`PaymentConditionCode`	`NewBaseDebtor.PaymentTerms`	Mapped if `NewBaseDebtor.PaymentTerms` has a value and exists in the retrieved `PaymentConditions`. Otherwise, `null`.
`Created`	`NewBaseDebtor.CreatedDate`	Directly mapped from the NewBaseV4 Debtor's creation date. Truncated to the second during processing.
`Sysmodified`	System Timestamp	Truncated to the second during processing.
`CustomerManager`	`NewBaseAccountManager` (mapped to name)	Mapped from NewBaseV4 Account Manager data, linked by `OrganisationId`.
`LanguageCode`	`NewBaseDebtor.LanguageCode`	Directly mapped from the NewBaseV4 Debtor's language code. Further mapped using the `MappedLanguagesContext`.
`UsesPriceField`	`NewBaseDebtor.PriceListId`, `NewBaseDebtor.Currency`	Resolved by `PriceListServices.GetPriceListId` using `PriceListId` and `Currency`. Defaults to `0` if `PriceListId` is empty. If price deduplication is enabled, this field might be updated to a migrated price list ID.
`Currency`	`NewBaseDebtor.Currency`	Directly mapped from the NewBaseV4 Debtor's currency.
`InternalCode`	`NewBaseDebtor.CustomerGuid` or `NewBaseDebtor.CustomerCode`	Directly mapped from the NewBaseV4 Debtor's GUID. If the `AddOrganisationsThatAreLinkedToDebtorsToTheCustomers` setting is enabled, this is mapped from `NewBaseDebtor.CustomerCode`.
`ActionPriceList`	(Derived from settings)	If `ConnectorBaseSettings.DefaultActionPriceListCode` is set and `ActionPriceList` is empty, it's assigned based on `_priceListsContext`. If price deduplication is enabled, this field might be updated to a migrated price list ID.
`PasswordWebshop`	(N/A)	Set to `null` if the connector is running in 'Updater' mode.
`CustomerDashboard`	(Base64 encoded string from ERP)	The Base64 encoded string is decoded and stored as `DashboardData` in a separate `Dashboard` object. The `CustomerDashboard` property on the `Customer` object is then set to `null` to conserve memory.
`ItemFilter`	`CsvCustomer.UnknownElements` (prefixed with "ItemFilter_")	If `SyncSettings.CustomerItemFilter` is enabled, this is generated by processing elements from `CsvCustomer.UnknownElements` starting with "ItemFilter_" and matching them against existing item classes.
`Phone`	`NewBaseOrganisation.Reachabilities`	Retrieved from NewBaseV4 Organisation's reachabilities for `ReachabilityType.Phone`.
`Email`	`NewBaseOrganisation.Reachabilities`, `NewBaseContact.Reachabilities`	Initially from NewBaseV4 Organisation's reachabilities (`ReachabilityType.Email`). If empty, it attempts to use the email from an associated NewBaseV4 Contact. Trimmed during processing.
`Fax`	`NewBaseOrganisation.Reachabilities`	Retrieved from NewBaseV4 Organisation's reachabilities for `ReachabilityType.Fax`.
`FreeFields`	Derived from `NewBaseDebtor`, `NewBaseOrganisation`, `NewBaseContact`, `pricelist`	XML string containing `CustomerFreeField` entries. Includes: `OrganisationId`: From `NewBaseOrganisation.OrganisationId`. `OrganisationSerial`: From `NewBaseOrganisation.OrganisationSerial`. `DebtorId`: From `NewBaseContact.ContactId` if a matching contact person is found. `Pricelist`: If the `AddPricelistToDebtorFields` setting is true, includes the description of the price list used. `CustomerType`: If the `AddOrganisationsThatAreLinkedToDebtorsToTheCustomers` setting is true, set to "Organisation" (for organisations) or "Debtor" (for debtors). If `FreeFieldList` is populated, it's serialized to `FreeFields`.
`FreeFieldList`	`CsvCustomer.UnknownElements` (with prefix "FreeField_")	Populated by processing elements from `CsvCustomer.UnknownElements` starting with "FreeField_" and creating `CustomerFreeField` objects.
`DynamicFreeFields`	XML string (ERP source)	Deserialized from ERP source XML into `DynamicFreeFields` and then converted to an `ExtraFields` XML structure using custom extra field definitions.
(Various Customer properties)	`CsvCustomer` properties	Properties of the `Customer` object can be overwritten by similarly named properties from `CsvCustomer`.

Addresses

Addresses are obtained from NewBaseOrganisation.Addresses and processed for App4Sales.

App4Sales Field	Source Field (API/Excel/DB)	Logic/Notes
`AddressType`	`NewBaseAddress.AddressType`	Mapped to "VIS" (Visit), "DEL" (Delivery), or "INV" (Invoice). If `NewBaseAddress.AddressType` is "Mail" and there's only one address, it defaults to "DEL". If empty, defaults to "VIS".
`ExternalId`	`NewBaseAddress.AddressId`	Directly mapped from the NewBaseV4 Address ID.
`AddressLine.Street`	`NewBaseAddress.Street`	Directly mapped.
`AddressLine.Number`	`NewBaseAddress.AddressNumber`	Directly mapped.
`PostCode`	`NewBaseAddress.Zipcode`	Directly mapped.
`City`	`NewBaseAddress.City`	Directly mapped.
`Country`	`NewBaseAddress.CountryIso`	Derived from `CountryIso` using `CultureHelper.GetCountryFromIso2`. If mapped country codes are present, `addr.Iso2` is further mapped. If empty, it's set to `addr.Iso2`.
`Iso2`	`NewBaseAddress.CountryIso`	Directly mapped. Trimmed and converted to lowercase for mapping with `mappedCountryCodes`. If empty, derived from `Country`.
`Phone`	`NewBaseOrganisation.Reachabilities`	Retrieved from NewBaseV4 Organisation's reachabilities for `ReachabilityType.Phone`.
`Email`	`NewBaseOrganisation.Reachabilities`, `NewBaseContact.Reachabilities`, `Customer.Email`, other Addresses/ContactPersons	Initially from NewBaseV4 Organisation's reachabilities (`ReachabilityType.Email`). If empty, it attempts to use the email from an associated NewBaseV4 Contact. Trimmed during processing. If `IsMainAddress` and empty, inherits from `Customer.Email` or other available addresses/contact persons.
`IsMainAddress`	`NewBaseAddress.IsMain`	Set to `true` if `NewBaseAddress.IsMain` is `1`. Otherwise `false`. If no main address is explicitly set for Visit/Delivery, the first one found for that type is marked as main.
`CustomerGuid`	`Customer.CustomerGuid`	Assigned from the parent customer's GUID.
`AddressId`	Generated hash or ERP address id	If empty, a hash code is generated. Can also contain the ERP address ID.

Contacts

Contact persons are obtained from NewBasePerson, NewBaseContact, and NewBaseOrganisation.

App4Sales Field	Source Field (API/Excel/DB)	Logic/Notes
`ContactId`	`NewBaseContact.ContactId`	Directly mapped. If the connector is in 'Updater' mode and `ContactId` is empty, a hash code is generated as `ContactId`.
`FirstName`	`NewBasePerson.Firstname`	Directly mapped. If `FullName` is available but `FirstName`, `MiddleName`, `LastName` are not, they are parsed from `FullName`.
`MiddleName`	Derived from `FullName`	Parsed from `FullName` if available.
`LastName`	`NewBasePerson.Lastname`	Directly mapped. If `FullName` is available but `FirstName`, `MiddleName`, `LastName` are not, they are parsed from `FullName`.
`FullName`	`NewBasePerson.Firstname`, `NewBasePerson.Lastname`	Composed as `"{Firstname} {Lastname}"`. If `FullName` is empty, it's composed from `FirstName`, `MiddleName`, `LastName`.
`Initials`	`FirstName`	First character of `FirstName`.
`Email`	`NewBaseContact.Reachabilities`, `NewBaseOrganisation.Reachabilities`	Initially from NewBaseV4 Contact's reachabilities (`ReachabilityType.Email`). If empty, it attempts to use the email from the associated NewBaseV4 Organisation. Trimmed during processing.
`MobileNumber`	`NewBaseContact.Reachabilities`	Retrieved from NewBaseV4 Contact's reachabilities for `ReachabilityType.Mobile`.
`Phonenumber`	`NewBaseContact.Reachabilities`	Retrieved from NewBaseV4 Contact's reachabilities for `ReachabilityType.Phone`.
`Gender`	`NewBasePerson.Gender`	Mapped via `OrganisationExtensions.GetGenderCode` (1 for Male, 2 for Female, 3 for Unknown).
`IsMainContactPerson`	`NewBaseContact.IsMain`	Set to `true` if `NewBaseContact.IsMain` is `1`. Otherwise `false`. If no contact is marked as main, the first one in the list is marked as main.
`CustomerGuid`	`Customer.CustomerGuid`	Assigned from the parent customer's GUID.
`PasswordWebshop`	(N/A)	Set to `null` if the connector is running in 'Updater' mode.
`DynamicFreeFields`	XML string (ERP source)	Deserialized from ERP source XML into `DynamicFreeFields` and then converted to an `ExtraFields` XML structure using custom extra field definitions.

Discounts & Dashboards

This section outlines how item class value discounts and customer dashboards are handled.

App4Sales Field	Source Field (API/Excel/DB)	Logic/Notes
`DbItemClassValueDiscount.CustomerGuid`	`Customer.CustomerGuid`	Assigned from the parent customer's GUID.
`DbItemClassValueDiscount.Discount`	`ItemClassValueDiscount.discount` (from source)	Mapped if `discount` is greater than 0. Stored in `itemClassValueDiscounts` and updated via the `OPTA4SUpdateItemClassValueDiscountsBatch` stored procedure.
`Dashboard.CustomerCode`	`Customer.CustomerCode`	Mapped from the customer's code.
`Dashboard.DashboardData`	`Customer.CustomerDashboard` (Base64 string)	Base64 decoded from the `Customer.CustomerDashboard` property. Stored via `_customerDataManager.SaveDashboards`.

Special Logic & Filters

Customer Filtering:
- Customers with empty CustomerName or CustomerCode are filtered out and logged as invalid.
- Customers can be skipped if their names start with prefixes defined in the SkipCustomersWithNamePrefix setting.
Customer GUID Reactivation:
- When the connector is in 'Updater' mode, existing GUIDs of inactive customers (identified by customer codes prefixed with "~") are reused to prevent data loss.
Price Deduplication:
- If EnablePriceDeduplication is enabled, customer price list IDs (UsesPriceField, ActionPriceList) are migrated to deduplicated versions based on a lookup table (KeySettings.GenericPriceListMigrations).
Address Normalization:
- The system ensures that each customer has at least one "Visit" and one "Delivery" address. If only one type exists, it's duplicated to create the other. If no addresses are provided, a dummy address is created to store the ISO2 country code.
- Country codes are normalized using MappedCountriesContext and `CultureHelper.GetCountryFromIso2`.
- Email addresses within addresses are trimmed, and if empty and marked as main, they can inherit from the customer's main email or other contact emails.
- Address lines are parsed into street, house number, and addition components, or formatted back into a single line.
Contact Person Name Filling:
- The FillContactName method attempts to parse a full name into first, middle, and last names, or compose a full name from individual name components. It also sets initials.
Contact ID Generation (Updater Mode):
- If the connector is in 'Updater' mode and a ContactId is not provided by the ERP for a contact person, a hash code is generated to serve as a temporary ID.
XML Character Removal:
- Invalid XML characters are removed from Customer and ContactPerson objects before persistence to prevent data corruption.
Customer Notes:
- If a CustomerNote is present, any existing customer notes from the "BackOffice" source are deleted from the CustomerNotes table, and the new note is inserted. This ensures that only the latest note from the ERP is present.
VAT Handling for USA:
- For administrations configured for the USA, VatCode is set to null and VatLiable is set to false, aligning with regional tax regulations.
Extra Data Overwrite (from CSV):
- The ProcessExtraDataForCustomer method allows properties of the Customer object to be overwritten by similarly named properties found in the CsvCustomer object (typically from FTP/CSV imports).
JINT Scripting:
- The connector supports custom JINT scripts that can be executed at various stages: "BeforeUpdateCustomers" (before processing any customers), and "UpdateCustomers" (both before and after batch updates). This allows for highly customized transformations or business logic.
Batch Processing:
- Customer updates are processed in batches of 100 to optimize performance and reduce database load.

Settings & Prerequisites

The following connector settings and prerequisites influence the behavior of the Customer Update process:

Setting Name	Description	Connector Component
`AddOrganisationsThatAreLinkedToDebtorsToTheCustomers`	When enabled, customers can also be created based on organisations that are linked to debtors, in addition to direct debtors. This expands the scope of customer synchronization.	`CustomersService`, `OrganisationExtensions`, `DebtorExtensions`
`CustomerNumberGenerationLastId`	The last used sequential ID for generating new customer codes. This value is incremented for each new customer created through the connector.	`CustomersService`
`CustomerNumberGenerationPrefix`	A configurable prefix (e.g., "CUST-") to be used for newly generated customer codes. This helps in identifying customers originating from this connector.	`CustomersService`
`CustomerNumberGenerationNumDigits`	Defines the minimum number of digits for the numeric part of generated customer codes. The default is 5 digits, but can be set to a minimum of 2. For example, a setting of 5 for ID '1' would result in '00001'.	`CustomersService`
`UseAddressesListForOrganisationAddresses`	This setting influences how visit addresses are retrieved for organisations from NewBaseV4.	`CustomersService`
`CountryCodeForVatLiable`	Specifies the ISO 2-letter country code that, if matched, will mark a customer as VAT liable. If this setting is left empty, all customers are by default considered VAT liable.	`DebtorExtensions`, `OrganisationExtensions`
`SkipCustomersWithNamePrefix`	A comma-separated list of string prefixes. Any customer whose name begins with one of these prefixes will be entirely excluded from the update process.	`DebtorExtensions`
`EnablePriceDeduplication`	When enabled, the connector will attempt to migrate customer price list IDs (`UsesPriceField`, `ActionPriceList`) to deduplicated versions. This relies on a migration lookup table stored in `KeySettings.GenericPriceListMigrations`.	`CustomerUpdateHandler`
`DefaultActionPriceListCode`	The code of a specific price list that will be set as the default `ActionPriceList` for customers if an action price list is not explicitly provided by the ERP.	`CustomerUpdateHandler`
`CustomerItemFilter`	When enabled, the connector processes item filters provided in `CsvCustomer.UnknownElements` (specifically those prefixed with "ItemFilter_") and applies them to the `customer.ItemFilter` field.	`CustomerUpdateHandler`
`AddPricelistToDebtorFields`	If true, the description of the customer's assigned price list will be added to their free fields (`FreeFields` XML).	`DebtorExtensions`, `OrganisationExtensions`
`CustomerCustomFieldsMapping`	A JSON string that defines custom mappings between App4Sales fields (or extra fields) and specific fields within the NewBaseV4 entities (Organisation, Debtor, Contact).	`CustomerExtensions`

Known Limitations

Email Ambiguity in Addresses: If a NewBaseV4 organisation's email is empty, the system may pick the email from the *first random* associated contact. This might not always correspond to the primary contact's email or the intended email for the address.
Customer Note Overwriting: When a CustomerNote is provided by the ERP, any existing customer notes marked as from "BackOffice" in App4Sales are deleted and replaced. This effectively overwrites historical notes from this specific source, potentially losing previous versions if not managed externally.
Contact ID Stability in Updater Mode: If the connector is in 'Updater' mode and the ERP does not provide a ContactId for a contact person, a hash code is generated. This generated ID might not be stable across connector updates if the underlying data used to generate the hash changes, potentially leading to inconsistencies.

NewBaseV4 - Customer sync