Files
bb28968151 fix(import): chunk SIE account creation and close the import row on every exit (#2451)
* fix(import): chunk SIE account creation and close the import row on every exit

A full-BAS Bokio SIE file creates 1 200+ chart_of_accounts rows in one
INSERT. PostgREST runs it under the authenticated role's 8 s
statement_timeout, and with four row-level triggers plus the RLS WITH
CHECK that single statement measured 6.5 s to 8.2 s on prod: it was
cancelled for one company and passed for the next (2026-09-09).

- syncMappedAccounts inserts in chunks of 100 rows (INSERT_CHUNK_SIZE),
  so every statement stays an order of magnitude inside the limit. A
  failed chunk leaves the earlier ones committed; the next attempt reads
  the chart again and inserts only what is still missing.
- executeSIEImport closes its pending sie_imports row in a finally
  block. Every early `return result` after createPendingImportRecord
  (account sync failure, missing fiscal year, overlapping import,
  vouchers outside the year) used to leave the row 'pending'. That row
  holds the (company_id, file_hash) slot in the partial unique index,
  so a retry inside the five-minute cleanup gate failed on the index
  instead of on the real error.
- The slot-held message no longer names "gnubok", an "Ersätt import"
  button the import history has never had, or Fortnox; it says the
  same file is being imported or was interrupted moments ago and to
  retry in a few minutes. The thrown-error prefix is Swedish
  ("Importen misslyckades:") and the two undo hints name
  accounted_undo_sie_import.

Tests: chunk sizes and first-failing-chunk behaviour in
account-sync.test.ts; failed-row finalize on two early exits and the
new slot message in sie-import.account-names.test.ts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBTRraVXkCnx93Pa9wxJ6G

* fix(import): write account chunks as ignore-duplicates upserts with an exact count

Review follow-up on #2451. A plain INSERT per chunk still had the old
race: a concurrent import (or the replace flow) creating one account
between our read and write raised a duplicate-key error that the code
swallowed as success, while PostgREST had rolled back the whole chunk,
so every other account in it was silently missing.

Each chunk is now an upsert with onConflict (company_id, account_number)
and ignoreDuplicates, selecting the landed rows: the race becomes a
skipped row, `created` counts exactly what was written (also across a
mid-loop failure, which the compliance review flagged), and the
"duplicate" string special-case is gone.

Tests: conflict-skipped row not counted; a race yields no error; a
failing later chunk reports the rows the earlier chunks committed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBTRraVXkCnx93Pa9wxJ6G

---------

Co-authored-by: Jakob Wennberg <311770904+jakobwennberg-oss@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-09 12:24:29 +02:00

328 lines
13 KiB
TypeScript

/**
* Chart-of-accounts synchronization for SIE imports.
*
* Single home for the "ensure every mapped target account exists" logic that
* previously lived in three near-identical copies (executeSIEImport, the
* /api/import/sie/execute route, and the arcim-migration extension), plus the
* rename pass that carries customized #KONTO names from the SIE file into
* accounts that already exist (e.g. the K1-seeded defaults).
*/
import type { SupabaseClient } from '@supabase/supabase-js'
import { getBASReference, type BASReferenceAccount } from '@/lib/bookkeeping/bas-reference'
import { classifyAccount } from '@/lib/bookkeeping/account-classifier'
import { computeSRUCode } from '@/lib/bookkeeping/bas-data/sru-mapping'
import { fetchAllRows } from '@/lib/supabase/fetch-all'
import type { AccountMapping } from './types'
import {
defaultRateForVatTreatment,
isAccountVatTreatment,
isVatTreatmentAllowedForAccountClass,
type AccountVatTreatment,
} from '@/lib/vat/account-vat-treatment'
export interface AccountSyncResult {
/** Accounts inserted into chart_of_accounts */
created: number
/** Existing accounts whose name was updated from the SIE file */
renamed: number
/** Detail of each rename, for warnings/logging */
renamedAccounts: Array<{ accountNumber: string; from: string; to: string }>
/** Renames that failed (non-fatal: the import proceeds with old names) */
renameFailed: number
/** Fatal error from the create pass; null on success */
error: string | null
}
function emptyResult(): AccountSyncResult {
return { created: 0, renamed: 0, renamedAccounts: [], renameFailed: 0, error: null }
}
/**
* Rows per INSERT statement in the create pass.
*
* PostgREST runs every request under the authenticated role's 8 s
* statement_timeout, and each chart_of_accounts row fires four row-level
* triggers (audit log, writer-role guard, updated_at, known VAT rate) plus the
* RLS WITH CHECK. A full-BAS SIE import creates 1 200+ accounts in one go; on
* prod (2026-09-09) that single statement took 6.5 s for one company and was
* cancelled at 8.2 s for the next, so whether an import went through was a
* coin flip. 100 rows keeps every statement an order of magnitude inside the
* limit. Exported for the chunking test.
*/
export const INSERT_CHUNK_SIZE = 100
/**
* Build a chart_of_accounts insert row with the richest metadata available:
* BAS reference when the number is in BAS_REFERENCE (incl. description and
* k2_excluded), otherwise derived from the account number.
*/
function buildInsertRow(
accountNumber: string,
accountName: string,
basRef: BASReferenceAccount | undefined,
companyId: string,
userId: string,
vatDefaults?: { treatment: AccountVatTreatment | null; rate: number | null },
) {
const sortOrder = /^\d+$/.test(accountNumber) ? parseInt(accountNumber, 10) : null
if (basRef) {
return {
user_id: userId,
company_id: companyId,
account_number: accountNumber,
account_name: accountName,
account_class: basRef.account_class,
account_group: basRef.account_group,
account_type: basRef.account_type,
normal_balance: basRef.normal_balance,
sru_code: basRef.sru_code ?? computeSRUCode(accountNumber),
k2_excluded: basRef.k2_excluded,
plan_type: 'full_bas' as const,
is_active: true,
is_system_account: false,
description: basRef.description,
sort_order: sortOrder,
default_vat_treatment: vatDefaults?.treatment ?? null,
default_vat_rate: vatDefaults?.rate ?? null,
}
}
// Sub-account not in the BAS reference (e.g. 1932 Sparkonto). Derive
// metadata from the account number.
const classified = classifyAccount(accountNumber)
return {
user_id: userId,
company_id: companyId,
account_number: accountNumber,
account_name: accountName,
account_class: parseInt(accountNumber.charAt(0), 10),
account_group: accountNumber.substring(0, 2),
account_type: classified.account_type,
normal_balance: classified.normal_balance,
sru_code: computeSRUCode(accountNumber),
plan_type: 'full_bas' as const,
is_active: true,
is_system_account: false,
description: accountName,
sort_order: sortOrder,
default_vat_treatment: vatDefaults?.treatment ?? null,
default_vat_rate: vatDefaults?.rate ?? null,
}
}
/**
* Ensure every mapped target account exists in chart_of_accounts and,
* when `updateAccountNames` is true, carry the SIE file's #KONTO names into
* the chart.
*
* Name resolution: the file's name applies only to IDENTITY mappings
* (sourceAccount === targetAccount with a non-empty sourceName): when the
* user remaps a source to a different target, the file name describes the
* source account, not the target, so the target keeps its BAS/current name.
*
* - Create: account_name = file name (identity, flag on)
* ?? BAS reference name
* ?? targetName/sourceName fallback (non-BAS numbers)
* ?? `Konto ${number}`.
* - Rename (flag on only): existing accounts that are identity targets and
* whose stored name differs from the file name get a scoped UPDATE of
* account_name, and nothing else. Applies to is_system_account rows too
* (K1-seeded defaults); the flag itself is never touched. Equal names are
* a no-op, so replace-mode re-imports (Fortnox re-sync) are idempotent.
*
* When `updateAccountNames` is false the behavior matches the legacy code
* exactly: BAS defaults on create, existing accounts untouched.
*/
export async function syncMappedAccounts(
supabase: SupabaseClient,
companyId: string,
userId: string,
mappings: AccountMapping[],
updateAccountNames: boolean,
): Promise<AccountSyncResult> {
const result = emptyResult()
const targetAccounts = [...new Set(
mappings.filter((m) => m.targetAccount).map((m) => m.targetAccount)
)]
if (targetAccounts.length === 0) return result
// The SIE file's name for each identity-mapped account. Last write wins on
// duplicate #KONTO records (rare, benign).
const desiredNames = new Map<string, string>()
if (updateAccountNames) {
for (const m of mappings) {
const name = m.sourceName?.trim()
if (name && m.targetAccount && m.sourceAccount === m.targetAccount) {
desiredNames.set(m.targetAccount, name)
}
}
}
// Legacy create-time fallback for numbers outside the BAS reference.
const fallbackNames = new Map<string, string>()
for (const m of mappings) {
const fallback = m.targetName || m.sourceName
if (m.targetAccount && fallback) fallbackNames.set(m.targetAccount, fallback)
}
const vatDefaults = new Map<string, { treatment: AccountVatTreatment | null; rate: number | null }>()
for (const mapping of mappings) {
if (
!mapping.vatTreatmentReviewed ||
!mapping.targetAccount ||
mapping.sourceAccount !== mapping.targetAccount
) continue
if (
mapping.defaultVatTreatment !== null &&
mapping.defaultVatTreatment !== undefined &&
!isAccountVatTreatment(mapping.defaultVatTreatment)
) {
result.error = `Invalid VAT treatment for account ${mapping.sourceAccount}`
return result
}
const accountClass = Number(mapping.targetAccount.charAt(0))
if (
mapping.defaultVatTreatment &&
!isVatTreatmentAllowedForAccountClass(mapping.defaultVatTreatment, accountClass)
) {
result.error = `VAT treatment is not valid for account ${mapping.sourceAccount}`
return result
}
vatDefaults.set(mapping.targetAccount, {
treatment: mapping.defaultVatTreatment ?? null,
rate: mapping.defaultVatTreatment && mapping.defaultVatRate == null
? defaultRateForVatTreatment(mapping.defaultVatTreatment, accountClass)
: mapping.defaultVatRate ?? null,
})
}
// Fetch the company's chart once (paged) and filter in JS: avoids a huge
// .in() URL for full-chart imports and the silent 1000-row PostgREST cap.
let existingByNumber: Map<string, string>
try {
const targetSet = new Set(targetAccounts)
const allAccounts = await fetchAllRows<{ account_number: string; account_name: string }>(
({ from, to }) =>
supabase
.from('chart_of_accounts')
.select('account_number, account_name')
.eq('company_id', companyId)
// Stable total order on the unique account_number: paging is only
// correct with a deterministic order, else rows duplicate/skip across
// pages (see fetch-all.ts ordering invariant). The result is read into
// a Map below, so this order is invisible to callers.
.order('account_number', { ascending: true })
.range(from, to)
)
existingByNumber = new Map(
allAccounts
.filter((a) => targetSet.has(a.account_number))
.map((a) => [a.account_number, a.account_name])
)
} catch (err) {
result.error = err instanceof Error ? err.message : 'Failed to load chart of accounts'
return result
}
// Create pass: insert missing target accounts.
const missing = targetAccounts.filter((num) => !existingByNumber.has(num))
if (missing.length > 0) {
const inserts = missing.map((num) => {
const basRef = getBASReference(num)
const name =
desiredNames.get(num) ??
basRef?.account_name ??
fallbackNames.get(num) ??
`Konto ${num}`
return buildInsertRow(num, name, basRef, companyId, userId, vatDefaults.get(num))
})
// One statement per chunk (see INSERT_CHUNK_SIZE). A chunk that fails
// leaves the earlier ones committed, which is safe: the next attempt reads
// the chart again and only inserts what is still missing.
//
// ON CONFLICT (company_id, account_number) DO NOTHING: a concurrent import
// (or the replace flow) can create an account between our read and this
// write. With a plain INSERT that duplicate rolled back the whole
// statement while the caller treated "duplicate" as success, silently
// dropping every other account in it. Rows the conflict skipped are not
// returned, so `created` counts exactly what landed, chunk by chunk (the
// audit_log trigger on chart_of_accounts stays the per-row source of
// truth for behandlingshistoriken).
for (let i = 0; i < inserts.length; i += INSERT_CHUNK_SIZE) {
const chunk = inserts.slice(i, i + INSERT_CHUNK_SIZE)
const { data: insertedRows, error: insertError } = await supabase
.from('chart_of_accounts')
.upsert(chunk, { onConflict: 'company_id,account_number', ignoreDuplicates: true })
.select('account_number')
if (insertError) {
result.error = insertError.message
return result
}
result.created += insertedRows?.length ?? 0
}
}
const existingVatUpdates = [...vatDefaults]
.filter(([account]) => existingByNumber.has(account))
for (const [account, defaults] of existingVatUpdates) {
const { error: vatUpdateError } = await supabase
.from('chart_of_accounts')
.update({
default_vat_treatment: defaults.treatment,
default_vat_rate: defaults.rate,
})
.eq('company_id', companyId)
.eq('account_number', account)
if (vatUpdateError) {
result.error = vatUpdateError.message
return result
}
}
// Rename pass: carry the file's names into existing accounts. The diff set
// is small (only names that actually changed), so the UPDATEs run
// concurrently in bounded batches: a full-chart re-sync must not serialize
// N round trips, but also must not stampede the API with 1000+ in flight.
if (updateAccountNames) {
const renames: Array<{ num: string; from: string; to: string }> = []
for (const [num, currentName] of existingByNumber) {
const desired = desiredNames.get(num)
if (desired && desired !== currentName) {
renames.push({ num, from: currentName, to: desired })
}
}
const RENAME_BATCH_SIZE = 25
for (let i = 0; i < renames.length; i += RENAME_BATCH_SIZE) {
const batch = renames.slice(i, i + RENAME_BATCH_SIZE)
const outcomes = await Promise.allSettled(
batch.map(async ({ num, to }) => {
const { error: updateError } = await supabase
.from('chart_of_accounts')
.update({ account_name: to })
.eq('company_id', companyId)
.eq('account_number', num)
if (updateError) throw new Error(updateError.message)
})
)
outcomes.forEach((outcome, idx) => {
if (outcome.status === 'rejected') {
// Non-fatal: the import is still correct with the old name.
result.renameFailed++
return
}
result.renamed++
const { num, from, to } = batch[idx]
result.renamedAccounts.push({ accountNumber: num, from, to })
})
}
}
return result
}