Self-Hosted Event Collection (`/collect`) — Design

Status: Proposal. The /collect endpoint does not exist yet. hibot-cta-api itself is built and running — this document specifies a new controller inside it, plus the corresponding client, worker, and warehouse changes.

Problem

The chat widget’s only analytics path is Google Analytics 4, and it is conditional on the customer supplying their own GA4 measurement id.

Analytics.sendHostEvent (src/utils/Analytics.js:157) returns without doing anything when clientGa4Id is falsy. No id in loadSettings.googleAnalytics4Id, no event.
Analytics.sendFrameEvent (src/utils/Analytics.js:37) requires window.gtag. On the customer’s page that global only exists if registerGoogleTagManager already ran, which only happens via sendHostEvent. So when there is no customer GA4 id, frame events drop as well and only emit a Logger.warn.
Botsplash’s own tag (REACT_APP_GTM_ID_CHAT) is injected in public/frame.html and public/host.html. host.html is the port-9016 integration test page, so in production the Botsplash property only observes activity inside the iframe, never the customer page where host.js runs.

Consequences:

For any customer without a configured GA4 id, Botsplash collects nothing from host.js.
Even where GA4 is configured, gtag.js is blocked for a meaningful share of traffic by ad blockers and tracking prevention.
GA4 has no knowledge of visitorId, sessionId, chatboxId, routeId, or variantId, so it cannot answer questions the platform already generates the data for — most notably which A/B variant converted into a conversation.

Scope

/collect covers the widget funnel: load, open, chat start, agent connect, form submit, close. It is authoritative there.

It deliberately does not attempt site-wide page analytics. host.js loads only where the widget is enabled, after x.js bootstraps, and not at all on suppressed routes (webhookRoute.isSuppressed). Any site-wide pageview count derived from it will systematically undercount and will never reconcile with the customer’s GA numbers.

GA4 sending is retained. Customers want widget events in their own GA property; removing that is a downgrade for them. /collect is additive.

What already exists

Most of the ingestion path is built. The design reuses it rather than introducing a new pipeline.

Capability	Existing location	Reuse
Analytics queue	`constants.queue.publish.chatWidgetLoad` → exchange `ex.hibot.analytics`	Add a sibling `chatEventCollect` binding
Async consumer	`hibot-worker-analytics/src/notification.js:35`	Add a `collect` handler beside `ChatWidgetHandler`
IP address	`cta-api/src/utils/reqUtil.js` `getIpAddress` (CloudFlare → X-Forwarded-For → remote)	Server-side, no client trust
Geo	`reqUtil.getCountryCode` / `getStateCode` (CF header + MaxMind)	GA geo dimensions
Device / browser	`reqUtil.getDeviceInfo` (`ua-parser-js`)	GA device dimensions
Warehouse	`hibot-database/clickhouse` (`message_events`, `page_events_view`)	Report store
IP privacy	`globalFeatures.visitorIpPrivacyMode` (`cta-api/src/controllers/settings.js:411`)	Must be honored on this path too
Cache / rate limit	`cta-api/src/utils/appCache.js`	Batch dedupe, abuse throttling
Fingerprint	`getBrowserFingerprint()` (`src/providers/AppProvider.js:168`)	Stable `clientId`

The pageview is already collected

cta-api/src/controllers/settings.js:402-425 — behind the monitorChatWidgetLoad webhook feature — already publishes deviceInfo, ipAddress, fingerprint, chatboxId, channelId, webhookId, and pageUrl on every widget load, consumed by hibot-worker-analytics/src/handlers/chatWidget.js into WebhookIpLogDao.

That is effectively a first-party GA page_view hit, already queued and stored. /collect should not duplicate it. Its marginal value is the interaction events that follow the load.

Event model

The transport is deliberately application-agnostic. The chat widget is the first producer; bookme, directory, and the dialer are expected to follow, and the payload, endpoint, queue, and warehouse table are all shared.

Two discriminators carry that:

appType — which application produced the record.
recordType — what kind of record it is: event, info, error, metric, timing.

Only recordType = Event is populated in this phase. The other kinds are reserved in the schema and the client API now, so adding client-side error reporting later is a producer change with no endpoint, queue, or table migration.

Envelope

One request carries a batch. Context is sent once per visit, not per batch — the first request carries it, the server caches it against ctxId, and later requests carry only events:

{ "appId": "<publicKey>", "appType": 1, "ctxId": "m1a2b3c4d5e6",
  "events": [ { "recordType": 1, "eventType": 2, "ts": 1753970112345, "seq": 3,
                "params": { "elementText": "Chat with us" } } ] }

That is 166 bytes versus 678 for a one-event batch, a 75% reduction on every request after the first. Nothing is lost for reporting: the server merges the cached context before building the row, so every column is populated exactly as before.

The full envelope, sent on the first request of a visit and on the refresh cases below:

{
  "appId": "<publicKey>",
  "appType": 1,
  "appVersion": "20260731.1",
  "context": {
    "clientId":     "<browser fingerprint>",
    "sessionId":    "<widget session id>",
    "visitorId":    "<channel visitor id, when known>",
    "chatboxId":    "<uuid>",
    "routeId":      "<uuid>",
    "variantId":    "<uuid or null>",
    "entityId":     "<app-specific id, e.g. bookme meeting id>",
    "pageUrl":      "https://customer.example/loans",
    "pageTitle":    "Loans",
    "pageReferrer": "https://google.com/",
    "language":     "en",
    "screen":       { "w": 1512, "h": 982 },
    "tzOffset":     -240
  },
  "events": [
    { "recordType": 1, "eventType": 2, "ts": 1753970112345, "seq": 3,
      "params": { "elementText": "Chat with us" } }
  ]
}

eventType is numeric and scoped to appType. For the chat widget it is the existing FrameEventType id from src/constants/CtaEnum.js — no new taxonomy, the enum already has toString. bookme gets its own enum in the same numeric space under its own appType. The server resolves the readable eventName at ingest so reports are legible.
ts is client epoch ms. The server’s receive time is authoritative; the client value is kept only for intra-session ordering and engagement duration.
seq is a monotonic per-session counter, used to detect drops and to order records that share a millisecond.
params is a flat string map. Nested values are JSON-stringified by the client.

Context lives in its own table, not in Redis

Context is persisted to ClickHouse, not cached. There is no Redis read on the collect path and no correlation id on the client.

Table	Grain	Written by
`client_contexts`	One row per visitor per day	`/settings` on widget load, `/collect` when the client reports a change
`client_events`	One row per event	`/collect`

They join on (webhookId, eventDate, uniqueId). uniqueId is reproduced from request headers on every request, so the two tables stay in sync for the day without the browser holding or sending any correlation key — which is what removes the client-side bookkeeping.

/settings seeds the context. At widget load it already knows pageUrl, pageReferrer, chatboxId, channelId, routeId, variantId, device, geo, and the signature. So the common case is: context row already exists before the first event, and every /collect request is events-only.

The client sends context only when it changes — first send, setContext (the sessionId case), or a URL change. No ctxId, no refresh timer, no miss-recovery protocol. checkPageChanged compares window.location.href on every enqueue so a single-page app navigating between events re-sends rather than attributing the batch to the previous page.

client_contexts is a ReplacingMergeTree(insertedAt) keyed on (webhookId, eventDate, uniqueId), so both writers are idempotent upserts and neither needs to know whether a row already exists. The newest write wins.

Reporting

Report authors query client_events_view (or widget_events_view for the chat widget funnel), never the raw tables, so the join stays an implementation detail. The joined side uses client_contexts FINAL to collapse replaced rows.

client_events_daily rolls up from client_events alone, so the materialized view stays a pure insert trigger with no dependency on whether the context row arrived first. Funnel and daily-unique queries need no join at all; only dimensional cuts (variant, country, device) join client_contexts, which holds one row per visitor-day and is therefore small enough for ClickHouse to keep on the right-hand side of the join.

Trade-offs

Last write wins per visitor-day. A visitor who lands on page A and navigates to page B ends the day with page B’s context. Per-page journey analysis would need pageUrl on the event or a page key in both tables’ sort order; the current design trades that for a clean 1:1 join with no fan-out.
Events can outlive their context. If the /settings write and the client’s context send are both lost, events still land — uniqueId, eventType, sessionId, and timings are all present — but the joined view shows empty page/device columns. The upgrade script includes an unmatched-row check for exactly this.
A join replaces a wide flat read. That runs against the message_events precedent of full denormalization. It is justified here because the context is genuinely per-visitor-day rather than per-event, and duplicating ~40 columns onto every event row is the thing the payload reduction was trying to avoid in the first place.

Record kinds

`recordType`	Meaning	Populated now
1 `Event`	User or application event	Yes
2 `Info`	Informational client log	Reserved
3 `Error`	Client-side error with stack	Reserved
4 `Metric`	Named numeric measurement	Reserved
5 `Timing`	Duration measurement	Reserved

Applications

`appType`	Application
1	Chat widget (`hibot-chat-client`)
2	BookMe (`hibot-bookme-app`)
3	Directory (`hibot-directory-app`)
4	Dialer (`hibot-dialer-app`)

Events collected

Every FrameEventType reaches /collect, not only the subset sendHostEvent4 maps to GA. The funnel-relevant ones:

`FrameEventType`	Meaning	Funnel stage
`Init` (1)	Widget initialized on page	Load
`ChatWindowOpen` (2)	Visitor opened the widget	Open
`ChatWindowClose` (3)	Visitor closed the widget	—
`InitChat` (13)	Visitor started engaging	Engage
`AgentConnecting` (14)	Agent connect requested	Connect
`AgentConnected` (15)	Agent joined	Connected
`ChatFormSubmit` (16)	Form submitted	Convert
`CloseSession` (4)	Session closed or idled out	End

High-frequency types are excluded on both paths via getIsFrequentEvent: InitDragWidget (12) and ResizeChatFrame (11). They fire continuously during drag and resize and would dominate event volume.

Three types added to FrameEventType, raised from HostApp.showPopup:

`FrameEventType`	Raised when
`PopupOpen` (24)	Overlay is appended to the root element
`PopupClick` (25)	Visitor clicks the popup CTA image
`PopupClose` (26)	Visitor clicks the close button

All three carry the same payload, landing in eventParams:

{ popupId, popupName, popupUrl, urlMode, displayPosition, isMobileUI }

popupId and popupName come from the popups table; popupUrl is popup.ctaUrl. Since one popup is chosen at random from actionData.popupId, grouping open → click by popupId gives per-popup click-through, and by displayPosition / isMobileUI gives placement performance.

PopupClick on a same-tab CTA (urlMode not NEW_WINDOW) unloads the page immediately. Collect flushes on pagehide through sendBeacon, which is what keeps that event from being lost.

Where events are sent from

host.js and frame.js are separate bundles with separate module instances, so each has its own Collect. Both call startAnalytics, and exactly one sends per event:

Mode	Sender	Why
Iframe (normal)	`HostApp.onFrameEvent`	The frame forwards every event to the host over Postmate; `FrameApp`’s own `sendHostEvent` is guarded by `isTopWindow`, which is false here
Top window	`FrameApp.raiseFrameEvent`	No host wrapper exists, so `isTopWindow` is true
Popups	`HostApp.showPopup`	Popups are rendered host-side and never involve the frame

sendFrameEvent stays gtag-only. Routing it to Collect as well would double-count every frame event in iframe mode, because the same event already arrives at the host.

The host sets its own session context from the SaveSessionId frame event — without that, host-sent events would carry an empty sessionId, since FrameApp only populates the frame bundle’s instance.

GA4 field mapping

So that nothing available in GA is lost:

GA4 field	`/collect` source
`client_id`	`getBrowserFingerprint()`, client-supplied
`session_id`	Existing widget `sessionId`
`page_location`, `page_referrer`	`context`, same values `/settings` already receives
`page_title`	`context.pageTitle` (new)
`geo.country`, `geo.region`	`reqUtil.getCountryCode` / `getStateCode`, server-side
`device.`, `browser.`	`reqUtil.getDeviceInfo`, server-side
`traffic_source`, `campaign`	UTM parsed from `pageUrl` — new; `shortUrl.js` already has UTM parsing to borrow
`engagement_time_msec`	Client-side visible-time accumulator — new
`event_params`	Per-event `data` object

Client changes (`hibot-chat-client`)

Generic collector — `src/services/Collect.js`

Written. This is the reusable piece, and it is deliberately isolated so bookme and the other apps can adopt it unchanged:

Zero imports. No Environment, no Logger, no CtaEnum, no Preact. Everything it needs arrives through init(). Moving the file into a shared package is a file move with no edits.
No chat-widget vocabulary. It knows about appType, recordType, and numeric eventType; it does not know what event 2 means. That mapping stays in the caller.
Never throws into the host. Every send path is wrapped; a failed request, a missing sendBeacon, or a serialization error is swallowed. Telemetry must not break a customer’s page.
Bounded. MaxQueueSize (200) drops oldest records rather than growing without limit if the endpoint is unreachable. Strings truncate at 500 chars, stacks at 4000.
Disabled until configured. isEnabled() requires an endpoint, an appId, and an explicit enable; respectDoNotTrack honors navigator.doNotTrack.
Delivery. sendBeacon first so records survive unload, keepalive fetch as fallback. Flushes on a 2s timer, at 20 records, on visibilitychange → hidden, and on pagehide.

Public API:

Collect.init({ endpoint, appId, appType, appVersion, enabled, flushIntervalMs, respectDoNotTrack });
Collect.setContext({ sessionId, visitorId, chatboxId, routeId, variantId });
Collect.event(eventType, params);
Collect.info(message, params);
Collect.error(err, params);
Collect.metric(name, value, params);
Collect.flush();
Collect.shutdown();

The file carries no // @flow pragma on purpose — Flow is a hibot-chat-client convention, and omitting it keeps the module portable to apps that do not run Flow. Passes ESLint clean under this repo’s config.

Reuse path

Keep it in hibot-chat-client/src/services/ until a second consumer actually lands, then extract to a hibot-collect-js package. Extracting now would add a package, a build, and a release cycle to serve one caller. The constraints above are what make the later move free; the location is not what matters. For apps without a bundler it can also be built to the existing CDN (REACT_APP_CHATCDN_SERVER_URL) and loaded as a standalone script.

`Analytics.js` becomes a fan-out

Analytics.js keeps all chat-specific knowledge and becomes the adapter between FrameEventType and the generic collector. Buffering, batching, and delivery move into Collect.js, so the only changes here are:

sendHostEvent must no longer early-return when clientGa4Id is absent. It calls Collect.event unconditionally, and the GA4 path only when an id is configured.
Collect.init is called once with AppType.ChatWidget and CTA_SETTINGS_URL; Collect.setContext is called when the session, visitor, or variant becomes known.

import Collect from '../services/Collect';

const CollectAppType = 1;

function sendHostEvent(eventType: number, data: ?Object, clientGa4Id: string) {
  Collect.event(eventType, data);
  if (clientGa4Id) {
    registerGoogleTagManager(clientGa4Id);
    sendHostEvent4(eventType, data, clientGa4Id);
  }
}

No call sites change. HostApp.js:772 already invokes Analytics.sendHostEvent for every frame event, and FrameApp.js:1131 for every frame-originated one — both are unchanged.

Feature flag

The collect path is gated on a webhook feature so it can be ramped, mirroring how monitorChatWidgetLoad gates the existing load beacon. The flag arrives in the /settings response and is read off CtaSettings, alongside getGoogleAnalytics4Id().

Server implementation (`hibot-cta-api`)

Written. The endpoint writes to ClickHouse directly from the API rather than routing through RabbitMQ and hibot-worker-analytics. That removes the broker binding, the worker handler, and one hop of latency; the tradeoff is that buffered rows live in the API process and are lost if it is killed without a graceful shutdown. For telemetry that is an acceptable trade — it is not transactional data.

Files, in src/ (the repo has no lib/; dist/ is the Babel build output):

File	Role
`src/controllers/collect.js`	Endpoint, validation, server-side enrichment, row mapping
`src/utils/collectBuffer.js`	In-process batching and the ClickHouse `bulkInsert`
`src/constants/collectEnum.js`	`CollectAppType`, `CollectRecordType`, `CollectSeverity`, event-name resolution
`src/cta-api.js`	Buffer flush added to `gracefulShutdown`
`src/utils/reqUtil.js`	`getCountryCode` exported; `deviceType` added to `getDeviceInfo`; new `getDeviceInfoSafe`

Shared helpers

Coercion, URL, and UTM handling live in hibot-shared-lib, not in the controller.

Reused as-is: stringUtil.cleanValue, stringUtil.extractHostname, timeUtil.parseFlexibleDate (handles Date, epoch number, and string, which is exactly the clientAt case).

Added to hibot-shared-lib/lib/utils/stringUtil.js — pure string/number/URL work, no DB, so this is the correct layer:

Helper	Purpose
`toSafeString(value, maxLen)`	Always a string, never null — wraps `cleanValue` for non-nullable columns
`toBoundedInt(value, maxValue, defaultValue)`	Clamped non-negative integer for `UInt16`/`UInt32` columns
`toSafeFloat(value, defaultValue)`	Rejects `NaN` and `Infinity`
`isValidUuid` / `getValidUuid`	Anchored UUID check — the exported `UuidRegEx` is unanchored and accepts `junk/<uuid>`, so it is unsafe for validation
`getUrlParts(url)`	`{ host, path, query }`, falling back to `extractHostname` on a malformed URL
`getUtmParams(urlOrQuery)`	The five `utm_*` values; accepts a URL string or an already-parsed query object so the URL is not parsed twice

Added to hibot-shared-lib/lib/utils/objectUtil.js:

Helper	Purpose
`toStringMap(obj, options)`	Flattens an object to a bounded string map for `Map(String, String)` columns; stringifies nested values, drops null/undefined, caps keys and lengths

lib/utils/regex.js holds an anchored uuid regex but is dead code — nothing in hibot-shared-lib, hibot-model-lib, hibot-int-lib, or hibot-cta-api requires it, and its RFC-strict version/variant classes would reject otherwise-valid UUIDs. Left untouched.

Routes, registered automatically by sharedlib.loader.registerModules:

server.post('/collect', collectHandler);
server.post('/collect/:appId', collectHandler);

Why the request does not write directly

ClickHouse creates one part per insert and degrades badly under many small writes. A naive insert-per-request would be the single worst thing this endpoint could do to the warehouse. collectBuffer.js therefore batches in process:

Flushes at 500 rows or every 5 seconds, whichever comes first.
Uses async_insert: 1, wait_for_async_insert: 0, so ClickHouse also batches server-side and the insert call returns without waiting for the part to be written. Belt and braces: a process restart loses at most one buffer, a broker outage is no longer possible.
Caps the buffer at 20,000 rows and sheds load past that, emitting CtaCollectOverflow. An unreachable warehouse must not turn into unbounded memory growth in the API process.
The flush timer is unref()d so it never holds the process open.
A flushing guard prevents overlapping inserts; rows arriving mid-flush accumulate into the next batch.
gracefulShutdown flushes before exit, so a normal SIGTERM deploy loses nothing.

The handler itself never awaits the warehouse. CollectBuffer.add() is synchronous and the response returns immediately.

Validation and enrichment

appType and recordType are validated against the enums and unknown values are dropped rather than inserted — the shared-table blast-radius risk noted below.
Batch capped at 50 records; params capped at 25 keys; strings truncated (500 general, 2000 URLs, 4000 stacks).
Every non-nullable ClickHouse column gets '' or 0, never null. chatboxId, channelId, and webhookRouteId are UUID-validated and set to null when malformed, since they are the only Nullable(UUID) columns.
Geo (getCountryCode, getStateCode), device (getDeviceInfo), IP, and user agent are all resolved server-side. Nothing in that group is accepted from the client payload.
visitorIpPrivacyMode blanks ipAddress, matching settings.js:411.
UTM parameters and page host/path are parsed from pageUrl server-side.
WebhookCache.getIdByPublicKey with the cached flag — the same Redis lookup /settings already performs, not a new query.

CORS needs no work: utils/server.js already exports RestifyCors in dev and Restify otherwise, for every endpoint.

Response

asyncHandler always responds 200, so the endpoint returns { accepted, count } rather than a 202. Rejections return { accepted: false, reason } at 200 as well, matching how /settings reports { enabled: false, reason }.

Metrics (`hibot-shared-lib`)

Eight types added to lib/metrics/MetricType.js under hibot.cta.collect.* — accepted, dropped, overflow, inserted, insert_failed, and three error types — so collect failures never get conflated with /settings failures.

Gating

Deployment — constants.clickhouse.enabled (CLICKHOUSE_ENABLED=yes). Off means the endpoint returns { accepted: false, reason: 'collection disabled' }.

Per chatbox — the two destinations are independent, each with its own control on the chatbox Session tab. Neither affects the other, so an admin can run either alone, both, or neither.

Destination	Control	`loadSettings` key
Botsplash `/collect`	Disable Web Analytics checkbox — opt-out, must be checked to stop	`disableWebAnalytics`
Google Analytics	Google Analytics Tracking ID — empty means off	`googleAnalytics4Id`

Google Analytics needs no disable flag of its own: an unset tracking ID already is the off state, and that is the pre-existing behavior. Adding a second checkbox would create two ways to express the same thing.

Resolved in Analytics.js:

`disableWebAnalytics`	`googleAnalytics4Id`	`Collect.event`	gtag
false	set	✅	✅
true	set	❌	✅
false	empty	✅	❌
true	empty	❌	❌

disableWebAnalytics rides in the existing loadSettings JSON, so there is no migration. It is not a webhook feature column, so ramping is per chatbox rather than per account.

Cookieless visitor id

/settings returns a uniqueId — a per-day, cookieless visitor signature, echoed to the client for troubleshooting and stored on every client_events row.

uniqueId = SHA256( salt | utcDay | webhookId | ip | <stable client headers> )[0..32]

Inputs beyond site + IP + user agent

Input	Why
`salt`	`CTA_SIGNATURE_SALT`; without a server secret the hash is brute-forceable from a known IP/UA pair
`utcDay`	Rotates the id daily — see below
`accept-language`, `accept-encoding`	Stable per browser profile, cheap entropy
`sec-ch-ua`, `-mobile`, `-platform`, `-platform-version`, `-arch`, `-bitness`, `-model`	Client Hints; the highest-entropy stable signal on Chromium
`cf-ja4`, `cf-ja3-hash`	TLS fingerprints, the strongest available signal — a client’s TLS stack is stable and hard to spoof. Cloudflare only emits these on Bot Management / Enterprise; the code includes them when present and degrades silently when not
`cf-device-type`, `cf-ipcountry`	Low entropy, stable, free from the existing Cloudflare edge

Deliberately excluded: cf-ray (unique per request), and accept / sec-fetch-* (vary by request mode, so a fetch and a sendBeacon from the same browser would hash differently and split one visitor into two).

Why the day is inside the hash

The identifier is stable within a day and uncorrelatable across days. That is what keeps it a pseudonymous analytics key rather than a persistent tracking identifier, and it matches how uniqueId is meant to be counted — uniq(uniqueId) grouped by eventDate is daily unique visitors. Summing across days counts a returning visitor once per day, by design.

The client value is not trusted

/collect recomputes the signature from the request rather than reading context.uniqueId, which a client can forge. The echoed value is only compared, and a mismatch is logged for troubleshooting. Same algorithm, same inputs, so the two agree unless the visitor’s IP changed mid-session.

Accuracy limits

Shared NAT with identical browsers collides; mobile carrier IP rotation splits one visitor into several. It is materially better than nothing and materially worse than a cookie. Where cf-ja4 is available the collision rate drops sharply, which is the main argument for enabling Cloudflare Bot Management on the widget hostname.

CTA_SIGNATURE_SALT is unset by default and falls back to a constant. Set it before relying on the id — the fallback makes the hash reproducible by anyone who knows the scheme.

Warehouse schema (`hibot-database`)

Written to clickhouse/schema/dw_client_events.sql. One wide table client_events, sited next to message_events so the funnel can be joined to actual conversations on sessionId — the join GA4 cannot do.

Why one table and not one per application

ClickHouse is columnar: unused columns in a row cost close to nothing on disk, and a query that does not select a column does not read it. A wide shared table means a new application or a new record kind is an INSERT with different discriminator values, not a migration, a new consumer branch, and a new set of report queries. message_events already follows this shape — it carries campaign, task, and team columns that are empty for most rows.

Per-application and per-purpose readability is restored with views, following the page_events_view precedent rather than by splitting storage.

Column groups

Group	Columns	Notes
Discriminators	`recordType`, `appType`, `severity`, `eventType`, `eventName`, `appVersion`	`eventType` is numeric and scoped to `appType`; `eventName` is resolved server-side for legibility
Identity	`webhookId`, `clientId`, `sessionId`, `visitorId`, `batchId`, `seq`	`batchId` and `seq` make duplicate and dropped batches detectable
App entities	`chatboxId`, `channelId`, `webhookRouteId`, `variantId`, `entityId`	`Nullable(UUID)` since they are chat-specific; `entityId` is the generic slot for other apps (e.g. a bookme meeting id)
Page	`pageUrl`, `pageHost`, `pagePath`, `pageTitle`, `pageReferrer`, `referrerHost`	Host and path split at ingest so grouping does not require repeated string parsing
Acquisition	`utmSource`, `utmMedium`, `utmCampaign`, `utmTerm`, `utmContent`	Parsed from `pageUrl` server-side
Geo	`ipAddress`, `countryCode`, `regionCode`, `city`, `tzName`	Server-derived; `ipAddress` empty under `visitorIpPrivacyMode`
Device	`deviceType`, `browserName`, `browserVersion`, `osName`, `osVersion`, `language`, `screenWidth`, `screenHeight`, `userAgent`	Server-derived from the UA
Measures	`engagementMs`, `durationMs`, `metricValue`	Reserved for `Metric` / `Timing` records
Diagnostics	`message`, `errorName`, `errorStack`, `sourceFile`, `sourceLine`	Reserved for `Info` / `Error` records
Flexible	`eventParams Map(LowCardinality(String), String)`	GA4’s `event_params` shape
Time	`createdAt`, `clientAt`, `insertedAt`	`createdAt` is the server receive time and is authoritative

Engine and key choices

ENGINE = MergeTree()
PARTITION BY toYYYYMM(createdAt)
ORDER BY (webhookId, appType, recordType, createdAt)
TTL createdAt + INTERVAL 25 MONTH;

ORDER BY leads with webhookId, not createdAt. Every report filters to one webhook first. This differs from message_events, which uses (createdAt, webhookId) — that table is scanned across tenants for platform-wide reporting, client_events is not.
PARTITION BY month gives cheap date pruning and makes retention a partition drop.
TTL at 25 months bounds retention, which the privacy section requires. message_events has no TTL; an unbounded raw event stream is a different risk profile.
LowCardinality(String) on every repeating string — browser, OS, country, UTM source. On a table where these repeat across millions of rows this is the single largest storage and group-by win.
Map over JSON for eventParams. The repo has JSON precedent (extendedAttributes), but it needs allow_experimental_object_type and infers a column per key. Map is stable, needs no flag, and eventParams['formName'] is the natural access for a flat GA-style parameter bag. Nested values are stringified client-side.
Nullable only where genuinely optional. ORDER BY columns cannot be Nullable, so webhookId, appType, recordType, and createdAt are required — the ingest handler must reject records missing them rather than defaulting.

Views and pre-aggregation

widget_events_view — filtered to appType = 1 AND recordType = 1, with a funnelStage label derived from eventType. This is what Metabase points at for widget reporting.
client_events_daily (AggregatingMergeTree) plus client_events_daily_mv — daily rollup keyed by webhook, app, event, variant, country, and device, holding uniqState for sessions and clients. Reports read the rollup; the raw table stays for drill-down. This is what makes “reports generated server side” cheap rather than a full scan per dashboard load.

Both funnel and variant-conversion queries are included as comments at the bottom of the schema file.

Enums (`hibot-model-lib/lib/enums`)

Three new enums, following the WebhookLogSource style, so appType / recordType / severity are not bare integers across the API and worker:

CollectAppType.js — ChatWidget: 1, BookMe: 2, Directory: 3, Dialer: 4
CollectRecordType.js — Event: 1, Info: 2, Error: 3, Metric: 4, Timing: 5
CollectSeverity.js — Debug: 1, Info: 2, Warn: 3, Error: 4, Fatal: 5

Deployment

New tables ship as a numbered upgrade script under clickhouse/upgrades/, following the existing $ch_client --query= pattern. The next free number is 022.

Reporting

Delivered through Metabase against ClickHouse, following clickhouse/queries/20211103_metabase_queries.sql. No report UI is built. Rebuilding GA’s reporting surface is months of work and is not the goal.

Primary reports:

Widget funnel by webhook and date: load → open → engage → agent connect → convert.
A/B variant conversion, grouped by variantId — not answerable today in any system.
Funnel by route, chatbox, device, and geo.

Risks

Volume. /settings is one request per page load; /collect is 10–50x that. Batching, the 202-and-queue handler, and the per-webhook flag keep this controllable, but the RabbitMQ consumer and ClickHouse ingest need sizing before general rollout.
Reporting cost dominates. Ingestion is days of work. Anything resembling a GA report clone is not. Holding the line on “Metabase against ClickHouse” is what keeps this cheap.
Undercount versus GA. Even scoped to the widget funnel, /collect numbers will differ from the customer’s GA numbers, because GA loses blocked traffic that /collect captures. Expect the discrepancy to be reported as a bug; document the direction of the difference.
Privacy. Self-hosting improves the GDPR/CCPA position, but a fingerprint-derived clientId is the element most likely to require a consent story. visitorIpPrivacyMode must be honored; retention on client_events is bounded by the 25-month TTL.
Client payload trust. Everything in context originates in the browser. Geo, device, and IP are resolved server-side and must not be accepted from the client.
Shared table, shared blast radius. One table for every application means a malformed producer can degrade reporting for all of them. The handler validates appType and recordType against the enums and drops unknown values rather than inserting them.
Buffered rows are not durable. Dropping the queue in favour of a direct insert means up to 5 seconds or 500 rows live only in the API process. SIGTERM flushes; SIGKILL, an OOM, or a crash loses them. Acceptable for telemetry, and the reason this pattern must not be reused for anything transactional.
Row mapping is unverified. The Map, Nullable(UUID), and DateTime64(3) columns are the likely failure points on first real insert. bulkInsert failures are caught and logged as CtaCollectInsertFailed rather than surfacing to the client, so a mapping bug will look like silence, not an error. Watch that metric on first deploy.
visitorId is always 0 for the chat widget. The widget is never told its visitor id — there is no visitorId anywhere in hibot-chat-client — so the column stays empty. It can only be populated by joining client_events.sessionId to message_events in the warehouse. Resolving it per request would mean a DB lookup on the hot path, which this endpoint deliberately avoids.
Signature salt defaults to a constant. CTA_SIGNATURE_SALT is unset out of the box. Until it is set per environment, uniqueId is reproducible by anyone who knows the inputs.

Status

Artifact	State
`hibot-chat-client/src/services/Collect.js`	Written, lints clean
`hibot-database/clickhouse/schema/dw_client_events.sql`	Written, not run against ClickHouse
`hibot-cta-api/src/controllers/collect.js`	Written, lints and transpiles clean
`hibot-cta-api/src/utils/collectBuffer.js`	Written
`hibot-cta-api/src/constants/collectEnum.js`	Written
`hibot-shared-lib` collect metric types, coercion/URL/UTM helpers	Written
`hibot-shared-lib` `aesUtil.createHash256Hex`, `cta.signatureSalt`	Written
`hibot-cta-api` cookieless `getVisitorSignature` + `/settings` `uniqueId`	Written
ClickHouse upgrade script `022`	Written
`hibot-web-app` Disable Web Analytics checkbox	Written
`Analytics.js` fan-out + `Collect` wiring	Written
Tests	Not started
`client_events.visitorId` backfill	Not started — see below

Nothing here has been executed end to end. The schema has not been applied to a ClickHouse instance and no request has been posted to the endpoint, so the row mapping is unverified against real column types — see Risks.

Phasing

Ingestion. Queue binding, enums, /collect controller, worker handler, ClickHouse upgrade script 022. Behind the feature flag, GA4 untouched. Verifiable end to end with no client changes by posting synthetic batches.
Client. Wire Collect.js into Analytics.js, and fix the case where events are lost because no GA4 id is configured.
Reporting. Metabase funnel dashboard; variant conversion joined to message_events.
Documentation. Rewrite docs/analytics.md — see below.
Second producer. Adopt Collect.js in hibot-bookme-app under appType = 2; extract to a shared package at that point.

docs/analytics.md is currently wrong independent of this proposal. It documents the Universal Analytics taxonomy (Web Chat category, Chat Window Open action), while src/utils/Analytics.js emits GA4/ASC event names (botsplash_chat, botsplash_chat_cta, botsplash_chat_engagement, asc_cta_interaction, asc_comm_engagement, asc_form_submission). It should be corrected regardless of whether /collect is built.

Self-Hosted Event Collection (/collect) — Design

Problem

Scope

What already exists

The pageview is already collected

Event model

Envelope

Context lives in its own table, not in Redis

Reporting

Trade-offs

Record kinds

Applications

Events collected

Popup events

Where events are sent from

GA4 field mapping

Client changes (hibot-chat-client)

Generic collector — src/services/Collect.js

Reuse path

Analytics.js becomes a fan-out

Feature flag

Server implementation (hibot-cta-api)

Shared helpers

Why the request does not write directly

Validation and enrichment

Response

Metrics (hibot-shared-lib)

Gating

Cookieless visitor id

Inputs beyond site + IP + user agent

Why the day is inside the hash

The client value is not trusted

Accuracy limits

Warehouse schema (hibot-database)

Why one table and not one per application

Column groups

Engine and key choices

Views and pre-aggregation

Enums (hibot-model-lib/lib/enums)

Deployment

Reporting

Risks

Status

Phasing

Related correction

Self-Hosted Event Collection (`/collect`) — Design

Client changes (`hibot-chat-client`)

Generic collector — `src/services/Collect.js`

`Analytics.js` becomes a fan-out

Server implementation (`hibot-cta-api`)

Metrics (`hibot-shared-lib`)

Warehouse schema (`hibot-database`)

Enums (`hibot-model-lib/lib/enums`)