KCL Proxy

KCL Proxy serves as a bridge between HTTP-based systems and KCL. This service should be used in scenarios when:

• Producer service is deployed outside of InCrowd infrastructure and doesn't have direct access to FanX Message Broker - external integrations and third-party data providers.
• Producer project is built in language other than Go and has no KCL wrapper available.

Event Transformation

If External Provider requires any event transformation and/or enrichment to be implemented on InCrowd side, KCL Proxy should be forked to a separate project. Please refer to KCL Proxy README.md for more context.

Features

• Cross-Language Compatibility: Eliminates the need for language-specific KCL wrapper implementations by offering a language-agnostic HTTP interface.
• Simplified Integration: Abstracts away the intricacies of FanX Message Broker integration, reducing the effort required to incorporate event-driven functionality into projects outside of FanX product ownership.
• Reliability: Automatic scaling based on incoming traffic, ability to handle data surges and temporary network outages.

Events

Priority

KCL Proxy supports optional priority event parameter that can be provided to boost Rules Engine performance for high-volume clients.

If priority is specified, KCL Proxy will choose the proper batching strategy:

• LowPriority ([0:10)) - event will be agressively batched and can be dropped if internal buffer is being overflowing. This priority level should be used only for informational events that are used for FDP.
• NormalPriority ([10:20)) - event will be buffered but is guaranteed to be produced as soon as batch is filled (100 events buffered) or flush timeout is hit (2000ms). Normal priority level should be used for non time sensitive events.
• HighPriority ([20:30)) - similar to NormalPriority level but has time-sensitive batching. Usually will take less than 500ms before being produced to FanX Message Broker.
• ImmediatePriority ([30:∞)) - event will be produced as soon as it is received and parsed. Should be used for events that require very low latencies (< 100ms).

info

Please note that if UserEvent event is used, ImmediatePriority does not guarantee that target event will be executed before others - Rules Engine implements separate priority queue that can result immediate-priority event being buffered in some cases. Please refer to Rules Engine documentation for more context.

Request ID

KCL libarary implement retry mechanism that ensures that in case of temporary errors (network connection loss)

By default, KCL library will mark message as sent only after all FanX Message Broker instances will acknowledge receiving it. This means, that if KCL Proxy returns no error for ImmediatePriority event, it is guaranteed to be produced. Optional requestId event parameter can be passed to /events bulk enpoint if producer service implements retry mechanisms.

If requestId is provided and KCL was unable to produce the event (non-retryable errors only), underlying error will be retured in the HTTP response body alongside with the requestId. As priority levels other than ImmediatePriority have batching enabled, no errors will be provided in HTTP response, webhook URL should be specied via http header - KCL Proxy will call provided webhook URL with for each message that it failed to produce.

Third Party Integrations

We do not recommend building complex retry mechanisms on top of KCL Proxy, if that is really required, client should be granted with direct access to FanX Message Broker.

If client is not able to utilise Message Broker directly, dedicated Kafka or SQS queue can be provided for client to consume (DLQ-like approach). Please note that FanX Message Broker uses protobuf for all event types, so external service should be built to decode internal event structure back to the client-defined one.

UserEvent Record

Structure

The UserEvent record represents an event that is used by the Rules Engine. It should always contain the following fields:

Field	Type	Description
requestId	String?	The unique event identifier that is passed back to the producer if event can't be produced.
createdAt	DateTime?	The timestamp when the event was created. Will be set to receivedAt timestamp if not specified.
sourceSystem	String	Event source system.
sourceSystemId	String	Event type (unique per source system).
priority	Integer?	Event priority (NormalPriority if not specified).
authId	Integer	Target user id.
data	Object	Data associated with the event.

? - optional field

Authorization

KCL Proxy should be used only by server-side applications to ensure that generated credentials could not be used by any unauthorized party. Please refer to Client-side Events for more information on how to publish events from client-side applications.

Please note that API Tokens used by KCL Proxy are limited to:

• Specific ClientID (e.g. INCROWD, ICPRODUCT, ...)
• Specific topics (e.g. rules-engine.raw.INCROWD, ...)

The following headers should be passed with each KCL Proxy request:

• X-REALM
• X-API-KEY
• X-APP-ID
• KCL-TOPIC