When insurance applications keep separate copies of premiums, claims, and contracts, teams repeat the work of explaining differences and correcting records in several places.
A producer/consumer design gives a team responsibility for publishing data that others can use. The publisher owns its meaning, quality, and updates. This article describes that design on Snowflake.
What a data product contains
A data product is a maintained set of information with an owner and an agreement about its contents. Examples include premium transactions, claims, and commission calculations.
Specify the fields, meanings, validation rules, update schedule, and permitted uses. Preserve source references. Define how corrections are made and how changes are communicated to consumers.
For bordereaux, the periodic premium and claims statements exchanged with insurers, start with the outputs finance and operations need. Work backward to the records and checks required to produce them.
Separating source history from business rules
The design can use these layers:
- Landing: Retain received files and messages so the team can examine original inputs and repeat processing. Set retention and access rules.
- Raw Vault: Integrate source history using Data Vault structures. Hubs hold business keys, Links record relationships, and Satellites hold descriptive information and its history.
- Business Vault: Apply business rules and calculations while retaining the connection to source history.
- Data products: Publish tables or views with definitions, validation results, and an owner.
Use these distinctions where they help manage history, changes, and review. Keep the design as simple as the requirements allow.
Who owns the data
Finance, underwriting, claims, and operations understand different parts of the business. Publishing teams own the meaning and quality of their data. The technology team provides shared ingestion, storage, access, testing, and monitoring.
A team can both consume and publish data. Finance may use premium and claims records, apply accounting rules, and publish information for reporting. Decide who approves rule changes and settles disagreements between teams.
Choosing integrations
Evaluate connectors against the required records, update frequency, error handling, and access controls. Some sources require custom code for a document format, validation rule, or business process.
Before building another integration, check whether an existing data product provides the records. Define any difference before extending it or creating another product.
Record what happens when a file is late, a field is missing, or processing fails. Assign responsibility for fixing the problem and notifying consumers.
Consistent, traceable records
Where related processes need the same version, publish a dated snapshot and make that version visible. Decide how corrections affect completed reports and calculations.
Reviewers should be able to trace a published value through its calculation to the source. Retain source references and transformations, and give reviewers access to the evidence.
Measuring the change
Establish how long records take to reach users, how many need correction, and how much staff time the work uses. Include integration maintenance and investigation costs. Compare those measures after implementation, keeping targets separate from observed results.