Data lineage: trace a business number to its inputs

A source link needs the transformation and run that produced the reported value, plus a declared boundary where the trace stops.

Kartik Deshpande··6 min read

A report shows 110 shipped pieces, and the next refresh shows 90. Data lineage should let you trace each result to the inputs, transformation, and execution that produced it. Start with the output and follow the recorded dependencies backward; then follow a changed input forward to identify which reports or applications need review. State where the trace stops if a step was not captured.

Google Cloud’s data lineage documentation organizes lineage into a process definition, a run of that process, and events recording movement between sources and targets. That distinction matters when the same report is refreshed more than once: a current dependency diagram alone cannot establish which source state produced yesterday’s number.

Choose the question and required level of evidence

For a schema change, you may need to know which tables, fields, and downstream consumers depend on a source column. For a disputed business total, you also need the selected population, inclusion rule, units, and input versions. State the question before selecting coverage. A table-level connection can support dependency review without explaining every contributing record.

Separate the declared path from observed execution. A design can show that shipment and return tables feed an output even if a failed run never read returns. OpenLineage’s object model distinguishes runtime RunEvents from design-time JobEvents and DatasetEvents. Use execution evidence to establish what a particular run observed, and keep declared dependencies available for planning changes.

Record the output identifier, run reference, source identifiers, transformation version, and relevant event times. Add retained snapshots or version references when reproducing the historical value requires them. A stable link to an editable spreadsheet (whose cells may have changed) gives access to its present contents; it does not establish the cells that an earlier calculation used.

Distinguish lineage from data mapping. Mapping defines how source fields and meanings become destination fields. Lineage connects the input and transformation evidence to the resulting output, including the execution where available. Keep both so a reviewer can inspect the rule and determine whether that version contributed to the disputed result.

Reconstruct a changed unit conversion

Assume a hypothetical report of net shipped pieces for one order line. Its source shipment is 10 cases and its linked return is 10 pieces. The approved conversion for the shipment’s item revision is 12 pieces per case, and the report subtracts the return. There are no other records, filters, rounding rules, or stock movements in this example. The reported result is 10 times 12 minus 10, or 110 pieces.

EvidenceOriginal runChanged conversion run
Shipment input10 cases, shipment S10 cases, shipment S
Return input10 pieces, return R10 pieces, return R
Conversion used12 pieces per case, version A10 pieces per case, version B
Calculation10 × 12 − 1010 × 10 − 10
Reported output110 pieces90 pieces

Assume a later packaging revision uses 10 pieces per case. If a new run applies that conversion to the old shipment, its output becomes 90, a decrease of 20. The lineage needs the conversion version as well as shipment S and return R to explain the change. The shared source links are identical in both runs, so citing those records without the applied rule would leave the discrepancy unresolved.

Have the definition owner decide whether the older shipment should retain version A or be corrected. Preserve the reason, effective period, prior output, and corrected output. If version B is valid only for new shipments, repair the transformation selection and rerun affected outputs under approval. Do not call the changed result a source correction when the source shipment and return did not change.

Trace the output through the applied rule
Inputs
Source IDs and retained versions
Execution
Rule version and run evidence
Output
Reported value and consumers
The rule version can explain a changed result even when the source records remain the same.

Inspect the evidence behind a disputed value

Bring permitted source extracts and the calculation definition into Permute to check their identities and business meaning.

Follow a correction to the outputs that depend on it

Use forward dependencies to scope the review before making a change. A carton conversion may feed shipped-unit reporting, inventory movements, and a customer-facing application. Identify which consumers use that conversion and which period or item revision is affected. Treat a dependency as a candidate impact until you check whether the consumer used the changed field and population.

Distinguish dependencies that change values from dependencies that select rows. A customer table might determine which orders qualify for a segment without supplying the amount field. If a customer classification changes, the report total may change even though every amount is untouched. Include filter and join effects in the review when those operations can alter the answer.

Retain acceptance evidence after recomputation. Data reconciliation checks whether the output agrees with sources under declared rules; lineage identifies the path and versions that need checking. A traceable calculation can still apply an incorrect rule. Run the relevant population and field checks after the repair instead of treating a complete graph as proof of correctness.

Assign the affected output to an owner who can decide whether to replace, annotate, or withdraw it. Preserve the earlier version when someone has already used it for a decision. An updated dashboard does not establish that an exported spreadsheet or previously delivered report was also corrected. Track those consumers within the actual coverage of the workflow.

Make gaps and historical limits visible

Build a coverage inventory for the chosen question. Mark which source systems, transformations, reports, and manual steps emit or retain evidence. A copied spreadsheet, hand-entered adjustment, or unsupported transformation may break the observed path. Show that boundary as unknown or declared-only instead of drawing an inferred connection that looks observed.

Test a missing run event, a failed run, and a manual correction. Ask the owner to distinguish absent metadata from proof that no dependency exists. A pipeline can produce an output while its lineage capture fails; the output’s existence alone does not establish capture completeness. Keep a way to report that coverage failure beside the output.

Specify retention around the decisions the organization needs to revisit. If source versions expire before the review window, retained run identifiers may identify the calculation without allowing reconstruction. Record what remains available: the original values, a source version reference, a calculation receipt, or only a dependency. Do not describe those levels as equivalent historical evidence.

A source citation in an AI answer needs the same scrutiny. Check whether it supports the stated population and calculation, whether it reflects the version used, and whether the user may open it. A valid URL can be attached to a wrong result, so validate the supporting relationship. Keep unsupported inferences labeled and distinguish them from transformed source values.

Where Permute preserves business evidence

Permute is the context layer we sell between business data and the AI tools people already use. We connect permitted records and reconcile identities across sources so an output can retain the evidence behind its inputs. Claude, ChatGPT, and Copilot can use governed business data without needing the reader to rebuild those relationships in each question.

We can reuse declared definitions and transformation rules across reports, applications, and scoped answers. An operations owner could inspect which source record and unit definition explain a result, subject to current permissions. Confirm the available history and trace coverage for the workflow before depending on reconstruction. How the data is handled describes the access boundary around that evidence.

Set the trace boundary without implying complete coverage

We do not claim automatic lineage across every external tool or transformation. Missing input versions, inaccessible records, and unrecorded manual steps limit the path that can be shown. This analytical workflow reads connected systems and does not repair source records or downstream applications by itself. Business owners approve definitions and corrections; a lineage relationship does not authorize access to its underlying data.

Retain the path needed to explain the result

Data lineage connects an output with the inputs, rules, and execution evidence that explain how it was produced. Choose coverage around the question, preserve the relevant versions, and separate observed runs from declared dependencies. The worked example changes because the conversion changes, even though the source record links stay the same. Follow a correction to its affected consumers and verify the replacement output under the approved rule.

Review a number whose history cannot be explained

Bring the output, available source versions, and transformation definition to establish which trace boundaries need attention.

Questions about data lineage

Does lineage prove that a number is correct?

It helps identify how the number was produced. Correctness also requires valid inputs, accepted business rules, and checks that those rules were applied to the intended population. A complete trace can expose an incorrect transformation without making its result correct.

Do all decisions require record-level lineage?

No. A table or column dependency can be enough to scope a schema change, while a disputed customer amount may need contributing records and rule versions. State the required level and the coverage that was observed. Avoid claiming record-level evidence from a table relationship alone.

Should everyone who sees a result see its source data?

Access to an output does not automatically authorize access to every underlying record. Evaluate source and output permissions under the organization’s rules. Where access is restricted, provide only the permitted explanation or route the investigation to an authorized owner.