Standard vs. Direct Load
OneStream offers two Workflow types for data ingestion: the Standard Workflow and the Direct Load Workflow. They achieve the same end result — data in the Cube — but take very different paths to get there. Choosing the right one depends on your data complexity, performance requirements, and auditability needs.
How They Differ
The fundamental difference is whether data passes through Stage tables on disk:
- Standard Workflow — Data is parsed, transformed, and stored in Stage tables. The user walks through Import → Validate → Load as discrete steps. Source and target records persist in the database.
- Direct Load Workflow — Data is parsed, transformed, validated, and loaded in a single in-memory operation. Source and target records are not written to Stage tables. Everything happens in one step called Load And Transform.
Feature Comparison
| Feature | Standard Workflow | Direct Load Workflow |
|---|---|---|
| Data path | File → Stage tables → Cube | File → memory → Cube |
| Source/target records | Stored in Stage — available for auditing and analysis | Not stored — no audit trail in Stage |
| Retransform | Supported — fix rules and re-apply without re-importing | Not available — must re-import the file |
| Drill-Back | Supported from Finance Engine to Stage | Not available |
| Validation error limit | No limit — pageable error list | 1,000 records per data load |
| Total error storage | Unlimited (persisted in tables) | 50,000 records (in-memory limit) |
| Workflow steps | Import → Validate → Load (separate tasks) | Single combined Load And Transform step |
| Performance | Slower — Stage table I/O adds overhead | Faster — no disk writes for staging |
| Stage persistence | Full source and target history retained | No Stage history |
| Data management | Source file import history available for reference | No historical file records |
| Time/Scenario constraints | Standard loading rules apply | Data must match the Workflow Scenario and Time — cannot load beyond current Workflow year |
When to Use Each
Standard Workflow — Best For
- Development and initial implementation — When you are building and refining Transformation Rules, Retransform and Drill-Back are invaluable
- Complex mappings — When the source-to-target relationship is not straightforward and requires iterative debugging
- Audit requirements — When you need to retain source file history and transformation history for compliance
- Low-to-moderate volume data — Where Stage I/O overhead is negligible
- First-time integrations — When you are connecting a new source system and expect mapping issues
Direct Load — Best For
- Production loads with stable mappings — When Transformation Rules are proven and rarely change
- High-volume nightly batch loads — Where optimal performance is critical and data is disposable (frequently deleted and reloaded)
- Tightly coupled metadata — When source system metadata mirrors OneStream metadata, allowing simple
* to *pass-through rules - Extended application data moves — Where data from a detailed application feeds a summarized target application
- Non-durable data — Data that has a high frequency of changes and may only be valid for a short time
The Recommended Development-to-Production Approach
The best practice is to use both types sequentially:
-
Start with a Standard Workflow during development. Build your Transformation Rules with full Retransform and Drill-Back support. Use the pageable validation to identify and fix every mapping error.
-
Once the core Transformation Rules are stable, create a production Direct Load Workflow. The stable mappings will pass through with minimal errors, and you get the performance benefit of in-memory processing.
-
Keep the Standard Workflow available as a debugging tool. When new source values appear or mappings break, switch back to the Standard Workflow to investigate, fix the rules, and then return to Direct Load.
Direct Load Limitations Summary
For quick reference, here are the key limitations of the Direct Load:
- No source/target record persistence — No Stage audit trail
- No Retransform — Source records are not stored, so there is nothing to retransform. Must re-import the file.
- No Drill-Back — Cannot drill from Finance Engine to Stage records
- 1,000 validation errors per load — If more than 1,000 errors exist, you must re-import to see the next batch
- 50,000 total error storage limit — After 50,000 cumulative errors, older errors are discarded
- Time/Scenario locked — Data records' Time and Scenario must match the Workflow Scenario and Time exactly
- No import history — No record of which files were loaded historically