How We Build Ventana Models

Ventana® models are usually built through a process of rapid evolution with client participation. Improvements in modeling technology now allow major model enhancement and testing in hours to days, so the client can see and critique progress frequently and regularly.
Rapid evolution with frequent client feedback brings important project benefits:

• The client can regularly inspect inputs, outputs, and detail granularity of the model to ensure relevance to the target decisions.

• The client can contribute knowledge toward ongoing tests of the model framework, improving reliability.

• Client participation in the model evolution develops familiarity and expertise, which are the basis of model credibility.

• As client personnel become familiar with possible uses for the model, they can direct choices of data management, interface, and business process for best fit.

For more information on how Ventana's modeling process meets these objectives, please see Why Ventana Models Work. Client Resources

Generally, two groups greatly benefit from regular participation: the decision team, and the model team.
• The decision team has direct responsibility for making the decisions supported by the model. These people typically participate in half-day meetings to guide model development once every 3 to 6 weeks over a 3 to 5 month period. Clients often feel these meetings, which focus on clarifying the outcomes to be measured and on the nature and requirements of decision-making, are extremely useful in and of themselves, independent of their contribution to model development.

• The model team has direct responsibility for model development and its implementation in the organization. During model development, these people work closely with Ventana to develop thorough knowledge of model capabilities, identify and provide relevant data, arrange meetings with stakeholders and knowledge resources, find answers to daily questions, and contribute their own expertise and critique. After delivery, the model team takes the lead to ensure the decision team gets full benefit of the tool.

Sometimes, the two teams are the same people. Sometimes a "team" is one person. In every case, however, the people in these roles develop a clearer sense of their business and their opportunities.
In addition to these people, Ventana invites to interviews or meetings client personnel who can provide useful information or data, and all other stakeholders in the model or the decisions it supports. These interviews and meetings, organized by the model team, not only ensure that the model profits from the broadest range of client knowledge, but also foster organization-wide involvement in and ownership of the resulting business intelligence.

Project Elements

Five elements work together to keep the delivered model up to date and connected to ongoing decisions:
• Knowledge - a documented and tested collection of client expertise on how each factor will affect other factors
• Data - a reviewed collection of historical data, and a system to manage ongoing data to connect to the model
• Model - the system of equations used to calculate the outcomes associated with decision options
• Business Process - The job descriptions, schedules, and assigned responsibilities by which the model gets used to inform decisions
• Interface & Training - The system connecting the model to human users or to automated decision systems, and any required training

Project Sequence

In a typical project, the five project elements - Knowledge, Data, Model, Business Process, and Interface & Training - proceed in parallel by iteration and critique. A specific plan for each project is drafted at project start, with milestones evaluated by the client at each review. The figure represents a typical plan.



Step 1: Project Start
A facilitated workshop with the decision team and the model team to introduce the project and complete the project definition

Agenda:
• Introduce decision team and model team to the project objectives and project plan.
• Begin work on project elements:
° Knowledge: Collect initial expectations on cause-and-effect mechanisms active between actions and outcomes.
° Data: Identify and discuss relevant data sources.
° Model: Develop first-pass structure, focusing on decisions to be made and outcomes to be managed.
° Business Process: Determine who will maintain and run the model, and how questions of and insights from using the model will be communicated.
° Interface & Training: Make a first assessment of requirements to support the business process design.
For more information on possibilities for knowledge and data management, business process, interface, and training, please see Using Ventana Models.

Step 2: Collect Information
In collaboration with the model team, begin shaping all five project elements: Knowledge: Initial collection
• Conduct one-on-one or group interviews of people with direct experience of how things work and why things happen the way they do, possibly including the decision team.
• Begin recording expertise in Reality Check(r) libraries.

Data: Initial data examination
• Collect relevant data.
• Interview people with expertise on how and why data were collected and what potential problems they may contain.
• Begin "cleaning" process:
° Aggregate to level of detail appropriate to modeling goals.
° Compare different data sets to identify and understand inconsistencies.
° Understand and document likely biases and uncertainty in measurements.
° Prioritize unmet data needs to pursue.
Model: Begin forming rough draft of model equations from initial descriptions of decisions and outcomes to be managed.

Business Process: Initiate further conversations as required to establish business process for model use.

Interface & Training: Begin interface development and draft initial plan for training.

Step 3: Enhance System
Prepare a working draft of all elements for client review. Knowledge: Track down and clarify any conflicts with data or other inconsistencies.

Data: Initialize a rough system for management and use of the data collected to date. Refine the data set and the data management system in subsequent revisions.

Model: Build a working prototype, including:
Decision levers
• Outcomes / outputs
• All major relevant aspects of the business situation.
In subsequent revisions, increase fidelity and robustness. Business Process: Develop conversations and procedures to define the process.

Interface & Training: Enhance interface to meet need. Training ongoing as appropriate.

Step 4: Review System
The review is ideally a facilitated workshop for the decision team and the model team together.

Data & Knowledge Report:
• Review any findings with significant policy implications.
• Discuss any major shortcomings and options for redress.

Model Critique:
• Review behavior of current model draft in different scenarios.
• Discuss causes of behavior and judge realism.
• Examine current framework for specifying decisions and observing outcomes.
• Evaluate for usefulness. Business Process Review:
• Outline planned business process and plan for implementation.
- Discuss revisions if necessary.
Interface & Training Review:
• Discuss interface options for breadth of application, and for focusing output to highlight the most cared about results.
• Progress report on training and discussion as required.

Step 5: Handoff
With the model, Reality Check library, data management, and interface complete, and with client personnel oriented and trained to apply the model to decisions, a project typically ends in a facilitated workshop with decision team and model team. Key findings to date are presented, possibly by model team. This marks the beginning of using the model for ongoing decision support.

Deliverables
Project deliverables are determined with the client at project start, but typically include:
• Reality Check archive of documented organizational knowledge,
• Cleaned historical data set appropriate for model use with process in place for ongoing upkeep,
• Model, usually documented equations in Vensim software,
• Documented data & knowledge management system and user interface, and
• Documentation of planned business process for using model to improve decisions.

Project Duration
The first working model prototype usually appears 1-3 weeks after project start, typically followed by several rounds of testing, critique, and enhancement. The amount of time to a finished model depends greatly on the scope and intensity of the effort. Past projects have ranged from 6 weeks to 12 months, with most projects lasting 4-6 months. Scope, intensity, and schedule are all developed with and approved by the client before project start.  

 

+ Ventana Modeling Techniques
Ventana chooses its methods and techniques to help clients get maximum value from their experience. Ventana® technology refines client knowledge and applies it to help clients make smarter choices.

Refining client knowledge: triangulation

Ventana refines organizational experience and data by knowledge triangulation:

• Experience: client expertise, collected through interviews and recorded in Reality Check® libraries as expectations about how changes in one factor will affect others
• Data: observations of past activity, collected from text, spreadsheet or database records and recorded as measurements in time
• Model: descriptions of how actions and environment combine to affect performance, created from client input and recorded in documented model equations>
Each knowledge source provides valuable insight, yet each has gaps, biases, and other possibilities for imperfection. Ventana triangulates among the three sources by cross-checking to identify discrepancies, finding the source of discrepancies with Vensim's Causal Tracing® technology, and resolving them. Combining the three sources through triangulation produces a fuller, more accurate picture than can be obtained from any one or two sources alone, and Ventana's tools make triangulation fast and efficient.

Cross-checking

Experience and Data
Parallel to model development, Ventana examines available data for consistency with client hypotheses about the reasons behind their historical results. This process often reveals the surprising impact of data collection and handling procedures on client understanding, and informs choices about storing, using and acquiring data. The data analysis also inspires deeper thinking about the true drivers of performance, as well as the role of measurements in organizational self-assessment.

Experience and Model
As the model develops, it is continually checked for consistency with client experience. Clients contribute their expertise through common sense cause and effect statements requiring no knowledge of the model or the mathematics. These statements can be stored in Vensim's Reality Check® format, allowing the model to be tested against the entire Reality Check library with a single click. As the model develops, model output is examined for consistency with expected behavior over a wide range of conditions. If the model behaves contrary to expectations, it is a signal to examine the model structure, the expectations, or both. The Reality Check library can grow over time, deepening and strengthening the experience base against which the model is compared.

Data and Model
A model must be able to reproduce historical outputs when given historical inputs. If during the development of a model, it is unable to reproduce history closely enough, it is a sign that the model, the data, or both must be examined to find the cause of the inconsistency. How close is "closely enough"? To be reliable, a model should contain every important effect, but for simple interpretation it should not contain the numerous effects that make little difference. Therefore, while model results should be close to exact, they will not generally be exact. Likewise, properly interpreted data provide a close approximation to the truth, but numerous factors can prevent data from giving a perfectly accurate picture. Ventana applies statistical hypothesis tests to determine when discrepancies between the model and the data are due to noise in the data, and when they are due to gaps in model structure and understanding.

Causal Tracing®

When cross-checking reveals conflicting or missing information, it signals an opportunity to both deepen understanding and improve the model. Vensim's patented Causal Tracing technology provides automated visualization of the causes of model behavior, making it feasible and efficient to track down the root cause of the discrepancy, even through complex models and data sets.

Resolution

Once the cause of a discrepancy is identified, the resolution is often obvious. When it is not, Ventana works with the client to judge the importance of the discrepancy and to determine the most expedient solution. > Applying Knowledge to Make Smarter Choices

A refined knowledge base is the fuel that powers better decisions. Through the best choices of model architecture, decision analysis techniques, and knowledge management tools, Ventana technology applies the knowledge base to help clients understand what choices are most likely to give maximum performance and why.
>
Model architecture
>
The model equations form the bridge from knowledge and data to decisions. Model structure must be chosen to include the client's decision options and measures of success, while remaining consistent with knowledge and data as described above. These requirements dictate the amount of detail and level of focus the model must address. Ventana draws on several analytical architectures, sometimes combining more than one in a single model, to meet these requirements efficiently.> System Dynamics
A widely applicable modeling framework, system dynamics emphasizes cause-and-effect understanding of why things change over time. System dynamics begins by describing the world as a number of stocks, representing the status at any given time of resources (capital, inventories, people, money) and of psychology (morale, satisfaction, awareness). Changes in conditions are represented as flows that increase or deplete each stock over time, calculated from the real world factors which drive them. System dynamics traditionally focuses at an aggregate level, where stocks describe overall or average properties rather than separately tracking each element of a large population.> Agent based modeling
Agent based models represent dozens to millions of individual agents, and separately calculate the status and actions of each one over time. Each agent's actions are determined by rules governing its response to its status, its surroundings, and the actions of other agents with which it is in contact. Overall properties are then observed from the population of individuals. Agent based modeling is a helpful way to understand overall behavior when the complexity of possible interactions among individuals makes an aggregate description difficult.> Single agent modeling
Single agent modeling calculates the status and actions over time of a single actor, independent of the actions of others. Single agent modeling can be viewed as system dynamics applied to a single individual, or as agent based modeling when individuals have little short-term influence on one another's behavior. This approach is most profitable when data are available on the actions of individuals over time. > Statc optimization
Some models require for each period of time a separate optimization specific to that point in time. These situations, such as repeated, short-term resource allocation or pricing decisions, are often amenable to widely used optimization approaches such as linear programming, integer programming, or network programming.> Calibration
Specific model settings that yield the most accurate results are found through automated searches over many possibilities. To avoid relying on trial-and-error, several optimization algorithms exist to steer the search process to the best solutions as quickly as possible while avoiding getting stuck on sub-optimal solutions. Ventana's tests have shown that for many realistic problems, standard algorithms such as the Powell algorithm with multiple starting points outperform alternative techniques such as simulated annealing, genetic algorithms, and simultaneous perturbation stochastic approximation (SPSA). However, these alternatives can be used to advantage in special circumstances.> Decision Analysis

Once a model has been built which is consistent with all available data and experience, a host of techniques can be applied to evaluating potential actions and discovering optimal strategies.> Dynamic optimization
The same automated search procedures used for calibration can be brought to bear on finding the combination of policies that maximize performance. These powerful techniques reveal strategic potential, including new opportunities not yet considered.> Monte Carlo simulation
This common technique explores future possibilities and uncertainty through hundreds or thousands of repeated simulations, representing unknowns as a pool of possible values from which values are drawn at random. > Estimation
Important but hidden factors can be estimated by combining data with an appropriate model. Maximum likelihood estimation is a general approach to this type of statistical inference, including specific procedures such Kalman filtering. > Scenario planning
By understanding potential results--and the reasons for them--in several possible future conditions, it is possible to define actions that hedge against a range of futures, and plan in advance for contingencies. > Real options
The real options framework evaluates investment decisions in uncertain conditions, identifying the actions that will produce the best average likely outcome while accounting for the fact that future information will be used in future decisions.> Knowledge management

The knowledge and data collected for a Ventana model tends to be drawn from all around the client organization, cutting across boundaries to produce a central source for comprehensive understanding. This central repository grows with use, providing an ongoing and improving record of organizational understanding -- and an ever-richer knowledge base, to sharpen ongoing strategy.> Reality Check® archives
Personal knowledge is elicited with expert facilitation in interviews and workshops, and is collected and archived in documented Reality Check libraries.> Documented data collections
Data are absorbed from multiple sources, with the basis and source of each data stream documented and immediately accessible. > Model result archives
Model output is stored together with all relevant inputs for later review, analysis, and comparison to other scenarios. Causal Tracing and other Vensim tools allow fast comparison and visualization of model results and the reasons for differences. > Natural language equation descriptions
The meaning, key assumptions, and source of each model equation is documented in plain language and is immediately accessible as part of the equation definition. > Nearly all of the above techniques are supported directly in the Vensim® modeling environment. The rest are easily connected to Vensim as external functions, or used in parallel to the main model, as expediency demands. Ventana's experience and tools allow for rapid testing and selection of techniques to best meet client needs.>