Modeling (Fourth Exposure Draft)

Modeling

Fourth Exposure Draft

TRANSMITTAL MEMORANDUM

December 2018

TO: Members of Actuarial Organizations Governed by the Standards of Practice of the Actuarial Standards Board and Other Persons Interested in Modeling

FROM: Actuarial Standards Board (ASB)

SUBJ: Proposed Actuarial Standard of Practice (ASOP) on Modeling

The ASB accepts comments by either electronic or conventional mail. The preferred form is e-mail, as it eases the task of grouping comments by section. However, please feel free to use either form. If you wish to use e-mail, please send a message to comments@actuary.org. You may include your comments either in the body of the message or as an attachment prepared in any commonly used word processing format. Please do not embed your comments in the exposure draft and do not password protect any attachments. If the attachment is in the form of a PDF, please do not “copy protect” the PDF. Include the phrase “ASB COMMENTS” in the subject line of your message. Please note: Any message not containing this exact phrase in the subject line will be deleted by our system’s spam filter. Also, please indicate in the body of the e-mail if your comments are being submitted on your own behalf or on behalf of a company or organization.

If you wish to use conventional mail, please send comments to the following address:

Modeling (Fourth Exposure)
Actuarial Standards Board
1850 M Street, NW, Suite 300
Washington, DC 20036

The ASB posts all signed comments received to its website to encourage transparency and dialogue. Comments received after the deadline may not be considered. Anonymous comments will not be considered by the ASB nor posted to the website. Comments will be posted in the order that they are received. All posted comments will be available to the general public on the ASB website. The ASB disclaims any responsibility for the content of the comments, which are solely the responsibility of those who submit them.

For more information on the exposure process, please see the ASB Procedures Manual.

Deadline for receipt of responses in the ASB office: May 15, 2019

History of the Standard

The ASB first began work on a standard for modeling in the late 1990s. Motivated primarily to address the role catastrophe modeling of earthquakes and hurricanes played in casualty ratemaking, this work was focused on the use of specialized models where actuaries would have to rely on a model that was developed by professionals other than actuaries. As a result of this work, ASOP No. 38, Using Models Outside the Actuary’s Area of Expertise, was approved by the ASB in June of 2000 with the scope of the standard limited to the Property/Casualty area of practice. Historically, ASOP No. 38 had been the only ASOP that specifically addressed modeling.

Recently, the number and importance of modeling applications in actuarial science has increased, with the results of actuarial models often entering financial statements directly. Recognizing this trend, the ASB asked the Life Committee in 2010 to begin work on an ASOP focused on modeling. The Life Committee formed a task force to address this issue and, in February of 2012, a discussion draft titled Modeling in Life Insurance and Annuities was released and nineteen comment letters were received. The transmittal letter also mentioned that the scope might be expanded to all practice areas and asked for comments on this idea.

Based upon the feedback received, and numerous other discussions on the topic of modeling, in December of 2012 the ASB created two multi-disciplinary task forces under the direction of the General Committee: i) a general Modeling Task Force, charged with developing an ASOP to address modeling applications in all practice areas, and ii) a Catastrophe Modeling Task Force to consider expanding ASOP No. 38 to all practice areas while focusing exclusively on using catastrophe models. The membership of these task forces has experience in all actuarial practice areas, including enterprise risk management.

As the guidance in this proposed modeling ASOP and in the working draft of ASOP No. 38, currently titled Catastrophe Modeling (for All Practice Areas), is intended to be coordinated, the ASB plans to issue final versions of both ASOPs to be effective concurrently. To facilitate review of this proposed modeling ASOP, a link to the current working draft of ASOP No. 38 is provided here for your information. The working draft of ASOP No. 38 is not being exposed for comment at this time but does reflect guidance that the ASB and General Committee believe works in concert with the guidance in this fourth exposure draft of this proposed modeling ASOP.

First Exposure Draft

The first exposure draft, titled Modeling, was released in June 2013 with a comment deadline of September 30, 2013. Forty-eight comment letters were received and considered in making changes that were reflected in the second exposure draft.

Second Exposure Draft

A second exposure draft, titled Modeling, was released in November 2014 with a comment deadline of March 1, 2015. Thirty-seven comment letters were received and considered in making changes that were reflected in the third exposure draft.

Third Exposure Draft

A third exposure draft, titled Modeling, was released in June 2016 with a comment deadline of October 31, 2016. Twenty-eight comment letters were received and considered in making changes that were reflected in this fourth exposure draft. For a summary of issues contained in these comment letters, please see appendix 2.

Notable Changes from the Third Exposure Draft

Changes made to the fourth exposure draft include the following:

1. clearer accommodation of forms of modeling other than financial projection modeling (such as predictive and statistical modeling), such as the clarification of the applicability of the new section 3.1.6;
2. a revision of section 1.2., Scope, as follows:

• removed the concept of “simple models” from the scope since the definition of “simple models” was not helpful;
• reorganized scope around the role of the actuary; and
• incorporated portions of prior sections 3.1 and 3.8 into section 1.2;

3. addition of new section 3.1.6(f), Reasonable Model in the Aggregate, and related disclosure in section 4.1(d);
4. inclusion of new section 3.4, Reliance on Experts; and
5. elimination of the Presentation of Results section (section 3.6 in the third exposure draft) and revisions to section 4.

Request for Comments

The ASB appreciates comments and suggestions on all areas of this proposed standard.

Please provide comments that are succinct, identifying particular sections in the exposure draft for which specific alternative wording is suggested, including the rationale for each suggestion.

The ASB voted in December 2018 to approve this exposure draft.

Modeling Task Force
Dale S. Hagstrom, Chairperson
Maryellen J. Coggins	Stephen Mildenhall
Julie H. Fried	Judy K. Stromback
Kenneth R. Kasner

General Committee of the ASB
Margaret Tiller Sherwood, Chairperson
Shawna S. Ackerman	Susan E. Pantely
Ralph S. Blanchard III	Judy K. Stromback
Andrew M. Erman	Hal Tepfer
Dale S. Hagstrom	Christian J. Wolfe

Actuarial Standards Board
Beth E. Fitzgerald, Chairperson
Christopher S. Carlson	Darrell D. Knapp
Maryellen J. Coggins	Cande J. Olsen
Robert M. Damler	Kathleen A. Riley
Mita D. Drazilov	Barbara L. Snyder

The Actuarial Standards Board (ASB) sets standards for appropriate actuarial practice
 in the United States through the development and promulgation of Actuarial Standards of Practice (ASOPs). These ASOPs describe the procedures an actuary should follow when performing actuarial services and identify what the actuary should disclose when communicating the results of those services.

PROPOSED ACTUARIAL STANDARD OF PRACTICE

MODELING

STANDARD OF PRACTICE

Section 1. Purpose, Scope, Cross References, and Effective Date

1.1 Purpose

This actuarial standard of practice (ASOP or standard) provides guidance to actuaries when performing actuarial services with respect to designing, developing, selecting, modifying, using, reviewing, or evaluating models.

1.2 Scope

This standard applies to actuaries in any practice area when performing actuarial services with respect to designing, developing, selecting, modifying, or using all types of models. For example, an actuary using a model developed by others in which the actuary is responsible for the model output is subject to this standard.

If the actuary’s actuarial services involve reviewing or evaluating models, the reviewing or evaluating actuary should be reasonably satisfied that the actuarial services were performed in accordance with this standard. The reviewing or evaluating actuary should use the guidance in this standard to the extent practicable within the scope of the actuary’s assignment.

The guidance in this ASOP applies when, in the actuary’s professional judgment, intended users of the model rely on the output of the model, and their use of the output of the model has a material effect for the intended user. This judgment should be made within the context of the use of the model output and the needs of the intended user, based on facts known by the actuary at the time the actuarial services are performed. For example, actuarial services performed in relation to pension plan contribution and cost projection models, insurance pricing (including predictive) or reserving models, and insurance company financial planning models may require application of the guidance in this ASOP. In assessing materiality, the actuary should be guided by ASOP No. 1, Introductory Actuarial Standard of Practice, section 2.6.

The guidance in this ASOP does not apply to actuaries when performing services with respect to individual pension benefit calculations, as described in section 1.2 of ASOP No. 4, Measuring Pension Obligations and Determining Pension Plan Costs or Contributions.

The actuary should understand the extent of the actuary’s responsibilities. The actuary’s responsibilities may extend to performing actuarial services related to an entire model or to a small portion of a model. This standard only applies to the extent of the actuary’s responsibility.

Other ASOPs provide specific requirements for actuarial services that involve models. If the actuary determines that such specific guidance from an applicable ASOP conflicts with the guidance of this ASOP, the guidance of such other ASOPs will govern.

If the actuary departs from the guidance set forth in this ASOP in order to comply with applicable law (statutes, regulations, and other legally binding authority), or for any other reason, the actuary should refer to section 4. If a conflict exists between this standard and applicable law, the actuary should comply with applicable law.

1.3 Cross References

When this ASOP refers to the provisions of other documents, the reference includes the referenced documents as they may be amended or restated in the future, and any successor to them, by whatever name called. If any amended or restated document differs materially from the originally referenced document, the actuary should consider the guidance in this ASOP to the extent it is applicable and appropriate.

1.4 Effective Date

This ASOP is effective for work performed on or after nine months after adoption by the Actuarial Standards Board.

Section 2. Definitions

The terms below are defined for use in this actuarial standard of practice.

2.1 Assumption

A type of input to a model that represents expectations, represents possibilities based on professional judgment, or may be prescribed by law or by others.

2.2 Data

Facts or information that are either direct input to a model or inform the selection of input. Data may be collected from sources such as records, experience, experiments, surveys, observations, or output from other models.

2.3 Governance and Controls

The application of a set of procedures and an organizational structure designed so that intended users can place their confidence in the output of the model.

2.4 Input

Information used in a model to produce output.

2.5 Intended Purpose

The goal or question, whether generalized or specific, addressed by the model within the context of the assignment.

2.6 Intended User

Any person whom the actuary identifies as able to rely on the actuarial findings.

2.7 Model

A simplified representation of relationships among real world variables, entities, or events using statistical, financial, economic, mathematical, or scientific concepts and equations. A model consists of three components: an information input component, which delivers assumptions, data, and sometimes parameters to the model; a processing component, which transforms input into output; and a results component, which translates the output into useful business information. Models are used to help explain a system, to study the effects of different parts of a system, to predict the behavior of a system, or to derive estimates and guide decisions.

2.8 Model Risk

The risk of adverse consequences resulting from reliance on a model that does not adequately represent that which is being modeled or that is misused or misinterpreted.

2.9 Model Run

The process of transforming a particular selection of input to a particular set of output in a model. A model run could include the whole transformation process or part of the process, as applicable.

2.10 Output

The results of a model including point estimates, likely or possible ranges, parameters (as input for other models), or qualitative criteria on which decisions could be made.

2.11 Overfitting

A situation where a model fits sample data so closely that prediction accuracy decreases when the model is applied to different (for example, out-of-sample) data.

2.12 Parameter

A type of statistical, financial, economic, mathematical, scientific, or contractual input to certain types of models. Examples of parameters include expected values in probability distributions, coefficients of formula variables, and benefit plan or policy provisions. Some types of models, such as predictive or statistical models, produce estimates of parameters, which may be used as input to other models.

Section 3.  Analysis of Issues and Recommended Practices

3.1 Model Meeting the Intended Purpose

The actuary should understand the model’s intended purpose.

3.1.1 Designing, Developing, or Modifying the Model

When the actuary designs, develops, or modifies the model, the actuary should confirm that the capability of the model is consistent with the intended purpose. Items the actuary should consider, if applicable, include but are not limited to the level of detail built into a model, the dependencies recognized, and the model’s ability to identify possible volatility around expected values.

3.1.2 Selecting, Using, Reviewing, or Evaluating the Model

When selecting, reviewing, or evaluating the model, the actuary should confirm the model reasonably meets the intended purpose. When using the model, the actuary should make reasonable efforts to ensure that any revisions to the input and formulas, documentation, governance and controls, validation, and presentation of output are consistent with the intended purpose.

3.1.3 Understanding the Model

When expressing an opinion on or communicating results of the model, the actuary should understand the following:

a. important aspects of the model being used, including but not limited to, basic operations, important dependencies, and major sensitivities;

b. known weaknesses in assumptions and parameters used as input, known weaknesses in methods or other known limitations of the model that have material implications; and

c. any limitation of data or information, time constraints, or other practical considerations that could materially impact the model’s ability to meet its intended purpose.

3.1.4 Model Structure

The actuary should assess whether the structure of the model (including judgments reflected in the model) is appropriate for the intended purpose. The actuary should consider the following, as applicable, for a particular model:

a. which provisions and risks specific to a business segment, contract, or plan, if any, or interactions more broadly, are material and appropriate to reflect in the model;

b. whether the form of the model is appropriate, such as a projection model (deterministic or stochastic), statistical model, or predictive model;

c. whether the use of the model dictates a particular level of detail, for example, whether grouping inputs will produce reasonable output or whether a certain level of detail in the output is needed to meet the intended purpose;

d. whether the model is overfitting the data; and

e. whether the model appropriately represents options, if any, that could be reasonably expected to have a material effect on the output of the model. Examples include call options on fixed income assets, policyholder surrender options, and early retirement options.

3.1.5 Data

The actuary should use, or confirm use of, data appropriate for the model’s intended purpose and should refer to ASOP No. 23, Data Quality, when selecting, reviewing, or evaluating data used in the model, either directly or as the basis for deriving, estimating, or testing assumptions and parameters used as input to the model.

3.1.6 Assumptions and Parameters Used As Input

For models that use assumptions and parameters as input, the actuary should use, or confirm use of, assumptions and parameters that are appropriate in light of the model’s intended purpose. The following guidance applies only for models that use assumptions and parameters as input:

a. Setting Assumptions and Parameters—When setting assumptions and parameters, the actuary should consider using the following:

1. actual experience adjusted to current conditions where applicable, to the extent it is available, relevant, and sufficiently reliable;

2. other relevant and sufficiently reliable experience, such as industry experience that is properly modified to reflect the circumstances being modeled, if actual experience is not available or relevant, or is not sufficiently reliable;

3. future expectations, estimates inherent in market data when available and appropriate, or a combination of both; or

4. other relevant sources of information.

b. Margins—If appropriate, the actuary may consider adjusting an assumption or parameter to include margins. If the actuary considers inclusion of margins within a model, the actuary should consider the potential impact to the model output to ensure it has the intended impact in the aggregate. If the actuary does decide to use margins to adjust an assumption or parameter, the actuary should select margins that are reasonable in light of relevant, available experience, given the model’s intended purpose.

c. Range of Assumptions and Parameters—The actuary may consider using a range of assumptions and parameters and, if so, whether the number of model runs analyzed reflects a set of conditions consistent with the intended purpose.

d. Consistency—Where appropriate, the actuary should use, or confirm use of, assumptions and parameters for the model that are reasonably consistent with one another for a given model run.

If the actuary is aware of any material inconsistencies among assumptions and parameters used by the actuary in the model, the actuary should disclose the inconsistencies and any known reasons for the inconsistencies in accordance with section 4.1(c). In the case of assumptions and parameters prescribed by applicable law, the actuary’s disclosure may be limited to identifying the possibility of an inconsistency with other assumptions and parameters.

e. Appropriateness of Input in Current Model Run—Where practical and appropriate, the actuary reusing an existing model should evaluate whether the input is still appropriate for use in the current model run. For example, models used in financial reporting may offer opportunities to compare assumptions and parameters to emerging experience in the aggregate.

f. Reasonable Model in the Aggregate—The actuary should assess whether the assumptions and parameters are reasonable in the aggregate. While assumptions and parameters might appear to be reasonable individually, conservativism or optimism in multiple assumptions and parameters may result in a set of assumptions and parameters that produces unreasonable output.

3.2 Reliance on Data or Other Information Supplied by Others

When relying on data or other information supplied by others, the actuary should refer to ASOP No. 23 and ASOP No. 41, Actuarial Communications, for guidance.

3.3 Reliance on Models Developed by Others

If the actuary relies on a model designed, developed, or modified by others, such as a vendor or colleague, and the actuary has a limited ability to obtain information about the model or to understand the underlying workings of the model, the actuary should disclose the extent of any such reliance. In addition, the actuary should make a reasonable attempt to have a basic understanding of the model, including the following, as appropriate:

a. the designer’s or developer’s original intended purpose for the model;

b. the general operation of the model;

c. major sensitivities and dependencies within the model; and

d. key strengths and limitations of the model.

When relying on models developed by others, the actuary should make practical efforts to comply with other applicable sections of this standard.

3.4 Reliance on Experts

The actuary may rely on experts in the fields of knowledge used in the development of the model. In determining the appropriate level of reliance, the actuary may consider the following:

a. whether the individual or individuals upon whom the actuary is relying are experts in the applicable field;

b. the extent to which the model has been reviewed or validated by experts in the applicable field, including any known significant differences of opinion among experts concerning aspects of the model that could be material to the actuary’s use of the model;

c. whether there are industry or regulatory standards that apply to the model or to the testing or validation of the model, and whether the model has been certified as having met such standards; and

d. whether the science underlying the expertise is likely to produce useful models for the intended purpose.

The actuary should disclose the extent of any such reliance.

3.5 Mitigation of Model Risk

The actuary should evaluate model risk and, if appropriate, take reasonable steps to mitigate model risk. The type and degree of model risk mitigation that is reasonable and appropriate may depend on the following:

a. the model’s intended purpose;

b. the nature and complexity of the model;

c. the operating environment and governance and controls related to the model;

d. whether there have been any changes to the model or the model environment; and

e. the balance between the cost of the mitigation efforts and the reduction in potential model risk.

3.5.1 Model Testing

For a model run or set of model runs generated at one time or over time that is to be relied upon by the intended user, the actuary should perform sufficient testing to ensure that the model reasonably represents that which is intended to be modeled. Model testing may include the following:

a. reconciling relevant input values to the relevant system, study, or other source of information, addressing and documenting the differences appearing in the reconciliation, if material;

b. checking formulas, logic, and table references; and

c. reconciling the output of a model run to prior model runs, given any changes in assumptions and parameters used as input, data, formulas, or other aspects of the model since the prior model run.

3.5.2. Model Validation

The actuary should take appropriate steps to validate that the model output reasonably represents that which is being modeled. Depending on the intended purpose, model validation may include the following:

a. testing, where applicable, preliminary model output against historical actual results to verify that modeled output would bear a reasonable relationship to actual results over a given time period if input to the model were set to be consistent with the conditions prevailing during such period;

b. performing statistical or analytical tests on model output to assess their reasonableness;

c. running tests of variations on key assumptions and parameters used as input to test that changes in the output are consistent with the changes in those assumptions and parameters; and

d. comparing model output to those of an alternative model(s), where appropriate.

3.5.3 Review by Another Professional

The actuary may consider obtaining a review by a second, qualified professional, depending upon the nature and complexity of the model.

3.5.4 Reasonable Governance and Controls

The actuary should use, or, if appropriate, may rely on others to use, reasonable governance and controls to mitigate model risk.

3.5.5 Mitigating Misuse and Misinterpretation

The actuary should refer to the guidance in ASOP No. 41, in particular sections 3.4.1 and 3.7, to mitigate possible misuse and misinterpretation of the model.

3.6 Documentation

The actuary should consider preparing and retaining documentation to support compliance with the requirements of section 3 and the disclosure requirements of section 4. The actuary should consider preparing such documentation in a form such that another actuary qualified in the same practice area could assess the reasonableness of the actuary’s work or could assume the assignment if necessary. The degree of such documentation should be based on the professional judgment of the actuary, and may vary with the complexity and purpose of the actuarial services. In addition, the actuary should refer to ASOP No. 41, section 3.8, for guidance related to the retention of file material other than that which is to be disclosed under section 4.

Section 4. Communications and Disclosures

4.1 Required Disclosures in an Actuarial Report

When issuing an actuarial report under this standard, the actuary should refer to ASOP Nos. 23 and 41. In addition, the actuary should disclose the following in such actuarial reports:

a. the intended purpose of the model, as discussed in section 3.1;

b. material limitations that result from the items discussed in section 3.1.3;

c. material inconsistencies, if any, among assumptions and parameters, and any known reasons for such inconsistencies, as discussed in section 3.1.6(d);

d. unreasonable output resulting from the aggregation of assumptions and parameters used as input, if any, as discussed in section 3.1.6(f);

e. extent of reliance on models developed by others, if any, as discussed in section 3.3; and

f. extent of reliance on experts, if any, as discussed in section 3.4.

4.2 Additional Disclosures in an Actuarial Report

The actuary should include the following, as applicable, in an actuarial report:

a. the disclosure in ASOP No. 41, section 4.2, if any material assumption, parameter, or method was prescribed by applicable law;

b. the disclosure in ASOP No. 41, section 4.3, if the actuary states reliance on other sources and thereby disclaims responsibility for any material assumption, parameter, or method selected by a party other than the actuary; and

c. the disclosure in ASOP No. 41, section 4.4, if, in the actuary’s professional judgment, the actuary has otherwise deviated materially from the guidance of this ASOP.

4.3 Confidential Information

Nothing in this ASOP is intended to require the actuary to disclose confidential information.

Appendix 1

Background and Current Practices

Note: This appendix is provided for informational purposes and is not part of the standard of practice.

Background

Models are used to help explain a system, to study the effects of different components, and to derive estimates and guide decisions. Models have always played a fundamental role in actuarial work, with every discipline relying on a broad range of modeling applications, ranging from simple spreadsheets to complex capital models. The number and importance of modeling applications in actuarial science have continued to increase, with the results of actuarial models frequently entering financial statements directly.

Actuaries often develop and use models when analyzing uncertain outcomes. Even a model that is prudently developed and carefully used does not eliminate inherent uncertainty and variability, and actual experience may differ, sometimes significantly, from the estimates derived from the model results. A model is only an approximation of reality, not the reality itself, and the differences between the model and actual experience, by themselves, do not indicate a flawed model or noncompliance with standards.

This standard covers a wide range of models, ranging from projection models to statistical models and predictive models. These models have different purposes. Some models evolve through a life cycle consisting of: (1) a specification phase, (2) an implementation phase, and (3) a production phase, which consists of one or more model runs. Other models evolve through a life cycle consisting of the following: (1) a specification phase, (2) a possibly iterative, parameter estimation phase fitting one or more model structures, and (3) an output evaluation, validation, and selection phase. For other models, other combinations of functionally similar phases may exist.

When a model will be used repeatedly, it is common that the model will be subject to appropriate governance and controls. Examples of model governance and controls include the following:

• limitations on access to use and modify the model (that is, restricting access to model input, model code and calculations, and model output);
• confirmation that model output is reproducible upon rerun (if the model allows for such reproducibility);
• implementing a model change management process;
• specification, documentation, and programming standards for the implementation;
• procedures for secure back-up of the media storing the implementation and data;
• appropriate staff training or cross-training for continuity of use and mitigation of key-person risk;
• plans for periodic consideration of the organization’s continued ability to access and maintain the model, including data, software, staff, hardware, and vendor relationships;
• plans for periodic updating of model input; and
• plans for periodic review of the assumptions, parameters, functionality, and methodology.

The actuary designing or developing a model often considers whether the model can be easily updated for anticipated changes in data, parameters, or assumptions.

Current Practices

The use of margins in model assumptions may differ by practice area. In some practice areas, the ideal model would use only assumptions without margins. In other practice areas, the current practice is to use assumptions that may include margins. Possible reasons for using margins include adding an element of conservatism or adjusting for the cost of bearing risk. The size of the margins may be driven by future unpredictability, experience data that are not fully reliable, or both.

Appendix 2

Comments on the Third Exposure Draft and Responses

The third exposure draft titled Modeling was approved by the ASB in June 2016 with a comment deadline of October 31, 2016. Twenty-eight comment letters were received, some of which were submitted on behalf of multiple commentators, such as by firms or committees. For purposes of this appendix, the term “commentator” may refer to more than one person associated with a particular comment letter. The Task Force and General Committee carefully considered all comments received, and the ASB reviewed (and modified, where appropriate) the changes proposed by the General Committee.

Click here to view a summary of the significant issues and questions contained in the comment letters and responses. Minor wording or punctuation changes that are suggested but not significant are not reflected in the appendix, although they may have been adopted.

The term “reviewers” includes the Task Force, General Committee, and the ASB. Unless otherwise noted, the section numbers and titles used below refer to those in the third exposure draft, which are cross referenced with those in the fourth exposure draft.

Comments on this Exposure Draft

Click here to view the comments on this exposure draft in their entirety.