 |
|
sponsored by
Title : Probability Assessment and Verification: Calibration and Scoring Rules
Date : April 14, 2008
Speaker : J. Eric Bickel, Assistant Professor
Affiliation : Department of Industrial and Systems Engineering, Texas A&M University
Abstract
Bruno de Finetti's began his magnum opus The Theory of Probability with a profound statement. He wrote that "PROBABILITY DOES NOT EXIST" and goes on to compare the belief in "objective" probability with the belief in "fairies and witches." If probability does not exist then how are we to tell if a probability assessment is, in some sense, good?
In this talk I will discuss two properties of probability assessments: calibration and sharpness. Suppose a person has provided many probability assessments f for an event x (e.g., rain today). A probability assessment is well calibrated if p ( x | f ) = f . A set of assessments is sharp (or refined) if p ( f ) = 0 when f does not equal 0 or 1. A "good" probability assessment is sharp and well calibrated. I will present the results of a calibration study of The Weather Channel's probability of precipitation forecasts.
I will also discuss strictly proper scoring rules, which are used to ensure that experts report their true beliefs. These rules reward sharpness and penalize miscalibration. I will discuss important properties of quadratic, spherical, and logarithmic scoring rules. This discussion should be of great interest to students as logarithmic scoring is used on the ISEN 627 midterm.
Biosketch
Dr. Bickel is an Assistant Professor in the Department of Industrial and Systems Engineering at Texas A&M University and Director of the Decision Analysis Systems Lab. His research interests include the theory and practice of decision making under uncertainty (decision and risk analysis), modeling probabilistic dependence, probability assessment, value of information, personal and organizational risk preference, applications of decision analysis to engineering, business, and public policy—especially energy, and applications of decision analysis to sport—especially baseball. Prior to joining Texas A&M, Dr. Bickel was a Senior Engagement Manager for Strategic Decisions Group, where he applied decision analysis to corporate strategy. He has consulted around the world in a range of industries, including oil and gas (upstream and downstream), electricity generation/transmission/delivery, energy trading and marketing, metals and mining, commodity and specialty chemicals, life sciences, and financial services.
Dr. Bickel holds an M.S. and Ph.D. from the Department of Engineering-Economic Systems from Stanford University and a B.S. in mechanical engineering with a minor in economics from New Mexico State University .
