DRAM Evaluation Plan Metrics Workshop Cathy Fogel Bruce

DRAM Evaluation Plan & Metrics Workshop Cathy Fogel & Bruce Kaneshiro Demand Response Section California Public Utilities Commission March 1, 2017 1

Safety Instructions Meeting Spot: Garden Plaza, next to War Memorial Opera House 2

Agenda 3 Time 9: 00 - 9: 15 Topic Introductions, Purpose of Workshop 9: 15 – 10: 00 Presentation of Criterions 1 -2 10: 00 – 10: 45 Presentation of Criterions 3 -4 10: 45 -11: 30 Presentation of Criterions 5 -6 11: 30 – noon Data Collection & Analysis Timing Noon – 12: 30 Data Collection – confidentiality, granularity, sources 12: 30 – 1: 00 Any additional issues Written Comment deadline: March 13 TBD Additional discussion, as needed, see below

Introduction § Goal 1: Research the full DRAM I and DRAM II pilots and the solicitation phase of DRAM III § Goal 2: Provide recommendations based on the research and analysis to guide the Commission in determining if the auction mechanism pilots may be deemed successful 4

Project Objectives § Objective 1: Assess the DRAM pilots based on the Commission’s adopted criteria for determining its success § Objective 2: Explore these criteria using metrics based on input from Parties to R. 13 -09 -011 § Objective 3: Provide data and factual analysis to guide recommendations § Objective 4: Undertake a balanced analysis based on input from the range of market, utility and regulatory actors 5

Project Management Team Project Oversight Bruce Kaneshiro Non-financially interested ies Project Direction D. 1609056 Project Guidance Data Management Guidance ED Project Manager David Miller, Jamie Rose Gannon Cathleen Fogel CAISO BID & Dispatch Data Analysis 6 Simon Baker CPUC Modeling Team

Success Criteria Adopted in D. 16 -09 -056 Were new, viable third-party providers engaged? Were new customers engaged? Were bid prices competitive? Were offer prices competitive in the wholesale markets? Did demand response providers aggregate the capacity they contracted, or replace it with demand response from another source in a timely manner? § Were resources reliable when dispatched, i. e. , did customers perform appropriately? § § § 7

Additional Direction in D. 16 -09 -056 § Final DRAM Eval Plan & Metrics- by April 1, 2017 § ED to release a draft Resolution with findings / recommendations by June 1, 2018 § If Resolution approved, workshop within 30 days § First auction in spring 2019 for 2020 delivery 8 • Adopts eight provisions for possible future DRAM

Criteria & Proposed Metrics Assess by or Compare to: 1. Were new, viable third-party providers engaged? 1. 1 Number of participating third party providers bidding into and winning 1. 2 1. 3 9 bids; Number of DRPs registered with CPUC Quantity (MW) and percentage of accepted bids compared to all bids Data Source IOUs / CAISO / IE report - Aggregators, DRPs, SCs; New 3 Ps, or those previously operating in California DRAM I - III - New 3 P, returning MW, number, budget DRAM I – III IOUs C? N What were the challenges to third-party engagement- general? Compare perceptions of ED survey Third party perceptions of: winning and non-winning and/or 1) Ease of participation; aggregators interviews with 2) Fairness / transparency of bid Aggregators, selection process; SCs, DRPs, IOUs; 3) Primary barriers to participation; 4) Primary source of transaction costs; IOU surveys 5) Market confidence; from bidders conferences 6) Understanding of CAISO products, performance requirements and markets N N

Criteria & Proposed Metrics Assess by or Compare to: Data Source What were the challenges to third-party engagement- in the IOU auction process? (Were there barriers to participation? ) 1. 4 Aggregators IOUs Number of potential third party DRAM I - III providers contacted by IOUs 1. 5 1. 6 1. 7 1. 8 1. 9 10 Number of potential third party providers attending DRAM bidding conference(s) Number of questions at bidding conference by potential third party providers, the topics and average IOU response times Number/ percentage of potential third party providers bidding into DRAM compared to attendees at conference - Number / percent of DRAM bidders with conformance check issues Number / percent of winning bidders unwilling to sign contracts and reasons. - Aggregators, DRPs, SCs DRAM I - III (optional, time permitting) - C? N IOUs N - (optional, time permitting) New or returning aggregators - - IOUs N ED interviews with. Aggregators, Y SCs, DRPs, IOUs;

Criteria & Proposed Metrics Assess by or Compare to: Data Source C? What were the challenges to third party engagement – in the customer enrollment process? (Were there barriers to signing up customers? ) 1. 10 Number and percent of DRPs or This will include exploration of barriers Interviews with aggregators experiencing integration due to lack of availability of needed Aggregators, challenges with processes to enroll customer data due to Green Button, SCs, DRPs, IOUs; N customers. Type of challenge and status. My. Connect, and/or other issues. Quantitative data when 1. 11 DRP, SC and/or aggregator perceptions available of differences between DRAM and IOU DR programs that could limit customer N participation. What were the challenges to third party participation the CAISO bidding process ? (Were there barriers to making bids? ) 1. 12 Number and percent of DRPs or SCs By customer class Interviews with experiencing supply-side integration By Aggregator SCs, DRPs, Agg challenges with CAISO processes. Type By new / returning third party providers ; Quantitative N of challenge and status. By PDR or RDR resource data when available 1. 13 Quantity (MW) of DRAM resources in By type of RA: System / Flexible / Local capacity; Supply Plans and quantity bid into IOUs By August, month and year CAISO DRAM I & II only Compare to IOU DR resources Supply N Plans and quantity bid into CAISO 11

Criteria & Proposed Metrics 2. Were new customers engaged? 2. 1 -Number and percentage of new customers; -Number and percentage of customers shifted from IOU DR programs to DRAM -Quantity (MW) and percentage capacity provided by shifted and new DRP customers 2. 2 2. 3 2. 4 Assess by or Compare to: As indicated by registrations; IOUs / DRPs By customer class, DRP; PDR / RDRR; By type of RA: System / Flexible / Local capacity; Type of resource – DR / storage / EV; By August; annual Average duration of customer participation in DRAM and As indicated by registrations; By customer class, IOU DR programs and reasons for withdrawal Customer perception of differences between direct participation and IOU DR programs that could limit customer participation. Location of DRAM customers (number and percent)? DRP PDR / RDRR C? N ED review of DRP and IOU surveys; or, Y ED surveys or interviews (optional, time permitting) N By county By sub. LAP By local reliability area and/or transmission constrained areas? In disadvantaged communities? Total and percentage of participating customers on CARE (optional, time permitting) rates, ESA participants, in multifamily dwellings and/or in top 5% of electricity consumers 2. 6 Number / percent of registrations requested in nonwinning bids [due to: 1) budget; 2) registration; or 3) 12 “outlier” limitations] 2. 5 Data Source IOUs/ 3 Ps: Geo-map or list by county of N location of registrations IOUs N IOUs N

Criteria & Proposed Metrics Assess by or Compare to: Data Source C? 2. A. High level characterization of new customers [Optional, time permitting] 2. A. 1 2. A. 2 2. A. 3 2. A. 4 13 Percent of customers with automated response capabilities IOUs / DRPs Number and percent of DRAM customers participating via storage-tied response systems that have additional onsite DER resources such as EVs, PV, or EE incentives. High, low and average participating NR and Res customer loads for August and peak days. N As indicated by registrations; By customer class, DRP; By PDR / RDRR; By type of RA: System / By Flexible / Local capacity; By Type of resource – DR / storage / EV; By August; annual N IOUs/ DRP / ED survey of participating customers with storage N IOUs N

Criteria & Proposed Metrics Assess by or Compare to: 3, Were bid (auction, capacity) prices competitive? 3. 1 High, low and average bid As bid; procured; By PDR / RDR; By resource type: (auction, capacity) bid prices system, flexible and local capacity; By customer class; DRP; By New /returning 3 P; with / without scheduling coordinator costs included; for August / by month / by year) 3. 2 High, low and average By customer class Scheduling Coordinator costs 3. 3 3. 4 14 -High, low and average procured DRAM August capacity prices -Average non-procured bid August capacity prices Other metrics on procured DRAM such as: Net Present Value (NPV) All-in-cost Data Source C? IOUs / IE reports Y IOUs / IE reports As compared to the following benchmarks: IOUs DR from IOU programs All RA CAISO Capacity Procurement Mechanism the cost-effectiveness requirements of R. 14 -10 -003 As compared to the following benchmarks: IOUs DR from IOU programs All RA CAISO Capacity Procurement Mechanism costs Cost-effectiveness requirements of R. 14 -10 -003 Y Y Y

Criteria & Proposed Metrics Assess by or Compare to: Data Source C? 4. Were bid offer prices competitive in the wholesale market? (Defined as wholesale energy bid prices in CAISO markets) 4. 1 Quantity (MW) of DR resources bid into CAISO and quantity and percent dispatched See below. 4. 2 Quantity (MW) and percent of DR resources bid For what hours? During these hours, what quantity of DR into CAISO below the monthly NBT threshold? CAISO / DRPs resources were dispatched? 4. 3 4. 4 15 Quantity (MW) and percent of DR resources dispatched compared to total resources dispatched by CAISO High, low and weighted average August DR bid price not dispatched as compared to high, low and weighted average price of dispatched resources N As bid; procured; By PDR / RDR; CAISO / DRPs By resource type: system, flexible and local capacity; By customer class; DRP; By New /returning 3 P; with / without scheduling CAISO / DRPs coordinator costs included; for August / by month / by year) Y N Y

Criteria & Proposed Metrics Assess by or Compare to: Data Source 5. Did demand response providers aggregate the capacity they contracted, or replace it with demand response from another source in a timely manner? IOUs 5. 1 Quantity (MW) and percent of DRAM resources in Supply Plans as compared contracted amounts. Compare to IOU DR resources DRPs 5. 2 Quantity (MW) and percentage of DR resources using each of the three contractually-approved methods of receiving capacity payments By customer class; 5. 3 5. 4 5. 5 5. 6 By Aggregator IOUs / DRPs By new / returning Number of capacity payments withheld, reasons, and MW/value ED interviews third party affected providers ; ED interviews with Quantity and value of contracts terminated and/or penalized by IOUs for By PDR or RDR DRPs, IOUs; non-performance; Value of any assessed fees on 3 Ps resource Quantitative data By type of RA: Total and percentage of bidders found to have engaged in non. System / Flexible / ED staff / IE reports competitive behavior Local capacity CAISO / IOUs Quantity and percent of DRAM resources in Supply Plans provided to CAISO (by SCs at 45 days+) as compared to Supply Plans provided to C? N Y Y N IOUs (by 3 Ps) 5. 7 Quantity and percent of contracted capacity for which CAISO registrations occurred after SC’s Supply Plans provided to CAISO (at 45 days+) 5. 8 Quantity (MW) and percent of contracts for which Seller provided 16 capacity from non-residential meters, when contracted for residential CAISO /DRPs N IOUs N

Criteria & Proposed Metrics 6. 6. 1 Assess by or Compare to: Data Source C? Were resources reliable when dispatched, i. e. did customers perform appropriately? Quantity (MW) and percentage of total dispatched DR resources subject to RRAIM penalties DRAM II only PDR / RDRR August, annual, peak days CAISO Y Compare to IOU DR 6. 2 6. 3 6. 4 17 Quantity (MW) and percent of total dispatched RDRR / PDR resources subject to Uninstructed Energy DRAM and IOU DR Imbalance (UEI) fines Annually, peak days, August CAISO Load impacts of DRAM resources during CAISO RDRR / PDR resources dispatch events DRAM and IOU DR Annually, peak days, August CAISO; DRPs Response time and quantity (MW) of response RDRR / PDR resources to dispatch DRAM and IOU DR Annually, peak days, August CAISO Y N Y

Project Timeline Phase 1 DRAM I and II auctions. DRAM I delivery; first half of DRAM II deliveries (Jan- June). Phase 2 DRAM III auction Second half of DRAM II deliveries (July – December). Interviews with DRAM market participants and IOUs. 18

Treatment of Confidential Data § Will abide by Commission Decisions, Resolutions & PUC codes: • • • 19 Resolution E-4728 PUC Code Section 314, Subdivision (b) PUC Code Section 583, Subdivision (b) Commission General Order 66 -C D. 16 -08 -024 at page 19 D. 06 -06 -066

Data Granularity / Sources ? § Granularity • Will generally use smallest granularity available • Have requested 5 min dispatch data from CAISO • Input ? § Sources • Are there additional data sources not yet discussed? 20

Any Additional Questions / Comments? § Deadline for written comments March 13 § Please send your comments to cathleen. [email protected] ca. gov and copy the Service List of R. 13 -09 -011 § Request comments not exceed five pages, double spaced. § Some questions for comment are in Appendix A of the draft Plan (and next slide) § Energy Division will release a final DRAM Evaluation 21 Plan & Metrics by April 1, 2017.

Specific Questions for Comment (not required) § Are the goals & objectives complete? § Concerns / suggestions about handling and/or presentation of confidential data? § Missing metrics? § Data source / granularity suggestions? Can your organization provide data not mentioned? § What metrics are the most important to demonstrate the success of the DRAM? § What ones do you see as least important or fully optional? § Overall project and/or individual phase timeline? § Additional comments. 22

23