Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Fatkulin hotsos 2014

•
•
E
Enkitec

This document summarizes a presentation about leveraging in-memory storage to overcome Oracle PGA memory limits. The presenter is a senior consultant with experience designing and implementing clustered and high availability Oracle solutions. They discuss how data volumes and processing power have increased while database designs have decreased over time. They cover Oracle's PGA memory structure and limits, including how manually and automatically managing work areas. The document also summarizes how using techniques like Linux tmpfs or ZFSSA can dramatically improve temporary I/O performance by 10x to 50x times for large queries that hit PGA limits.

1 of 29
Leveraging In-Memory Storage to Overcome Oracle PGA Memory Limits March 5, 2014 Presented by: Alex Fatkulin Senior Consultant
Who am I ?  Senior Technical Consultant at Enkitec  12 years using Oracle  Clustered and HA solutions  Database Development and Design  Technical Reviewer  Blog at http://afatkulin.blogspot.com 3
Why This Presentation? 4
Data Growth and Processing  Data Volumes UP  Processing Power UP  Database Design (in general) DOWN 5 Data Volume Processing Power Database Design BetterWorse
6 How many people have systems with more than 128G of RAM?
“640K” Problems 7  Processing patterns change  Things that didn’t matter before matter now  Legacy RDBMS code vs 64-bit systems  KIWI (Kill It With Iron)
PGA Memory 8
PGA Memory (Dedicated Server) 9 Sort AreaHash Area Bitmap Area SQL Work Areas Session Memory Private SQL Area Process Memory
Query Execution Work Areas 10 Work Area Size Manual Management Auto Management hash_area_size sort_area_size bitmap_merge_area_size create_bitmap_area_size pga_aggregate_target
PGA Memory Limits 11
Manual Work Area Management 12 SQL> alter session set workarea_size_policy=manual; Session altered SQL> alter session set hash_area_size=4294967296; -- 4GB alter session set hash_area_size=4294967296 ORA-02017: integer value required  11.2.0.4 Linux x64 SQL> alter session set hash_area_size=2147483648; -- 2GB alter session set hash_area_size=2147483648 ORA-02017: integer value required SQL> alter session set hash_area_size=2147483647; -- 2GB - 1 Session altered  32-bit Signed Integer Limit
Auto Work Area Management 13 SQL> alter system set pga_aggregate_target=1024g; -- 1TB System altered  11.2.0.4 Linux x64  Where is the catch?  PGA_AGGREGATE_TARGET  _smm_max_size/_smm_max_size_static  Maximum work area size per process (px/serial)  _smm_px_max_size/_smm_px_max_size_static  Maximum work area size per query (px)  _pga_max_size  Maximum PGA size per process (px/serial)
Auto Work Area Management 14  PGA_AGGREGATE_TARGET 16M – 512M 0 100 200 300 400 500 600 16 32 64 128 256 512 pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size  _pga_max_size = 200M  _smm_max_size[_static] = 20%  _smm_px_max_size[_static] = 50%
Auto Work Area Management 15  PGA_AGGREGATE_TARGET 1G – 10G  _pga_max_size = 20%  _smm_max_size[_static] = 10%  _smm_px_max_size[_static] = 50% 0 2000 4000 6000 8000 10000 12000 pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size
Auto Work Area Management 16  PGA_AGGREGATE_TARGET 10G – 16G  _pga_max_size = 2G  _smm_max_size[_static] = 2G  _smm_px_max_size[_static] = 50% 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 10240 10752 11264 11776 12288 12800 13312 13824 14336 14848 15360 15872 16384 pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size
Auto Work Area Management 17  PGA_AGGREGATE_TARGET >= 10G  _smm_max_size[_static] = 1G (maximum value)  _pga_max_size = 2G (maximum value)  If you’re bumping into per process limits further rising pga_aggregate_target will not help
Per Process Limit 18  PGA_AGGREGATE_TARGET = 192G / DOP = 16  PGA_AGGREGATE_TARGET = 512G / DOP = 16
Per Process Limit 19  PGA_AGGREGATE_TARGET = 192G / DOP = 16  PGA_AGGREGATE_TARGET = 512G / DOP = 16  _pga_max_size * DOP
Run With Higher DOP? 20  Exadata X3-8  2TB RAM (per node)  80 CPU cores (per node)  PGA_AGGREGATE_TARGET = 1536G  Would require at least DOP 768 (_pga_max_size)  9.6x core count  Concurrency issues  Manageability issues  PX data distribution/algorithm issues
Run With Higher DOP? 21  DOP vs CPU Cores Used 5 (spill) 8 (spill) 64 (no spill) 0 16 32 48 64 16 32 64 CPUCoresUsed DOP
Run With Higher DOP? 22  PX Algorithm Issues (ex.: median function)
Play With Underscores? 23  MOS Doc ID 453540.1  Allows _smm_max_size=2097151  Allows _pga_max_size > 2GB with patch 17951233  Can get 4G per process limit  Can get > 4G per process limit  /proc/sys/vm/max_map_count (OS page count)  _realfree_heap_pagesize_hint (allocator page size)  Weird behavior and likely not fully supported for work areas (MOS Doc ID 453540.1)
Radically Improve TEMP I/O? 24  TEMP I/O (in general)  Doesn’t need redundancy  Doesn’t need persistency  Doesn’t need recoverability  In-Memory FS (tmpfs, etc.)  SAN/NAS LUN with write-back cache
Linux tmpfs (via loop device) 25  10.1B rows / 416G HCC (QUERY HIGH) SELECT /*+ parallel(t,16) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy') GROUP BY CUST_ID, DATE_ID; Exadata X3-8 TEMP tmpfs TEMP  13.8x faster TEMP I/O (237G)
Linux tmpfs (via loop device) 26  10.1B rows / 416G HCC (QUERY HIGH) SELECT /*+ parallel(t,16) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy') GROUP BY CUST_ID, DATE_ID;
ZFSSA (Infiniband SRP) 27  7.3B rows / 300G HCC (QUERY HIGH) SELECT /*+ parallel(t,32) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T GROUP BY CUST_ID, DATE_ID; Exadata X3-8 TEMP ZFSSA TEMP (write-back cache)  64.3x faster TEMP I/O (141G)
ZFSSA (Infiniband SRP) 28  7.3B rows / 300G HCC (QUERY HIGH) SELECT /*+ parallel(t,32) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T GROUP BY CUST_ID, DATE_ID;
Summary 29  Remember about per process limits  Larger PGA_AGGREGATE_TARGET may do nothing  Balance PGA/DOP/CPU  It’s possible to increase TEMP I/O performance by 10x- 50x and more  Oracle PGA code does not fully embrace 64-bit systems at the moment
Q & A Email: afatkulin@enkitec.com Blog: http://afatkulin.blogspot.com 30

More Related Content

Fatkulin hotsos 2014

  • 1. Leveraging In-Memory Storage to Overcome Oracle PGA Memory Limits March 5, 2014 Presented by: Alex Fatkulin Senior Consultant
  • 2. Who am I ?  Senior Technical Consultant at Enkitec  12 years using Oracle  Clustered and HA solutions  Database Development and Design  Technical Reviewer  Blog at http://afatkulin.blogspot.com 3
  • 4. Data Growth and Processing  Data Volumes UP  Processing Power UP  Database Design (in general) DOWN 5 Data Volume Processing Power Database Design BetterWorse
  • 5. 6 How many people have systems with more than 128G of RAM?
  • 6. “640K” Problems 7  Processing patterns change  Things that didn’t matter before matter now  Legacy RDBMS code vs 64-bit systems  KIWI (Kill It With Iron)
  • 8. PGA Memory (Dedicated Server) 9 Sort AreaHash Area Bitmap Area SQL Work Areas Session Memory Private SQL Area Process Memory
  • 9. Query Execution Work Areas 10 Work Area Size Manual Management Auto Management hash_area_size sort_area_size bitmap_merge_area_size create_bitmap_area_size pga_aggregate_target
  • 11. Manual Work Area Management 12 SQL> alter session set workarea_size_policy=manual; Session altered SQL> alter session set hash_area_size=4294967296; -- 4GB alter session set hash_area_size=4294967296 ORA-02017: integer value required  11.2.0.4 Linux x64 SQL> alter session set hash_area_size=2147483648; -- 2GB alter session set hash_area_size=2147483648 ORA-02017: integer value required SQL> alter session set hash_area_size=2147483647; -- 2GB - 1 Session altered  32-bit Signed Integer Limit
  • 12. Auto Work Area Management 13 SQL> alter system set pga_aggregate_target=1024g; -- 1TB System altered  11.2.0.4 Linux x64  Where is the catch?  PGA_AGGREGATE_TARGET  _smm_max_size/_smm_max_size_static  Maximum work area size per process (px/serial)  _smm_px_max_size/_smm_px_max_size_static  Maximum work area size per query (px)  _pga_max_size  Maximum PGA size per process (px/serial)
  • 13. Auto Work Area Management 14  PGA_AGGREGATE_TARGET 16M – 512M 0 100 200 300 400 500 600 16 32 64 128 256 512 pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size  _pga_max_size = 200M  _smm_max_size[_static] = 20%  _smm_px_max_size[_static] = 50%
  • 14. Auto Work Area Management 15  PGA_AGGREGATE_TARGET 1G – 10G  _pga_max_size = 20%  _smm_max_size[_static] = 10%  _smm_px_max_size[_static] = 50% 0 2000 4000 6000 8000 10000 12000 pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size
  • 15. Auto Work Area Management 16  PGA_AGGREGATE_TARGET 10G – 16G  _pga_max_size = 2G  _smm_max_size[_static] = 2G  _smm_px_max_size[_static] = 50% 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 10240 10752 11264 11776 12288 12800 13312 13824 14336 14848 15360 15872 16384 pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size
  • 16. Auto Work Area Management 17  PGA_AGGREGATE_TARGET >= 10G  _smm_max_size[_static] = 1G (maximum value)  _pga_max_size = 2G (maximum value)  If you’re bumping into per process limits further rising pga_aggregate_target will not help
  • 17. Per Process Limit 18  PGA_AGGREGATE_TARGET = 192G / DOP = 16  PGA_AGGREGATE_TARGET = 512G / DOP = 16
  • 18. Per Process Limit 19  PGA_AGGREGATE_TARGET = 192G / DOP = 16  PGA_AGGREGATE_TARGET = 512G / DOP = 16  _pga_max_size * DOP
  • 19. Run With Higher DOP? 20  Exadata X3-8  2TB RAM (per node)  80 CPU cores (per node)  PGA_AGGREGATE_TARGET = 1536G  Would require at least DOP 768 (_pga_max_size)  9.6x core count  Concurrency issues  Manageability issues  PX data distribution/algorithm issues
  • 20. Run With Higher DOP? 21  DOP vs CPU Cores Used 5 (spill) 8 (spill) 64 (no spill) 0 16 32 48 64 16 32 64 CPUCoresUsed DOP
  • 21. Run With Higher DOP? 22  PX Algorithm Issues (ex.: median function)
  • 22. Play With Underscores? 23  MOS Doc ID 453540.1  Allows _smm_max_size=2097151  Allows _pga_max_size > 2GB with patch 17951233  Can get 4G per process limit  Can get > 4G per process limit  /proc/sys/vm/max_map_count (OS page count)  _realfree_heap_pagesize_hint (allocator page size)  Weird behavior and likely not fully supported for work areas (MOS Doc ID 453540.1)
  • 23. Radically Improve TEMP I/O? 24  TEMP I/O (in general)  Doesn’t need redundancy  Doesn’t need persistency  Doesn’t need recoverability  In-Memory FS (tmpfs, etc.)  SAN/NAS LUN with write-back cache
  • 24. Linux tmpfs (via loop device) 25  10.1B rows / 416G HCC (QUERY HIGH) SELECT /*+ parallel(t,16) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy') GROUP BY CUST_ID, DATE_ID; Exadata X3-8 TEMP tmpfs TEMP  13.8x faster TEMP I/O (237G)
  • 25. Linux tmpfs (via loop device) 26  10.1B rows / 416G HCC (QUERY HIGH) SELECT /*+ parallel(t,16) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy') GROUP BY CUST_ID, DATE_ID;
  • 26. ZFSSA (Infiniband SRP) 27  7.3B rows / 300G HCC (QUERY HIGH) SELECT /*+ parallel(t,32) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T GROUP BY CUST_ID, DATE_ID; Exadata X3-8 TEMP ZFSSA TEMP (write-back cache)  64.3x faster TEMP I/O (141G)
  • 27. ZFSSA (Infiniband SRP) 28  7.3B rows / 300G HCC (QUERY HIGH) SELECT /*+ parallel(t,32) */ CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D FROM TRANS T GROUP BY CUST_ID, DATE_ID;
  • 28. Summary 29  Remember about per process limits  Larger PGA_AGGREGATE_TARGET may do nothing  Balance PGA/DOP/CPU  It’s possible to increase TEMP I/O performance by 10x- 50x and more  Oracle PGA code does not fully embrace 64-bit systems at the moment
  • 29. Q & A Email: afatkulin@enkitec.com Blog: http://afatkulin.blogspot.com 30