Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DefUse is consuming lots of memory #5061

Open
asl opened this issue Dec 12, 2024 · 2 comments
Open

DefUse is consuming lots of memory #5061

asl opened this issue Dec 12, 2024 · 2 comments
Labels
compiler-performance Topics on improving the performance of the compiler core. core Topics concerning the core segments of the compiler (frontend, midend, parser)

Comments

@asl
Copy link
Contributor

asl commented Dec 12, 2024

I am constantly seeing use-def to consume lots of memory. In many cases I'm seeing behavior like:

      heap after P4::ComputeWriteSet: in use 6.4GB, max 6.4GB
      ProcessDefUse invoking P4::(anonymous namespace)::FindUninitialized

or

      ProcessDefUse invoking P4::ComputeWriteSet
      heap after P4::ComputeWriteSet: in use 4.0GB, max 4.2GB
      ProcessDefUse invoking P4::(anonymous namespace)::FindUninitialized

In one case I had to kill p4c process as it started to consume more than 16 Gb of RAM inside ProcessDefUse.

Also, ProcessDefUse usually takes 30-40% of all frontend time (timing minutes and tens of minutes).
@asl asl added core Topics concerning the core segments of the compiler (frontend, midend, parser) compiler-performance Topics on improving the performance of the compiler core. labels Dec 12, 2024
@asl
Copy link
Contributor Author

asl commented Dec 12, 2024

Tagging @ChrisDodd

Is there something that could be improved? Could midend def-use be reused if it has better performance?

@asl
Copy link
Contributor Author

asl commented Dec 12, 2024

Allocation tracing looks like as follows:

Allocated a total of 3.6GB memory
allocated 1.7GB in 15321 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa84bc _ZN2P415ComputeWriteSet8preorderEPKNS_2IR19AssignmentStatementE + 608 0x102aa84bc
  2   p4c                        0x0000000102aa0698 _ZNK2P411Definitions6writesENS_12ProgramPointERKNS_11LocationSetE + 96 0x102aa0698
  1   p4c                        0x0000000102aae618 _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 92 0x102aae618
allocated 583MB in 5446334 calls from:
  4   p4c                        0x0000000102a9f2b0 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 440 0x102a9f2b0
  3   p4c                        0x0000000102aacf48 _ZN4absl12lts_2024011618container_internal12raw_hash_setINS1_17FlatHashSetPolicyIN2P412ProgramPointEEENS4_4Util4HashENSt3__18equal_toIS5_EENS9_9allocatorIS5_EEEC2ERKSE_RKSD_ + 172 0x102aacf48
  2   p4c                        0x0000000102aad300 _ZN4absl12lts_2024011618container_internal12raw_hash_setINS1_17FlatHashSetPolicyIN2P412ProgramPointEEENS4_4Util4HashENSt3__18equal_toIS5_EENS9_9allocatorIS5_EEE6resizeEm + 80 0x102aad300
  1   p4c                        0x0000000102aad610 _ZN4absl12lts_2024011618container_internal19HashSetResizeHelper15InitializeSlotsINSt3__19allocatorIcEELm80ELb0ELm8EEEbRNS1_12CommonFieldsEPvT_ + 56 0x102aad610
allocated 502MB in 15321 calls from:
  4   p4c                        0x0000000102aa84bc _ZN2P415ComputeWriteSet8preorderEPKNS_2IR19AssignmentStatementE + 608 0x102aa84bc
  3   p4c                        0x0000000102aa0698 _ZNK2P411Definitions6writesENS_12ProgramPointERKNS_11LocationSetE + 96 0x102aa0698
  2   p4c                        0x0000000102aae5dc _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 32 0x102aae5dc
  1   p4c                        0x0000000102ce4fd0 _ZN2P416hash_vector_baseC2ERKS0_ + 236 0x102ce4fd0
allocated 179MB in 9548 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa738c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 572 0x102aa738c
  2   p4c                        0x0000000102a9f358 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 608 0x102a9f358
  1   p4c                        0x0000000102a9f650 _ZN2P48hvec_mapIPKNS_2IR12IDeclarationEPKNS_15StorageLocationENSt3__14hashIS4_EENS8_8equal_toIS4_EENS8_9allocatorINS8_4pairIKS4_S7_EEEEE7emplaceIJRPS5_EEENSE_INSI_5_iterISI_SG_EEbEERSF_DpOT_ + 304 0x102a9f650
allocated 144MB in 3596419 calls from:
  4   p4c                        0x0000000102cb15b8 _ZN2P47Visitor5visitERKPKNS_2IR10StatOrDeclEPKc + 36 0x102cb15b8
  3   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  2   p4c                        0x0000000102aa738c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 572 0x102aa738c
  1   p4c                        0x0000000102a9f298 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 416 0x102a9f298
allocated  74MB in 682 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa7304 _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 436 0x102aa7304
  2   p4c                        0x0000000102aa0698 _ZNK2P411Definitions6writesENS_12ProgramPointERKNS_11LocationSetE + 96 0x102aa0698
  1   p4c                        0x0000000102aae618 _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 92 0x102aae618
...

and

Allocated a total of 6.9GB memory
allocated 2.0GB in 14324302 calls from:
  4   p4c                        0x0000000102a9f2b0 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 440 0x102a9f2b0
  3   p4c                        0x0000000102aacf48 _ZN4absl12lts_2024011618container_internal12raw_hash_setINS1_17FlatHashSetPolicyIN2P412ProgramPointEEENS4_4Util4HashENSt3__18equal_toIS5_EENS9_9allocatorIS5_EEEC2ERKSE_RKSD_ + 172 0x102aacf48
  2   p4c                        0x0000000102aad300 _ZN4absl12lts_2024011618container_internal12raw_hash_setINS1_17FlatHashSetPolicyIN2P412ProgramPointEEENS4_4Util4HashENSt3__18equal_toIS5_EENS9_9allocatorIS5_EEE6resizeEm + 80 0x102aad300
  1   p4c                        0x0000000102aad610 _ZN4absl12lts_2024011618container_internal19HashSetResizeHelper15InitializeSlotsINSt3__19allocatorIcEELm80ELb0ELm8EEEbRNS1_12CommonFieldsEPvT_ + 56 0x102aad610
allocated 1.0GB in 6640 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa84bc _ZN2P415ComputeWriteSet8preorderEPKNS_2IR19AssignmentStatementE + 608 0x102aa84bc
  2   p4c                        0x0000000102aa0698 _ZNK2P411Definitions6writesENS_12ProgramPointERKNS_11LocationSetE + 96 0x102aa0698
  1   p4c                        0x0000000102aae618 _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 92 0x102aae618
allocated 685MB in 20100 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa738c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 572 0x102aa738c
  2   p4c                        0x0000000102a9f358 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 608 0x102a9f358
  1   p4c                        0x0000000102a9f650 _ZN2P48hvec_mapIPKNS_2IR12IDeclarationEPKNS_15StorageLocationENSt3__14hashIS4_EENS8_8equal_toIS4_EENS8_9allocatorINS8_4pairIKS4_S7_EEEEE7emplaceIJRPS5_EEENSE_INSI_5_iterISI_SG_EEbEERSF_DpOT_ + 304 0x102a9f650
allocated 642MB in 18789 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa8e7c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR8P4ActionE + 1252 0x102aa8e7c
  2   p4c                        0x0000000102a9f46c _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 884 0x102a9f46c
  1   p4c                        0x0000000102a9f650 _ZN2P48hvec_mapIPKNS_2IR12IDeclarationEPKNS_15StorageLocationENSt3__14hashIS4_EENS8_8equal_toIS4_EENS8_9allocatorINS8_4pairIKS4_S7_EEEEE7emplaceIJRPS5_EEENSE_INSI_5_iterISI_SG_EEbEERSF_DpOT_ + 304 0x102a9f650
allocated 513MB in 12813082 calls from:
  4   p4c                        0x0000000102cb15b8 _ZN2P47Visitor5visitERKPKNS_2IR10StatOrDeclEPKc + 36 0x102cb15b8
  3   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  2   p4c                        0x0000000102aa738c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 572 0x102aa738c
  1   p4c                        0x0000000102a9f298 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 416 0x102a9f298
allocated 269MB in 6640 calls from:
  4   p4c                        0x0000000102aa84bc _ZN2P415ComputeWriteSet8preorderEPKNS_2IR19AssignmentStatementE + 608 0x102aa84bc
  3   p4c                        0x0000000102aa0698 _ZNK2P411Definitions6writesENS_12ProgramPointERKNS_11LocationSetE + 96 0x102aa0698
  2   p4c                        0x0000000102aae5dc _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 32 0x102aae5dc
  1   p4c                        0x0000000102ce4fd0 _ZN2P416hash_vector_baseC2ERKS0_ + 236 0x102ce4fd0
allocated 218MB in 1400 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aaaa7c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR19MethodCallStatementE + 156 0x102aaaa7c
  2   p4c                        0x0000000102aa0698 _ZNK2P411Definitions6writesENS_12ProgramPointERKNS_11LocationSetE + 96 0x102aa0698
  1   p4c                        0x0000000102aae618 _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 92 0x102aae618
allocated 217MB in 1693976 calls from:
  4   p4c                        0x0000000102aa738c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 572 0x102aa738c
  3   p4c                        0x0000000102a9f2b0 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 440 0x102a9f2b0
  2   p4c                        0x0000000102aad0f0 _ZN4absl12lts_2024011618container_internal12raw_hash_setINS1_17FlatHashSetPolicyIN2P412ProgramPointEEENS4_4Util4HashENSt3__18equal_toIS5_EENS9_9allocatorIS5_EEEC2ERKSE_RKSD_ + 596 0x102aad0f0
  1   p4c                        0x0000000102aad6ec _ZN4absl12lts_2024011623inlined_vector_internal7StorageIPKN2P415StorageLocationELm8ENSt3__19allocatorIS6_EEE8InitFromERKSA_ + 76 0x102aad6ec
allocated 210MB in 1345 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa7304 _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 436 0x102aa7304
  2   p4c                        0x0000000102aa0698 _ZNK2P411Definitions6writesENS_12ProgramPointERKNS_11LocationSetE + 96 0x102aa0698
  1   p4c                        0x0000000102aae618 _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 92 0x102aae618
allocated 207MB in 1328 calls from:
  4   p4c                        0x0000000102c7d678 _ZN2P49Inspector13apply_visitorEPKNS_2IR4NodeEPKc + 636 0x102c7d678
  3   p4c                        0x0000000102aa8eec _ZN2P415ComputeWriteSet8preorderEPKNS_2IR8P4ActionE + 1364 0x102aa8eec
  2   p4c                        0x0000000102aa1b44 _ZN2P415ComputeWriteSet9exitScopeEPKNS_2IR13ParameterListEPKNS1_13IndexedVectorINS1_11DeclarationEEENS_12ProgramPointE + 340 0x102aa1b44
  1   p4c                        0x0000000102aae618 _ZN2P48hvec_mapIPKNS_12BaseLocationEPKNS_13ProgramPointsENSt3__14hashIS3_EENS7_8equal_toIS3_EENS7_9allocatorINS7_4pairIKS3_S6_EEEEEC2ERKSH_ + 92 0x102aae618
allocated 105MB in 13735 calls from:
  4   p4c                        0x0000000102aa738c _ZN2P415ComputeWriteSet8preorderEPKNS_2IR11IfStatementE + 572 0x102aa738c
  3   p4c                        0x0000000102a9f358 _ZNK2P411Definitions15joinDefinitionsEPKS0_ + 608 0x102a9f358
  2   p4c                        0x0000000102a9f550 _ZN2P48hvec_mapIPKNS_2IR12IDeclarationEPKNS_15StorageLocationENSt3__14hashIS4_EENS8_8equal_toIS4_EENS8_9allocatorINS8_4pairIKS4_S7_EEEEE7emplaceIJRPS5_EEENSE_INSI_5_iterISI_SG_EEbEERSF_DpOT_ + 48 0x102a9f550
  1   p4c                        0x0000000102ce603c _ZN2P416hash_vector_base9hv_insertEPKvPNS0_12lookup_cacheE + 800 0x102ce603c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler-performance Topics on improving the performance of the compiler core. core Topics concerning the core segments of the compiler (frontend, midend, parser)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@asl