It's hard to tell how the MPS is performing

[mgood7123]

how would i go about obtaining GC statistics (eg, for profiling GC)

additionally how would we handle object resurrection ?

(eg, an object is assumed to have been collected, but is infact still live due to resurrection, leading to incorrect statistics related to live/dead objects)

at the moment i have the following

void managed_obj_print_stats(ManagedObjState * state) {
    size_t size_total;
    size_t size_free;
    size_t size_used;

    printf("Stats:\n");
    printf("  Memory:\n");
    printf("    Objects (malloc):\n");
    printf("      allocated: %zu\n", state->allocated_bytes);
    printf("      freed:     %zu\n", state->freed_bytes);
    printf("    Objects (gc):\n");
    printf("      allocated:           %zu\n", state->allocated_obj_bytes);
    printf("      freed:               %zu\n", state->freed_obj_bytes);
    printf("      (aligned) allocated: %zu\n", state->allocated_aligned_obj_bytes);
    printf("      (aligned) freed:     %zu\n", state->freed_aligned_obj_bytes);

    // TODO: figure out exactly what these mean, some are obvious, some are less obvious

    printf("    Pool Allocation Point:\n");
    printf("      init: %p\n", state->ap->init);
    printf("      alloc: %p\n", state->ap->alloc);
    printf("      limit: %p\n", state->ap->limit);
    printf("    Pool:\n");
    printf("      Used:  %zu\n", mps_pool_total_size(state->pool) - mps_pool_free_size(state->pool));
    printf("      Free:  %zu\n", mps_pool_free_size(state->pool));
    printf("      Total: %zu\n", mps_pool_total_size(state->pool));
    printf("    Arena:\n");
    printf("      Reserved:           %zu\n", mps_arena_reserved(state->arena));
    printf("      Commited:           %zu\n", mps_arena_committed(state->arena));
    if (mps_arena_commit_limit(state->arena) == -1) {
    printf("      Commit Limit:       %zu\n", mps_arena_reserved(state->arena));
    } else {
    printf("      Commit Limit:       %zu\n", mps_arena_commit_limit(state->arena));
    }
    printf("      Spare:              %g\n", mps_arena_spare(state->arena));
    printf("      Spare Commited:     %zu\n", mps_arena_spare_committed(state->arena));
    if (mps_arena_spare_commit_limit(state->arena) == -1) {
    printf("      Spare Commit Limit: Infinite\n");
    } else {
    printf("      Spare Commit Limit: %zu\n", mps_arena_spare_commit_limit(state->arena));
    }
    printf("      Pause Time:         %g\n", mps_arena_pause_time(state->arena));
}

allocated_bytes

void * managed_obj_malloc(ManagedObjState * state, size_t s) {
  void * p = malloc(s);
  if (p != NULL) {
    state->allocated_bytes += s;
  }
  return p;
}

freed_bytes

void managed_obj_free(ManagedObjState * state, void * p, size_t s) {
  if (p != NULL) {
    state->freed_bytes += s;
  }
  free(p);
}

allocated_obj_bytes/allocated_aligned_obj_bytes

managed_obj_t managed_obj_make_scanned_with_finalizer(ManagedObjState * state, void * pointer, managed_obj_scanned_pointer_scan_fn_t scanner, managed_obj_finalization_callback_t finalization_callback)
{
  managed_obj_t obj;
  size_t size = MANAGED_OBJECT_ALIGN_OBJ(sizeof(managed_obj_scanned_pointer_s));
  while(1) {
    mps_res_t res = mps_reserve((mps_addr_t*)&obj, state->ap, size);
    if (res != MPS_RES_OK) managed_obj_error("out of memory in make_pointer");
    obj->scanned_pointer.type = MANAGED_OBJECT_TYPE_SCANNED_POINTER;
    obj->scanned_pointer.pointer = pointer;
    obj->scanned_pointer.scanner = scanner;
    obj->scanned_pointer.finalization_callback = finalization_callback;
    if (mps_commit(state->ap, obj, size)) {
      break;
    }
  }
  state->allocated_obj_bytes += sizeof(managed_obj_scanned_pointer_s);
  state->allocated_aligned_obj_bytes += size;

  printf("reserved and comitted object %p (with size %zu, aligned size %zu) with pointer %p\n", obj, sizeof(managed_obj_scanned_pointer_s), size, obj->scanned_pointer.pointer);

  managed_obj_print_stats(state);

  mps_finalize(state->arena, (mps_addr_t*)&obj);

  return obj;
}

freed_obj_bytes/freed_aligned_obj_bytes

void managed_obj_mps_chat(ManagedObjState * state) {
  mps_message_type_t type;

  while (mps_message_queue_type(&type, state->arena)) {
    mps_message_t message;
    mps_bool_t b;
    b = mps_message_get(&message, state->arena, type);
    AVER(b); /* we just checked there was one */

    if (type == mps_message_type_gc_start()) {
      printf("\nCollection start %d due to '%s'\n", ++nStart, mps_message_gc_start_why(state->arena, message));

    } else if (type == mps_message_type_gc()) {

      // should these be part of stats?
      // we currently dont know exactly what they mean in terms of GC profiling

      size_t live, condemned, not_condemned;

      live = mps_message_gc_live_size(state->arena, message);
      condemned = mps_message_gc_condemned_size(state->arena, message);
      not_condemned = mps_message_gc_not_condemned_size(state->arena, message);

      printf("\nCollection complete %d:\n", ++nComplete);
      printf("  live %zu\n", live);
      printf("  condemned %zu\n", condemned);
      printf("  not_condemned %zu\n", not_condemned);

     } else if (type == mps_message_type_finalization()) {
      /* A finalization message is received when an object registered earlier
        with `mps_finalize` would have been recycled if it hadn't been
        registered. This means there are no other references to the object.
        Note, however, that finalization isn't reliable or prompt.
        Treat it as an optimization. See topic/finalization. */

    // TODO: in finalization we assume the user will supply a finalizer that will not resurrect the object

      managed_obj_t obj;

      mps_message_finalization_ref((mps_addr_t*)&obj, state->arena, message);

      AVER(MANAGED_OBJECT_TYPE(obj) == MANAGED_OBJECT_TYPE_SCANNED_POINTER || MANAGED_OBJECT_TYPE(obj) == MANAGED_OBJECT_TYPE_DYNAMIC_POINTER);

      if(MANAGED_OBJECT_TYPE(obj) == MANAGED_OBJECT_TYPE_SCANNED_POINTER) {
        state->freed_obj_bytes += sizeof(managed_obj_scanned_pointer_s);
        state->freed_aligned_obj_bytes += MANAGED_OBJECT_ALIGN_OBJ(sizeof(managed_obj_scanned_pointer_s));
        if (obj->scanned_pointer.pointer) {
          if (obj->scanned_pointer.finalization_callback) {
            printf("object %p with pointer %p is being finalized.\n", obj, obj->scanned_pointer.pointer);
            
            // TODO: the user could resurrect the object at field 'obj->scanned_pointer.pointer'
            // TODO: the field 'obj->scanned_pointer.pointer' could contain a GC object not finalized yet
            // TODO: the field 'obj->scanned_pointer.pointer' could contain a GC object that is finalized but not yet processed

            obj->scanned_pointer.finalization_callback(state, obj->scanned_pointer.pointer);
          }
          printf("object %p with pointer %p has been freed, setting pointer to zero.\n", obj, obj->scanned_pointer.pointer);
          obj->scanned_pointer.pointer = NULL;
        } else {
          // this could happen if a user explicitly sets a field to NULL
          printf("WARNING: object %p has already been freed.\n", obj);
        }
      }
    // } else {
      // printf("Unknown message from MPS!\n");
    }

    mps_message_discard(state->arena, message);
  }
}

[rptb1]

A few pointers.

You can retrieve some statistics from mps_message_type_gc().

You can gain a very large amount of information via Telemetry.

The MPS can collect a lot of statistics in a cool build.

We are currently investigating ways to improve the availability of statistics with our commercial clients.

(eg, an object is assumed to have been collected, but is infact still live due to resurrection, leading to incorrect statistics related to live/dead objects)

There is no resurrection in the MPS. Dead objects are recycled. Objects registered for finalization do not die until their finalization messages are deleted. I think we could improve the wording of Finalization. Finalization happens before death, not after it.

[mgood7123]

An object can be resurrected during finalization tho, right?

Eg the finalizer could store the reference in a static root or something , thus "ressurrecting" it

[rptb1]

Objects registered for finalization cannot die until their finalization message is deleted. You can copy a reference to those objects out of the finalization message before deleting the message, if you like. You can think of this as "resurrection" if you like, but it's important to realize that as far as the MPS is concerned, the object is alive. It will be counted as alive for all purposes, including statistics.

[mgood7123]

Objects registered for finalization cannot die until their finalization message is deleted. You can copy a reference to those objects out of the finalization message before deleting the message, if you like. You can think of this as "resurrection" if you like, but it's important to realize that as far as the MPS is concerned, the object is alive. It will be counted as alive for all purposes, including statistics.

yea

[mgood7123]

so, how can we know when an object is truly dead as far as MPS is concerned?

[rptb1]

You could create a weak reference to the object. When that reference gets zeroed, the MPS has determined that it is not reachable from any root (except via weak references). What happens next depends on the pool class, but in the case of AMC the memory block that contains the object will be recycled, and may even be unmapped and returned to the OS for use by other processes, depending on memory pressure.

[rptb1]

We do not currently have a system whereby you can receive a message when a weak reference is zeroed. In theory you could modify the Manual Rank Guardian pool class to have a weak variant, but you'd need to register some sort of ID with each reference so you could work out which one had gone away.

It's not clear to me what the use case for that is though. Do you have one?

[mgood7123]

It's not clear to me what the use case for that is though. Do you have one?

this would be related to #262

[mgood7123]

You could create a weak reference to the object. When that reference gets zeroed, the MPS has determined that it is not reachable from any root (except via weak references). What happens next depends on the pool class, but in the case of AMC the memory block that contains the object will be recycled, and may even be unmapped and returned to the OS for use by other processes, depending on memory pressure.

Tho this would be called when ANY weak reference is zeroed

Eg, if two weak references to the same reference get zeroed, then said message would be posted 2 times, once for each weak reference

[rptb1]

The text you quoted from me does not mention calling anything or any messages. Are you talking about something you have implemented on top of your weak tables?

[rptb1]

It's not clear to me what the use case for that is though. Do you have one?

this would be related to #262

Do you have a specific use case for wanting to know after an object has died? An example?

[mgood7123]

We do not currently have a system whereby you can receive a message when a weak reference is zeroed. In theory you could modify the Manual Rank Guardian pool class to have a weak variant, but you'd need to register some sort of ID with each reference so you could work out which one had gone away.

It's not clear to me what the use case for that is though. Do you have one?

never mind