LNXSDK/Kha/Backends/Kinc-hxcpp/khacpp/docs/ThreadsAndStacks.md

Threads And Stacks

Conservative, co-operation

Hxcpp uses conservative stop-the-world GC, where the threads need to co-operate.

• Threads must not change GC pointers in the collection phase
• The thread stacks/registers must be scanned for GC pointers
• Threads must not block without letting the GC system know not to wait for them, otherwise GC blocks until end of block
  • call hx::GCEnterBlocking() / gc_enter_blocking() / (cpp.vm.Gc.enterGCFreeZone() from Haxe) before potentially blocking system call (fs, network, etc)
  • call hx::GCExitBlocking() / gc_exit_blocking() / (cpp.vm.Gc.exitGCFreeZone() from Haxe) before making more GC calls
  • Might need to pre-allocate buffers
  • Don't forget the exit blocking on error condition

Foreign Threads

When you create a thread from haxe, it starts attached. Before a non-haxe created thread can interact with hxcpp, some care must be taken, since GC allocations are done using a GC context per thread, and all threads must respect the stopped world.

• Foreign threads must be attached-detached
  • SetTopOfStack(int * inTop,bool inPush)
  • inTop = pointer to top of stack to attach, or '0' to remove stack
  • inPush = usually true. recursive attachment/detachment
• Must not change things when the world is stopped
• Must define their stack range for scanning
• If you are attached, you may need to enter/exit gc free zone
• Must release context when done, if no more calls are going to be made
• Make sure local variables are covered in stack
  • compiler may reorder, so be careful
• Read documentation because some things, eg audio callbacks, happen on other threads
• You can use other techniques, eg
  • create a haxe thread, which blocks waiting for signal
  • foreign thread generates request and signals haxe thread
  • haxe thread performs job and generates data then signals foreign thread
  • foreign picks up data and carries on

Top of Stack

• To understand how to handle threads, you need a mental picture of the c++ stack
• The stack usually goes "down". That is, if the first stack location is 10000, the next one will be 9999 etc.
• Historical, but consistent. Except for emscripten which goes up - but still use same terminology/picture, just change the less-thans to greater-thans in code.

Say the system starts each program stack at 10000, the stack might look like this, with local variables and arguments pushed on the stack:

 10000 
 -----------------------------------------------
 9996  startup temp variable
 9992  startup temp variable
       -- main function --
 9988  main return address    - order and details of this are ABI specific
 9984  char ** argv
 9980  int     argc

Hxcpp then runs it main code, which starts with the macro HX_TOP_OF_STACK, which expands to something like:

   int t0 = 99;
   hx::SetTopOfStack(&t0,false);
   ...
   __boot_all();
   __hxcpp_main();

       -- main function --
 9988  main return address      order and details of this are ABI specific
 9984  char ** argv
 9980  int     argc
 9976  int     t0
       -- hx::SetTopOfStack --

    records '9976' as top of stack for this thread

Later, many generated functions deep, __hxcpp_main generates an allocation call which triggers a collection

 ...
 8100  Array<Bullet>   bullets
       -- alloc Enemy --
 ...
       -- Call collect --

 8050 int   bottomOfStackTemp
      MarkConservative(&bottomOfStackTemp, 9976) -> scans stack from 8050 -> 9976
      MarkConservative(Capture registers)

Enter/exit use similar technique, where the registers are captured and the bottomOfStack is 'locked-in' when the "enter gc free zone" call is made.

 8100  Array<Bullet>   bullets
       -- EnterGCFreeZone --
 8088 int   bottomOfStackTemp
      thread->setBottomOfStack(&bottomOfStackTemp)
      thread->captureRegisters()
      return
      * any changes here will not affect GC

Now, when another thread does a collection, the gc-free thread can be scanned from 8088 to 9976, regardless of any stuff happening lower dowsn the stack.

Not Called From Main

Top of stack can be tricky to get right when a gui framework does not really have a "main".

 10000 
 -----------------------------------------------
 9996  startup temp variable
 9992  startup temp variable
       -- main function --
       setupWindows(onReadyCallback)......
          ...
 8000  
       -- onReadyCallback --
 7976  int     t0
       SetTopOfStack(&t0,false) -> 7966
       __hxcpp_main();
          setOnFrameCallack(haxeOnFrame)
          return;

Later, the haxeOnFrame callback is trigger, but not "below" __hxcpp_main

 9800  -- haxeOnFrame ---
     // Top of stack will be below bottom of stack.

Solutions:

• Make sure you get in at top of main
  • may scan too much?
• Ratchet up top-of-stack in callbacks, inForce = false
  • gc_set_top_of_stack(void * inTopOfStack,bool inForce);
• Detach main thread after hxcpp_main and reattach each callback
  • android solution because render callbacks happen on different threads
  • gc_set_top_of_stack(&base,true); // attach
  • gc_set_top_of_stack(0,true); // detach

Debugging.

• in debug mode, hxcpp will check for calls from unattached threads
• hxcpp can log conservative ranges. With a native debugger you can check the address of your local variables and ensure they are included.
• hxcpp will scan native objects on the stack, but will not follow non-haxe pointers to other objects, so additional GC roots may be required.