* Optimize `TryAllocateRegWithtoutSpill` a bit
* Add a fast path for when all registers are live.
* Do not query `GetOverlapPosition` if the register is already in use
(i.e: free position is 0).
* Do not allocate child split list if not parent
* Turn `LiveRange` into a reference struct
`LiveRange` is now a reference wrapping struct like `Operand` and
`Operation`.
It has also been changed into a singly linked-list. In micro-benchmarks
traversing the linked-list was faster than binary search on `List<T>`.
Even for quite large input sizes (e.g: 1,000,000), surprisingly.
Could be because the code gen for traversing the linked-list is much
much cleaner and there is no virtual dispatch happening when checking if
intervals overlaps.
* Turn `LiveInterval` into an iterator
The LSRA allocates in forward order and never inspect previous
`LiveInterval` once they are expired. Something similar can be done for
the `LiveRange`s within the `LiveInterval`s themselves.
The `LiveInterval` is turned into a iterator which expires `LiveRange`
within it. The iterator is moved forward along with interval walking
code, i.e: AllocateInterval(context, interval, cIndex).
* Remove `LinearScanAllocator.Sources`
Local methods are less susceptible to do allocations than lambdas.
* Optimize `GetOverlapPosition(interval)` a bit
Time complexity should be in O(n+m) instead of O(nm) now.
* Optimize `NumberLocals` a bit
Use the same idea as in `HybridAllocator` to store the visited state
in the MSB of the Operand's value instead of using a `HashSet<T>`.
* Optimize `InsertSplitCopies` a bit
Avoid allocating a redundant `CopyResolver`.
* Optimize `InsertSplitCopiesAtEdges` a bit
Avoid redundant allocations of `CopyResolver`.
* Use stack allocation for `freePositions`
Avoid redundant computations.
* Add `UseList`
Replace `SortedIntegerList` with an even more specialized data
structure. It allocates memory on the arena allocators and does not
require copying use positions when splitting it.
* Turn `LiveInterval` into a reference struct
`LiveInterval` is now a reference wrapping struct like `Operand` and
`Operation`.
The rationale behind turning this in a reference wrapping struct is
because a `LiveInterval` is associated with each local variable, and
these intervals may themselves be split further. I've seen translations
having up to 8000 local variables.
To make the `LiveInterval` unmanaged, a new data structure called
`LiveIntervalList` was added to store child splits. This differs from
`SortedList<,>` because it can contain intervals with the same start
position.
Really wished we got some more of C++ template in C#. :^(
* Optimize `GetChildSplit` a bit
No need to inspect the remaining ranges if we've reached a range which
starts after position, since the split list is ordered.
* Optimize `CopyResolver` a bit
Lazily allocate the fill, spill and parallel copy structures since most
of the time only one of them is needed.
* Optimize `BitMap.Enumerator` a bit
Marking `MoveNext` as `AggressiveInlining` allows RyuJIT to promote the
`Enumerator` struct into registers completely, reducing load/store code
a lot since it does not have to store the struct on the stack for ABI
purposes.
* Use stack allocation for `use/blockedPositions`
* Optimize `AllocateWithSpill` a bit
* Address feedback
* Make `LiveInterval.AddRange(,)` more conservative
Produces no diff against master, but just for good measure.
* Add `Operand.Label` support to `Assembler`
This adds label support to `Assembler` and enables branch tightening
when compiling with relocatables. Jump management and patching has been
moved to the `Assembler`.
* Move instruction table to `Assembler.Table`
* Set PTC internal version
* Rename `Assembler.Table` to `AssemblerTable`
* Store constant `Operand`s in the `LocalInfo`
Since the spill slot and register assigned is fixed, we can just store
the `Operand` reference in the `LocalInfo` struct. This allows skipping
hitting the intern-table for a look up.
* Skip `Uses`/`Assignments` management
Since the `HybridAllocator` is the last pass and we do not care about
uses/assignments we can skip managing that when setting destinations or
sources.
* Make `GetLocalInfo` inlineable
Also fix a possible issue where with numbered locals. See or-assignment
operator in `SetVisited(local)` before patch.
* Do not run `BlockPlacement` in LCQ
With the host mapped memory manager, there is a lot less cold code to
split from hot code. So disabling this in LCQ gives some extra
throughput - where we need it.
* Address Mou-Ikkai's feedback
* Apply suggestions from code review
Co-authored-by: VocalFan <45863583+Mou-Ikkai@users.noreply.github.com>
* Move check to an assert
Co-authored-by: VocalFan <45863583+Mou-Ikkai@users.noreply.github.com>
Fix an issue introduced in #2190 where by 2 different count table entry
addresses were used for LCQ functions. E.g:
```asm
.L1:
mov rbp,COUNT_TABLE_0 ;; This gets an address.
mov ebp,[rbp]
lea esi,[rbp+1]
mov rdi,COUNT_TABLE_1 ;; This gets another address.
mov [rdi],esi
cmp ebp,64h
je near .L34
```
This caused LCQ functions to not tier up when they're loaded from the
PTC cache. This does not happen when they're freshly compiled.
This PR fixes the issue by ensuring only a single counter is created per
translation.
* Refactor `PtcInfo`
This change reduces the coupling of `PtcInfo` by moving relocation
tracking to the backend. `RelocEntry`s remains as `RelocEntry`s through
out the pipeline until it actually needs to be written to the PTC
streams. Keeping this representation makes inspecting and manipulating
relocations after compilations less painful. This is something I needed
to do to patch relocations to 0 to diff dumps.
Contributes to #1125.
* Turn `Symbol` & `RelocInfo` into readonly structs
* Add documentation to `CompiledFunction`
* Remove `Compiler.Compile<T>`
Remove `Compiler.Compile<T>` and replace it by `Map<T>` of the
`CompiledFunction` returned.
* Fix type mismatch in `BitwiseAnd` simplification
`TryEliminateBitwiseAnd` would turn the `BitwiseAnd` operation into a
copy of the wrong type. E.g:
Before `Simplification`:
```llvm
i64 %0 = BitwiseAnd i64 0x0, %1
```
After `Simplication`:
```llvm
i64 %0 = Copy i32 0x0
```
Since the with the changes in #2515, we iterate in reverse order and
`Simplication`, `ConstantFolding` does not indicate if it modified
the CFG, the second pass to "retype" the copy into the proper
destination type does not happen.
This also blocked copy propagation since its destination type did not
match with its source type. But in the cases I've seen, the
`PreAllocator` would insert a copy for the propagated constant, which
results in no diffs.
Since the copy remained as is, asserts are fired when generating it.
* Set PPTC version
* Turn `MemoryOperand` into a struct
* Remove `IntrinsicOperation`
* Remove `PhiNode`
* Remove `Node`
* Turn `Operand` into a struct
* Turn `Operation` into a struct
* Clean up pool management methods
* Add `Arena` allocator
* Move `OperationHelper` to `Operation.Factory`
* Move `OperandHelper` to `Operand.Factory`
* Optimize `Operation` a bit
* Fix `Arena` initialization
* Rename `NativeList<T>` to `ArenaList<T>`
* Reduce `Operand` size from 88 to 56 bytes
* Reduce `Operation` size from 56 to 40 bytes
* Add optimistic interning of Register & Constant operands
* Optimize `RegisterUsage` pass a bit
* Optimize `RemoveUnusedNodes` pass a bit
Iterating in reverse-order allows killing dependency chains in a single
pass.
* Fix PPTC symbols
* Optimize `BasicBlock` a bit
Reduce allocations from `_successor` & `DominanceFrontiers`
* Fix `Operation` resize
* Make `Arena` expandable
Change the arena allocator to be expandable by allocating in pages, with
some of them being pooled. Currently 32 pages are pooled. An LRU removal
mechanism should probably be added to it.
Apparently MHR can allocate bitmaps large enough to exceed the 16MB
limit for the type.
* Move `Arena` & `ArenaList` to `Common`
* Remove `ThreadStaticPool` & co
* Add `PhiOperation`
* Reduce `Operand` size from 56 from 48 bytes
* Add linear-probing to `Operand` intern table
* Optimize `HybridAllocator` a bit
* Add `Allocators` class
* Tune `ArenaAllocator` sizes
* Add page removal mechanism to `ArenaAllocator`
Remove pages which have not been used for more than 5s after each reset.
I am on fence if this would be better using a Gen2 callback object like
the one in System.Buffers.ArrayPool<T>, to trim the pool. Because right
now if a large translation happens, the pages will be freed only after a
reset. This reset may not happen for a while because no new translation
is hit, but the arena base sizes are rather small.
* Fix `OOM` when allocating larger than page size in `ArenaAllocator`
Tweak resizing mechanism for Operand.Uses and Assignemnts.
* Optimize `Optimizer` a bit
* Optimize `Operand.Add<T>/Remove<T>` a bit
* Clean up `PreAllocator`
* Fix phi insertion order
Reduce codegen diffs.
* Fix code alignment
* Use new heuristics for degree of parallelism
* Suppress warnings
* Address gdkchan's feedback
Renamed `GetValue()` to `GetValueUnsafe()` to make it more clear that
`Operand.Value` should usually not be modified directly.
* Add fast path to `ArenaAllocator`
* Assembly for `ArenaAllocator.Allocate(ulong)`:
.L0:
mov rax, [rcx+0x18]
lea r8, [rax+rdx]
cmp r8, [rcx+0x10]
ja short .L2
.L1:
mov rdx, [rcx+8]
add rax, [rdx+8]
mov [rcx+0x18], r8
ret
.L2:
jmp ArenaAllocator.AllocateSlow(UInt64)
A few variable/field had to be changed to ulong so that RyuJIT avoids
emitting zero-extends.
* Implement a new heuristic to free pooled pages.
If an arena is used often, it is more likely that its pages will be
needed, so the pages are kept for longer (e.g: during PPTC rebuild or
burst sof compilations). If is not used often, then it is more likely
that its pages will not be needed (e.g: after PPTC rebuild or bursts
of compilations).
* Address riperiperi's feedback
* Use `EqualityComparer<T>` in `IntrusiveList<T>`
Avoids a potential GC hole in `Equals(T, T)`.
* Add AddressTable<T>
* Use AddressTable<T> for dispatch
* Remove JumpTable & co.
* Add fallback for out of range addresses
* Add PPTC support
* Add documentation to `AddressTable<T>`
* Make AddressTable<T> configurable
* Fix table walk
* Fix IsMapped check
* Remove CountTableCapacity
* Add PPTC support for fast path
* Rename IsMapped to IsValid
* Remove stale comment
* Change format of address in exception message
* Add TranslatorStubs
* Split DispatchStub
Avoids recompilation of stubs during tests.
* Add hint for 64bit or 32bit
* Add documentation to `Symbol`
* Add documentation to `TranslatorStubs`
Make `TranslatorStubs` disposable as well.
* Add documentation to `SymbolType`
* Add `AddressTableEventSource` to monitor function table size
Add an EventSource which measures the amount of unmanaged bytes
allocated by AddressTable<T> instances.
dotnet-counters monitor -n Ryujinx --counters ARMeilleure
* Add `AllowLcqInFunctionTable` optimization toggle
This is to reduce the impact this change has on the test duration.
Before everytime a test was ran, the FunctionTable would be initialized
and populated so that the newly compiled test would get registered to
it.
* Implement unmanaged dispatcher
Uses the DispatchStub to dispatch into the next translation, which
allows execution to stay in unmanaged for longer and skips a
ConcurrentDictionary look up when the target translation has been
registered to the FunctionTable.
* Remove redundant null check
* Tune levels of FunctionTable
Uses 5 levels instead of 4 and change unit of AddressTableEventSource
from KB to MB.
* Use 64-bit function table
Improves codegen for direct branches:
mov qword [rax+0x408],0x10603560
- mov rcx,sub_10603560_OFFSET
- mov ecx,[rcx]
- mov ecx,ecx
- mov rdx,JIT_CACHE_BASE
- add rdx,rcx
+ mov rcx,sub_10603560
+ mov rdx,[rcx]
mov rcx,rax
Improves codegen for dispatch stub:
and rax,byte +0x1f
- mov eax,[rcx+rax*4]
- mov eax,eax
- mov rcx,JIT_CACHE_BASE
- lea rax,[rcx+rax]
+ mov rax,[rcx+rax*8]
mov rcx,rbx
* Remove `JitCacheSymbol` & `JitCache.Offset`
* Turn `Translator.Translate` into an instance method
We do not have to add more parameter to this method and related ones as
new structures are added & needed for translation.
* Add symbol only when PTC is enabled
Address LDj3SNuD's feedback
* Change `NativeContext.Running` to a 32-bit integer
* Fix PageTable symbol for host mapped
* Use branch instead of tailcall for recursive calls
Use a branch instead of doing a tailcall for recursive calls. This
avoids having to store the dispatch address, setting up the epilogue and
keeps guest registers in host registers for longer.
The rejit check is moved down into the entry block so that the rejit
behaviour remains the same as before.
* Set PTC version
Co-authored-by: gdkchan <gab.dark.100@gmail.com>
* Allow `LocalVariable` to be assigned more than once
This allows us to write flow controls like loops and if-elses with
LocalVariables participating in phi nodes.
* Add `GetLocalNumber` to operand
* Add EntryTable<TEntry>
* Add on translation call counting
* Add Counter
* Add PPTC support
* Make Counter a generic & use a 32-bit counter instead
* Return false on overflow
* Set PPTC version
* Print more information about the rejit queue
* Make Counter<T> disposable
* Remove Block.TailCall since it is not used anymore
* Apply suggestions from code review
Address gdkchan's feedback
Co-authored-by: gdkchan <gab.dark.100@gmail.com>
* Fix more stale docs
* Remove rejit requests queue logging
* Make Counter<T> finalizable
Most certainly quite an odd use case.
* Make EntryTable<T>.TryAllocate set entry to default
* Re-trigger CI
* Dispose Counters before they hit the finalizer queue
* Re-trigger CI
Just for good measure...
* Make EntryTable<T> expandable
* EntryTable is now expandable instead of being a fixed slab.
* Remove EntryTable<T>.TryAllocate
* Remove Counter<T>.TryCreate
Address LDj3SNuD's feedback
* Apply suggestions from code review
Address LDj3SNuD's feedback
Co-authored-by: LDj3SNuD <35856442+LDj3SNuD@users.noreply.github.com>
* Remove useless return
* POH approach, but the sequel
* Revert "POH approach, but the sequel"
This reverts commit 5f5abaa24735726ff2db367dc74f98055d4f4cba.
The sequel got shelved
* Add extra documentation
Co-authored-by: gdkchan <gab.dark.100@gmail.com>
Co-authored-by: LDj3SNuD <35856442+LDj3SNuD@users.noreply.github.com>
* Improve StoreToContext emission
Hoist StoreToContext in dynamic branch fast & slow paths out into
their predecessor.
Reduces register pressure, code size and compile time because we're
throwing less stuff down the pipeline.
* Set PTC internal version
* Turn EmitDynamicTableCall private
* Re-trigger CI
* Add VCLZ fast path
* Add VCLZ.8B/16B SSSE3 fast path
* Add VCLZ.4H/8H SSSE3 fast path
* Add VCLZ.2S/4S SSE2 fast path
* Improve CLZ.4H/8H fast path
* Improve CLZ.2S/4S fast path
* Set PPTC version
Before when splitting intervals, the end of the range would be included
in the split check, this can produce empty ranges in the child split.
This in turn can affect spilling decisions since the child split will
have a different start position and this empty range will get a register
and move to the active set for a brief moment.
For example:
A = [153, 172[; [1899, 1916[; [1991, 2010[; [2397, 2414[; ...
Split(A, 1916)
A0 = [153, 172[; [1899, 1916[
A1 = [1916, 1916[; [1991, 2010[; [2397, 2414[; ...
* Implement block placement
Implement a simple pass which re-orders cold blocks at the end of the
list of blocks in the CFG.
* Set PPTC version
* Use Array.Resize
Address gdkchan's feedback
* Relax block ordering constraints
Before `block.Next` had to follow `block.ListNext`, now it does not.
Instead `CodeGenerator` will now emit the necessary jump instructions
to ensure control flow.
This makes control flow and block order modifications easier. It also
eliminates some simple cases of redundant branches.
* Set PPTC version
* Add Compare instruction
* Add BranchIf instruction
* Use test when BranchIf & Compare against 0
* Propagate Compare into BranchIfTrue/False use
- Propagate Compare operations into their BranchIfTrue/False use and
turn these into a BranchIf.
- Clean up Comparison enum.
* Replace BranchIfTrue/False with BranchIf
* Use BranchIf in EmitPtPointerLoad
- Using BranchIf early instead of BranchIfTrue/False improves LCQ and
reduces the amount of work needed by the Optimizer.
EmitPtPointerLoader was a/the big producer of BranchIfTrue/False.
- Fix asserts firing when assembling BitwiseAnd because of type
mismatch in EmitStoreExclusive. This is harmless and should not
cause any diffs.
* Increment PPTC interval version
* Improve IRDumper for BranchIf & Compare
* Use BranchIf in EmitNativeCall
* Clean up
* Do not emit test when immediately preceded by and
* Use movd,movq for i32/64 VectorExtract %x, 0x0
* Increment PPTC interval version
* Use else-if instead
- Address gdkchan's feedback.
- Clean up Debug.Assert calls
* Inline `count` expression into Debug.Assert
Apparently the CoreCLR JIT will not eliminate this. :(
* Generalize tail continues
* Fix DecodeBasicBlock
`Next` and `Branch` would be null, which is not the state expected by
the branch instructions. They end up branching or falling into a block
which is never populated by the `Translator`. This causes an assert to
be fired when building the CFG.
* Clean up Decode overloads
* Do not synchronize when branching into exit block
If we're branching into an exit block, that exit block will tail
continue into another translation which already has a synchronization.
* Remove A32 predicate tail continue
If `block` is not an exit block then the `block.Next` must exist (as
per the last instruction of `block`).
* Throw if decoded 0 blocks
Address gdkchan's feedback
* Rebuild block list instead of setting to null
Address gdkchan's feedback
* Improve IRDumper
* Make Symbols.Add(ulong, ulong, ulong, string) thread safe
* Use a StringBuilder for MemoryOperand
* Add #if M_DEBUG guards
* Fix JMP_TABLE typo
* Fix using in Symbols
* Use Conditional("M_DEBUG") instead
Address gdkchan's feedback
* Use a struct instead of 4-tuple
Address gdkchan's feedback
* Place symbols in comments instead
Address gdkchan's feedback
* Use StringBuilder throughout
* Handle offsetted symbols
* Fix naming convention of Builder
* Avoid ArgumentException
* Remove unnecessary using
* Use switch expression instead
* Turn into a class
* Clean up
* Remove unnecessary using
FICTURE7andGitHub
