Also how does the low voltage end of the coil have enough voltage to arc anyways?

"Low" voltage end of secondary is low because it is grounded.  If not grounded, it is no longer low voltage.

edit: if i dont ground the plates the gdt gets corona discharges.

If current can't return to ground, it tries to get there through line hot and neutral, through GDT, or any other path it can find.  Much better to tie to ground.  Line ground is intended to pass current to ground.  Line hot and neutral are not intended for that purpose.

But short after i have realized that the ground wire (which is connected to ground plane by a tape) was getting arcs from the ground plane and it caught on fire.

I think you are referring to ground wire coming from bottom of secondary.  Presuming so, some of that current is flowing to that counterpoise sheet and not all the way back to line ground.  The larger your counterpoise, the more of the current it will take and less will pass on to line ground.

What should i do?

Use a large counterpoise for low side of secondary.  Connect it to line ground as well.  Clamp or screw wires to aluminum, not just tape.

BTW, I had a grounding problem with my QCW coil at last weekend's Science Festival here.  Used a clip lead (test lead) for grounding bottom of secondary.  Alligator clip at one end of lead was corroded, so made a bad connection.  Secondary low side voltage increased enough to arc through the bad connection, but also arced to bottom of primary coil, charring insulation there.

Capacitor units are 'F' for Farads.  Inductor units are 'H' for Henries.  You are asking about 100uF capacitor.

For most uses, the lower ESR of ceramic capacitors is better.  There are a few occasions where circuits rely on the ESR of electrolytic capacitors, especially for feedback stability of voltage regulators.

Ceramic capacitors have their own issues.  Z5U and Y5V dielectrics are worthless unless used strictly at room temperature.  Those and X5R and X7R often have severe capacitance reduction as DC voltage increases, sometimes down by 90% by rated voltage.  Actual capacitance in use may be far less than you expect.

I'm guessing these output DC, so like typical CRT flyback transformers.  If they output AC, that would make them much more useful.

Hello, I made a huge discovery, the whole thing works fine when I touch the 2nd or 3rd pin of the AD8561ANZ  ultra fast comparator that I used instead of the lt1001 with the tip of a oscilloscope probe
Now I am confused as to why does this solve the problem and what should I change on the board to make it work by itself
With red is where I probed and it stared working (the ground lead was not connected to anything)

Scope probe and lead cable are acting as an antenna.  You are picking up enough either random noise or more likely voltage from secondary.  That pickup makes extra driver output transitions to start oscillation.

Three options come to mind for fixing this without a scope probe:

1) Add an antenna.  NOT recommended.  Hard to control what phase lead will be or how much signal may be picked up and over-drive comparitor input.

2) Fix the issue(s) preventing normal startup.  I've already twice suggested next scope measurements to take, so won't repeat here.

3) Modify your driver to be self-oscillating.  I described this for UD2.7 in third paragraph of this post:
    https://highvoltageforum.net/index.php?topic=1373.msg10197#msg10197

1. Is there a way to measure the output voltage?

There are hard ways to get somewhat accurate at least for the beginning as breakout starts.  Hydron placed a scope on top of the top load to measure charge:
    https://highvoltageforum.net/index.php?topic=117.0#msg780
I've done somewhat similar experiments using home-made fiber-isolated scope probes:
    https://highvoltageforum.net/index.php?topic=1263.msg9259#msg9259
    https://highvoltageforum.net/index.php?topic=1950.msg14588#msg14588
However, you can get reasonably close by measuring current in ground lead to bottom of secondary.  If top load is reasonably large, electric field will be somewhat uniform down secondary, so secondary winding capacitance is a small contribution compared to top load capacitance.  Current in secondary (amplitude and frequency) and secondary inductance are enough to calculate voltage.

Measurement will be even more accurate if a scaled version of primary current is added to secondary current measurement.  Scale factor is mutual_inductance / secondary_inductance.  This adjusts for voltage created by transformer action from primary to secondary.  Inductive drop across secondary due to secondary current is generally a lot larger, but adding in scaled primary current may change result by 10% or even 20%.  Resulting current (from which voltage is calculated) may be higher or lower depending on relative phase of primary and secondary currents.

One way to add in scaled primary current:  Add a third primary CT output stage (or entire third CT) with its own adjustable burden resistor (or fixed burden resistor followed by POT to reduce voltage).  Wire this CT output in series with secondary CT output.  (Secondary CT of course has its own burden resistor.)  To adjust scale factor (POT on primary current):  Run coil with secondary shorted (top load wired to ground).  Adjust POT until sum of the two CT outputs is zero.  Swap polarity of one CT if zero can't be reached.  Will never be exactly zero due to phase shift in CTs, but should be closer than secondary current alone.

2. What affects the output voltage more. The length of the pulse or the frequency of the pulses?

For typical DRSSTCs, top voltage drops as arc grows.  Arc lowers secondary Q and makes a better breakout point.  Highest voltage is when breakout starts.  Voltage will be higher with shorter or no breakout point.  Fast voltage ramp also increases peak voltage.
Frequency also matters.  Higher frequency reduces breakout voltage.  However, high frequency and low voltage can make long arcs.  My normal DRSSTC at 80kHz and ~600kV initial peak voltage makes ~2m arcs when tuned for music, 3m when optimized for arc length.  My 450kHz QCW coil is about 40kV peak and makes arcs in the 2 to 2.5m range, perhaps slightly longer when I push the pulse width.  For QCW coils, voltage may increase slightly as arc grows and more voltage is needed to overcome arc resistance.

I reviewed the video footage and i can see some sort of arcing between bus + and -. and between the igbt legs. this is super weird since 320v i thought way too low spark anything.

The IGBT etc. arcs are likely a result of IGBT failure rather than a cause.  When IGBTs (or FETs etc.) fail shorted, sudden current burns/melts leads and interconnect.  Arcs are very high current and low voltage, like arc welding.  That is what happened when my QCW IGBTs failed a few months ago.

Also, you wrote that leaving a radius gap between both ends of the coil  and topload/ground is a good idea. (I guess to not block magnetic waves) If i don't do that, will it cause a problem or will it only decrease my output?

Yes, to avoid blocking magnetic field.  Removing that gap will increase secondary frequency a bit (reduce secondary inductance) and increase secondary loss a bit (lower secondary Q).  Neither will be large changes.

Maybe due to too much thermal paste or debris under my thermal pad both MOSFETs arced to the heatsink.

Looks likely.  I saw several similar failures recently at work on 60V FETs when silver thermal compound bridged gap from tab (drain) to source or gate leads.  There are now thermal compounds that are non-conductive electrically and as good as silver compounds for thermal conductivity.

Some quick scope shots, this is the halfbridge output working when the driver is fed a signal
I fed it 170 vac on the bus at some point and nothing blew up so the halfbridge is switching right

Do you have a bleed resistor pulling half-bridge output low?  Something is causing output to end up at 0V after enable pulse ends.
Looks like Vbus is 60V for this scope capture.  Is that accurate?  Is Vbus 60V for the current transformer output capture too?

but the really strange output is this "starting pulse" that I measured on the gdt when it has no feedback connected, to me it looks inverted, how is this possible?

Where on GDT are you probing?  Is this low-side Vge (or Vgs if using FETs)?  If so, it looks appropriate, other than possible CT inversion issue as flyingperson23 mentioned.  What does half-bridge output look like?  What does voltage across UD2.7 51-ohm resistor look like?
If this is low-side Vge, then high side Vge should be a matching positive pulse.  That should make a low-to-high transition on half-bridge output.  That output transition should cause current that should trigger another transition.

This is the signal on the input pins of the uccs

That looks like a great 671kHz square wave.  As NyaaX_X said, JavaTC can tell you if that is expected frequency for your secondary and top load.  If so, then coil is running, just without enough power.  If JavaTC indicates a lower frequency, then the antenna is picking up the wrong signal and/or phase is inverted.

Loaded output still has a weird jump on one MOSFET driver. If this indeed from Miller Capacitance, that peak should reduce dramatically when the bridge is at mains voltage? Would you expect the traces to be asymmetric? The spikes are only on of the outputs.

This capture does look less like Miller capacitance, more like driver chip ground bounce.  Loading may be adding just enough additional ground current spike to cause trouble.  Perhaps one chip has slightly higher inductance in ground path or has slightly faster output transition or more shoot-through current.  Previous loaded captures may have been ground bounce too.  They looked more like flat spots than spikes, which is typical of Miller capacitance.

I tried using a 4.7k pull down resistor, and the GDT output looked good unloaded, but loaded, something was still wrong. (I think still the same issue as my original post, just to a lesser degree? I'm not sure through what mechanism the solution identified in the next paragraph works if it isn't the same issue.)

I'd guess that the 4.7k pull down resistors are working fine.  Loaded waveform is likely due to IGBT miller capacitance loading driver chip outputs.  However, it could be that 4.7k isn't enough pull-down.  Any ~2k resistors around?  Or enough 4.7k resistors to parallel 2 or 3 for each pull-down?

I found a single 4700nf capacitor to throw across one of the 4.7ks, and voila!

Did you mean 4700pF?  Even if so, that is still large.  Probably OK when used without a pull-down resistor.  I was suggesting a much smaller capacitor combined with a pull-down resistor.  However, combining the two does require getting capacitance correct to avoid driver chip inputs glitching too far below ground.

The rising edge of the waveform looks pretty good! Maybe a little over damped?

Once the input is clean enough to not glitch due to driver chip ground bounce, any remaining characteristics of driver outputs are due to driver output impedance and load impedance (GDT and IGBT gates).

The high frequency ringing before the output goes high: is that a problem?

Probably not an issue.  For such relatively small ringing, it is difficult to know if it is real or magnetic pickup in loop from scope probe ground lead to probe tip.

I don't have another capacitor to check if that's a symptom of the other driver not having the capacitor across it's resistor yet.

Probably best to have both channels modified to be the same.  There can be some advantage to a slight delay in one driver relative to the other.  However, any of these fixes (caps or pull-down resistors) will affect rising and falling delays differently.  If you have any lower-value resistors (ie. 2k), I'd guess that to be a more reliable fix than capacitors alone.  (If you have extra 4.7k resistors, two in parallel for each pull-down would be better than just one each.)  The pull-down resistor solution should move duty cycle closer to 50%.