So I wanted to ask what is usually better? Running at the upper pole or the lower pole?

QCW coils usually run at upper pole frequency.  Normal DRSSTCs are a bit more complex to describe.  Upper and lower pole frequencies describe steady state.  For short enable pulses, DRSSTCs don't reach steady state.  Currents are amplitude-modulated as energy sloshes back and forth between primary and secondary, a mix of upper and lower pole frequencies.   For long on times (typically for higher impedance coils), operation will settle to one pole.  Though that pole can change as arc loading changes secondary frequency.  Once secondary frequency drops below primary, operation changes to upper pole for remainder of enable pulse duration.

Typical DRSSTC tuning is with primary frequency below secondary (below unloaded secondary frequency).  Improves performance as arc capacitance reduces secondary frequency to match primary.  Matching frequencies with simulated arc loading (wire) is one way to get there.  More common is to just try a specific amount of detuning such as primary 10% below secondary.

I have thought about using that bottle and after having the exact dimensions with a hot air gun, shrink the plastic in the secondary and if it does not reach the height, a joint would be made between the two layers with about 5 cm above the other, would it affect anything if I use this plastic?

Interesting idea.  If it would shrink only radially as heat-shrink tubing does, might work well for both insulation and to hold windings in place.  My concern is that I think PET bottles will shrink axially too, which may or may not create a problem.  I could imagine axial shrinking pulling windings out of place and creating overlap.

The varnish does little to increase the insulation. The coating on the wire does that job.

Depends on thickness.  I'd probably agree for standard air-dry varnish.  Two-part epoxy coatings can be thicker and provide useful insulation.  My DRSSTC has 7 coats of epoxy about 1mm thick each.  Initially had only two coats.  Additional five was a big help in minimizing racing sparks along secondary.

This is probably a stupid question but what if I use a counterpoise under the coil but also connect the counterpoise to mains ground?

Not at all a stupid question!  Opinions vary widely.  I always use a counterpoise under my coils and always connect them to mains ground too.

Looks nice from the datasheet! Running a few calculations based of times I think you don't want to push the speed to much, I got 200Khz. It is not worthy it says it's optimized for hard switching. A drsstc soft switches so you might not get as much benefit from that. I've never tried the transistor so I cannot say for sure. Still seems real nice!

I've not used this IGBT myself either, but expect it is capable of more than 200kHz with proper drive and H-bridge layout.  The key "slow" number is turn-off delay.  However, this is with 0-15Vge and 10 ohms gate resistance.  With +-15Vge or +-20Vge and typical diodes in gate drive to reduce turn-off gate resistance, turn-off will be MUCH faster.  That fast turn-off leads to need for low parasitic H-bridge inductance.  Even 13nH of IGBT leads can be problematic as I showed in previously linked thread.  IGBT turn-off at low current (just before zero) will help reduce spikes too since inductance can't be eliminated.  If spikes are still problematic, R+C snubbing on H-bridge (on IGBT C-E) will help further.

So, the two big resistors near the three output terminals start creeping over 60 degrees if I’m putting in more than 16v, (no fan yet obviously) does that seem wrong to anyone? The rest of the setup looks much cooler from my thermal camera. At 24v in the resistors kept heating up into the high 60s before I turned power off.

Power resistors like these are designed to run hot.  200C surface temperature is common.  Only real concern is their heat warming adjacent capacitor.  A fan will help with that.

How do these gate waveforms look? Is the rise time too slow?

I have no idea what rise time is because scope time/division is not listed.  Oh, after noticing title saying 100kHz, I can then deduce scope is at 2us/div for first capture.  Listing both volts/div and time/div for all traces would be helpful.  Or amps/div for current.
Rise time is probably fine.  Some rise time is needed to provide dead-time between turn-off and turn-on.

Also, do you know why I sometimes get the noise on the primary current waveform at the switching transitions but sometimes not? In this case it might be relevant to fix the noise because it lines up with some ringing/noise on gate turn off.

Presuming UD2.7 or similar driver, phase lead does not function well at low current.  Once current builds, phase lead works better.  Phase lead makes switching at proper time slightly before zero current which reduces spikes.

1.) Question: Are there any pitfalls to this idea?

Sudden discharge of flyback transformer secondary due to spark gap can destroy transformer (or internal diodes).  Depends on internal construction.  Either add a Terry filter or have a few spare flyback transformers for experimenting.  Search for Terry filter and/or study this thread:
    https://highvoltageforum.net/index.php?topic=822.0
Above thread includes other cautions about unused flyback transformer pins, which might be relevant to your design.

3.) Question: Can I estimate the output voltage and current of the flyback accurately enough to do so, or should I try to measure it?

Can be estimated from simulation, but only if you know transformer characteristics (inductances, coupling factor, and saturation current).  May be easier/faster to measure.  At least you are starting with a ZVS/transformer pair that presumably is known to work together.  It is easy to end up with little power due to ZVS frequency too high or low for transformer (or other issues).

Hope your project is successful and a good learning experience.

Is this explanation correct?

Looks accurate to me.
Try simulating for a longer time period and/or reduce value of C1 to say 100nF.  Then you will see the need for D1.  D1 isn't needed for low duty cycles (short enable times), but is for longer times.

I ordered a few more globes and filled them with 15 torr and then 50 torr xenon. No toroid
behavior. I began to wonder if baking was the difference. So I build a sketchy oven and baked a
one liter globe at 350C for an hour. Filled it with 15 torr of xenon.

Baking under vacuum is the ideal glass surface degassing process.  Neon sign makers use a process they call "bombarding".  It's not quite full vacuum because 2-3 torr of added gas is needed to pass high current through the tube to cause the heating.  Not clear if the hot plasma inside helps degassing or hurts it.

But, in my hypothesis, and as the observations of others, the Magnetic field has a far greater effect than the voltage, or current moving through the coils.

As a potentially more approachable experiment, I'd love to see what'd happen with drive frequencies outside of the now-customary 10-15MHz range.

A relatively high electric field is needed to start internal xenon glow discharge.  A lower field and higher current maintains the discharge (torus).  The starting electric field is usually capacitively coupled to xenon through tube wall, from a starting wand or from coil voltage.  Sustaining electric field is almost all induced by the AC magnetic field, which is why the discharge is a torus.  Not clear if the magnetically-induced electric field is ever high enough alone to start discharge.  That's something I hope to experiment with some day, using a single-turn coil (high current at relatively low voltage) to minimize capacitively coupled electric field.
Frequency will be a great variable for experimenting.  Lower frequency will require more current (stronger magnetic field) to induce the same electric field.  That stronger magnetic field should push the torus farther from coil due to stronger magnetic repulsion.  Farther from coil will reduce electric field.  May result in more self-extinguishing and restarting.  I expect high frequency will be most efficient at generating a stable torus close to coil.
Of course, buoyancy of of the hot torus also causes it to rise.  One of the videos shows the significant behavior difference between coil on top vs bottom of xenon-filled glass sphere.  With coil on top, torus is stable and closer to coil.  Don't recall which video shows this.

1) In the video I've attached I noticed the OCD triggering (the red light) more for small arc lengths than for long arc. Is this abnormal or is it because of the buildup current required to extend the arc which the brings the coil into tune because of the reduction of the resonant frequency? Should I leave it as is or does it require better tuning?

Notice in your video that the short arcs occur on lower notes.  Arc path from previous note cycle (previous enable pulse) has longer to cool before next enable pulse.  Cooler path requires more voltage to ionize air again.  Primary current hits OCD before secondary voltage has been high enough for long enough to grow arc longer.  Yes, you are correct: Once arc grows somewhat long, secondary frequency drops to better match primary frequency, allowing more energy transfer to secondary without exceeding primary current limit, further extending arc length.
Tuning primary frequency slightly higher will increase arc length for low notes, but likely reduce arc length for higher notes as secondary frequency drops below primary due to arc capacitance.  You can test to see which behavior you prefer.  No one "right" answer.
Your coil has a couple similarities to my DRSSTC: Secondary impedance is below typical 50k, and coupling factor is on the low side due to tall skinny secondary design.  Both tend to exacerbate above tuning tradeoff.  If mechanical construction allows, you could try raising primary slightly to increase coupling.  Higher coupling transfers more energy to secondary for given primary current and detuning.  If coupling is too high, secondary racing sparks become a problem.  I had to drop from my original 0.147 to 0.141 due to racing sparks (primary lowered from 50mm to 25mm above bottom of secondary).
Without remaking secondary, only option to increase impedance is smaller top load.  But that itself reduces performance, so not recommended.

2) The primary connection lead(the thick red wire) passes by close to the driver enclosure(the yellow box) which led to it heating up after long runs. I thought this won't be the case if the enclosure is grounded. Is it true and its not grounded properly? Or should I just increase the distance between them?

Likely issue is induction heating, not grounding or other electric field issue.  Steel (ferromagnetic) housing heats very efficiently due to local AC magnetic field.  Either move cable away from enclosure or change to an aluminum (or copper) enclosure.  Or add an aluminum partial enclosure around that portion of steel enclosure.
Did the entire enclosure heat up or just part near wire?  Looks to be far enough from primary coil, but induction heating from primary coil's magnetic field is also a possibility given how easy it is to heat steel.

Great accomplishment!  Thank you for sharing.