OCXO Reference devolopment - or a story about scaleability

[Da_Stier]

Hi all,

long time no see. I spent the better part of the last year on developing an OCXO based frequency reference.
Since it got a little out of hand and I learned a lot in the process, I thought why not tell the story here.

Let's start from the beginning. Last year I had some trouble sleeping so at one point I decided to make the best of it and let "night me" develop something instead of staring at the dark ceiling.
After thinking a bit I decided to tackle a problem that I had for a while now.
My QO-100 HAM radio station needs several reference frequencies - one at 25MHz for the LNB and one at 40MHz for the SDR transceiver I use. They also need to have different amplitues and can't have a lot of jitter.
My old setup used a dual channel GPSDO, made by Leo Bodnar - which is a great device - but I didn't like the dependence on a GPS signal.
I had trouble before with not being able to operate due to the setup being under a roof of some sort.

I previously tried to use two of my little PT7C4511 breakouts to sync everything to a common 10MHz source.
As the source I came up with a very simple OCXO brekout with an integrated 6dB divider.





While it looked promising on the oscilloscope, it turns out, that there is way to much jitter on the outputs to be of any use.
This caused me quite a bit of confusion first, since I couldn't figure out, why nothing would lock on to these signals, unitl I realized the horrible jitter.

Next I remembered the SI5351 IC that is very commonly used for all kinds of reference purposes, from both hobbyists as well as in actual commercial products.
The most common one that can be found on a lot of cheap chinese breakout modules is the SI5351A which is the 3 output version that runs off of a external chrystal.
These modules look like this:



These are pretty nice and can basically do what I want, however I didn't like the 3 channel limit or the clock source being a chrystal at some random frequency.
After looking at the datasheet, I realized, that there is a SI5351C version available, that offers up to 8 outputs and can run from an external single ended input.

PERFECT. 



These chips are controlled via a simple I²C interface, run off a single 3V3 supply and need almost no external components.
So I got myself a few of them on Digikey and build a little breakout board.
Since I didn't want to wait for PCBs to arrive, I decided on a deadbug breakout, which is a nice challenge in itself, considering the 4x4mm 20 pin QFN package.



I reused the OCXO board from earlier and built a first mockup of the basic system I wanted to create.



I found a great arduino library to control the SI5351 series of ICs by NT7S.
If anyone is interested, it can be found here:
https://github.com/etherkit/Si5351Arduino

As a controller I chose to use an ESP32, since it runs at 3V3, is pretty cheap and easy to assemble.
After getting the proof of concept version going, I started to work on an actual board.
I wanted it to be a rather clean looking unit and decided to use a Hammond enclosure, since I used them before and they are affordable.

So after a couple of nights, "night me" came up with the first design, orderd all the parts for a few prototypes and a couple of weeks later, "day me" received everything and started the assembly.
This is what the first board looked like:



It also fitted nicely into its enclosure.





For software devlopment and debuggind, I like to make a "sceleton version" of the enclosure.
This offers access to all the LEDs, measurement points, etc., while making sure that it will stay at the final formfactor.





After this came  A LOT of firmware nights to get all the features going, that I wanted.
This includes:
- serial terminal via USB connection with a command line interface for setup
- saving all settings to memory
- recalling all the settings on startup
- two LEDs for status
- setting all eight channels in frequency and amplitude
- an external 10MHz input / output, depending on the mode it works in

It didn't take long for the first issue to come up.
During the hardware design I planned to use a dedicated programmer to flash the ESP.
This worked fine but by then I preferred to flash it via the same USB that is used for the CLI (command line interface).
Therefore I did my first mod on the protoype units and added a small auto reset addon.



With this mod I was now able to flash it via USB - much more comfortable.
By then I got most of the firmware going, the code looked horrible but it worked.
At this point I sent one of the units to Gregory - a friend of mine, over in Brazil.

He runs a youtube channel and did a review / gave some comments on one of my prototypes.

I also attached one of the units to my QO-100 setup and it worked the first try. 
This was a great success for me, especially since all of this came out of boredom basically.
The unit attaches nicely to the tripod I use for the satellite dish and I am finally independent of any third party signal - except the satellite of course.




At this point I thought I am done with this project, I built 5 prototypes, with one being in Brazil ... but it turned out, I was far from done.
I will split up the post here to make it a bit easier to type out.

[Da_Stier]

So at this point, both Gregory and I had some ideas on what to improve, how to get better firmware going and also on what to get out of the project.
So once again "night me" opened KICAD and worked on the next iteration.
By far the most motivating reason for this was, that Gregory suprised me by completely rewriting the firmware and making it way more responsive and overall way better.
So while being at it I decided this would be a perfect opportunity to see how a project like this would scale up.

And while "day me" was still a little unimpressed by the thought to go all out and build a lot of these references, "night me" was collecting quotes from manufactures and working on a breakout box for testing.
This breakout box made it really easy to work with the device, since it drastically reduced the amount of wires and other clutter needed on the desk.
It has LEDs on all the outputs, switches on all the inputs and banana jacks for everything else.





This picture also gives a preview on the next change, I got some screenprinted enclosures, custom made for me by Hammond.
Their service was really helpful and it was pretty easy to get them printed.
.... and still cheaper then the standard  version off of Digikey.   
I don't think I mentioned this before but the endplates for the enclosures are actually aluminium PCBs.
I found this to be by far the cheapest way to get CNC milled, black lids with custom screenprinting on them.

So what was the plan from here ... make 50 units of course.
This is when the real work began, I got myself 50 boards of version V1.1.
Why? To learn how to scale things up, to see, what works and what doesn't if you have to do it 50 times.
Also to see what is important for getting boards populated.
I guess I'm in a rather lucky position here, since the company I work for actually has a PCB assembly line.
This meant that I was the one populating these boards, which meant I had all the possibilities to learn from it.



And while 50 pieces didn't sound too bad at first, I quickly realized, how much 50 actually is.





So the very first thing I learned is to reduce manual work as much as possible.
I think I did a good job by getting all the lids professionally made, by using SMA pigtails that just need to get plugged in, etc.
However some other parts, like having no mechanical index for the edge launch SMA connectors for example, turned out to be a timely oversight.

On the software side of things, Gregory did an outstanding job by implementing a Web UI on the ESP.
This means that you can either configure the device via the CLI and forget it or to have a live Web UI to change all the settings.
This expanded the use case of the unit from a "dumb" reference to an actual signal generator for lab use.



This also meant that I got a lot of new ideas on where to go from here.
For example I implemented a proof of concept IQ function to have up to two IQ channels, providing signals at 90° phase offset.

So where am I at right now?
The 50 units are built up but not yet calibrated or tested.
After thinking about how to do that, I decided on building a teststand for the unit.
This makes testing a lot easier, since it saves me from connecting and disconnecting a million SMA connectors.
I came up with a "quick connector" consisting of an aluminium shround and a pogo pin.
The goal being to have a "drop in a board and push it closed" kind of device.



So that's basically it for now, I will keep posting updates as soon as I got some more work done.
Also, if anyone is interested in getting a reference like this, feel free to contact me, I have a few. 



Greetings,
Michael

[Mads Barnkob]

That is some project!

From the faucet style heat sink of the QFN package in dead bug style to a finished product with a web interface. Very nice, but take care and do get some sleep. Too much night-me sounds unhealthy

I got a project myself, with a 10MHz reference, but its GPS based like what you came from. I have a Meinberg GPS170 sitting in a box, that really should get in a minirack and live on the bench.

[Da_Stier]

Thanks Mads. 
Yeah, I know, night-me got tired at last so it is slowly getting better.

I took a look at the GPS clock you have, seems like a really nice unit.
I think a permanent antenna and a little rack together with a distribution amplifier would make a great setup for your gear.
(Also a video about such a setup would be pretty cool)



I had one of my sample units running 24/7 for a little over a month now.
I fed a 10MHz output into a frequency counter with a known good GPSDO delivering the counters reference.
The maximum deviation between the two was 1 count on my counter.
This equals 0.1Hz or 0.01ppm or 10ppb.
This seems reasonable I think, however I don't like how I only got a single count to go by.
I know that I can look at the phase on a scope and measure the time it takes for the phase to do one period for calculating very small differences in frequency.
However it seems like there should be some frequency counter with mHz resolution out there, right?
Does anyone know about such a device?
All counters I came across until now, have 0.1Hz as the minimum resolution at a 10s gate time.

Other then that, I finished the tooling.
It turned out to be a combination of 3d printed parts and aluminium parts.
It uses some linear bearings on 8mm steel rods to provide a smooth operation.





I made some magnetic inserts with 2 guiding pins for alignment.
One that holds the bare PCB and one for the complete unit to have simple "drop in and push closed" operation.

The contactpoints for the SMA outputs are a combination of pogo pins with GND shrouds and SMA connectors without the nut.
It turns out the U.FL to SMA pigtails have slightly recessed inner conctacts so I needed to schange to top row over to the SMA connector style.



For multiplexing the signals to an oscilloscope, I used the RF switchmatrix, I made a while back.
Pretty overkill for the 10MHz testsignals but I already had it ready.



The tooling turns out to be an absolute life saver while going through all 50 units.





Greetings,
Michael

[ch_scr]

A very nice project! I'm more on the eevblog, and stumbled across this from a google search... Seems like you stumbled into the "Timenuts" rabbit-hole head first, and with some success!
Now, to the point of your question on "how to get better resolution on the frequency counter, without spending €€€", that compelled me to make an account here, since I've embarked on that road myself a few years ago. A word of "warning" beforehand, that road ineviteably leads to a thing called an "Allen deviation plot", since "staring at numbers on a counter" is hard to quantify and convey. There's quite a good explanation of that to be found here. How do we get one, then? There's a free software called Timelab that works with generic counters in "talk only mode". And how do we get the resolution to make a meaningfull one? The more modern way is a cross correlation measurement with a device like this. This is the "hobbyist upper end" with regard to price, but also capability. That's below the usual price for a used-market 12-digit counter, but not that much. It's more "tailored to the job" though. The entry level for a ready-made device is the "TinyPFA". Definitely the best price/performance option to get one's toes wet, IMHO. I've taken the "classical option", inspired by a post on EEVblog to build my own DMTD and TIC. Turns out it's a rabbit hole in itself... All of these options compare two (or more) oscillators! With an expensive counter, you'd hopefully get the appropriately expensive high stability oven option to make use of all the digits. With the TinyPFA or DMTD approach, you also compare two oscillators, but need to bring both yourself. So you need to have a "known to be better than the DUT" one to get "absolute" numbers. You can get there by comparing units until you find a "golden" one. The cross-correlation approach is a lot more elegant in that way, but at a significantly higher cost / complexity. Hope this was of some help.
Kind regards,
Christian

[Da_Stier]

Thanks, Christian!

These are some very interesting rabbit holes topics to look into.
I like the microchip unit (the TimeLab one) however I almost fell off my chair when I found the pricing. (22k€ for the base unit).
On the other hand the TinyPFA for example looks like a very interesting thing for further reading, a nanoVNA is certainly cheap enough to give it a try.

One (maybe very naive) question I still have is the following:
Starting from a very basic frequency counter architecture like the one found in the 2021 edition of the ARRL handbook, what limits the actual resolution?



With a 1s gate time you get a 1Hz resolution.
With a 10s gate time you get a 0.1Hz resolution.
What would stop me to get a 0.01Hz resolution with a 100s gate time?
With this logic I should get down to 1mHz at a 1000s gate time, for a long term measurement, a roughly 17 minute gate time would be ok for me.
Assuming my reference is known good, I should atleast be able to measure the long therm stability of my testsignal, right?




Now for the update on my units.
I just finished all 48 "production" units.
This finally allowed me to get rid of the chaos in my bedroom and get to a nice box of finished units.





From the 50 PCBs I got assembled, I had 5 with some sort of failure.
These were the following:
- solder bridge on the USB C connector -> easy fix, works now
- unsoldered pin on the USB C connector -> easy fix, works now
- unsoldered pin on the QFN -> easy fix, works now
- 13MHz OCXO instead of a 10MHz one -> error on my side, easy to fix, works now
- some error on the 3V3 LDO -> board literally went up in flames, got scrapped

This was some very interesting information to get a feel for what fails at what rate.





Greetings,
Michael

[ch_scr]

One (maybe very naive) question I still have is the following:
Starting from a very basic frequency counter architecture like the one found in the 2021 edition of the ARRL handbook, what limits the actual resolution?



With a 1s gate time you get a 1Hz resolution.
With a 10s gate time you get a 0.1Hz resolution.
What would stop me to get a 0.01Hz resolution with a 100s gate time?
With this logic I should get down to 1mHz at a 1000s gate time, for a long term measurement, a roughly 17 minute gate time would be ok for me.
Assuming my reference is known good, I should atleast be able to measure the long therm stability of my testsignal, right?

Hello Michael,
good question. Here's a document detailing some error sources in a counter. It says the "time base error is dominant for frequency higher than 1MHz" (page 17) (if you go farther, all you'll see is that the "digits on the end" show a different random number each time you (eventually) get an update). But you'll still end up comparing two "about equal" OCXO's. (Could have that a lot easier with a TinyPFA, IMHO.) For a "better timebase", you'd not just need "another" cheap OCXO (I bought 5 different cheaper types already, and have yet to find one that's clearly better). Rather you'd like one 10 times better than the "usual DUT", but that's already getting a lot closer to the actual physical limits, hence why these don't grow on trees. The cross correlation approach is what gets around that, but has it's own downsides (higher price and complexity of the machine).
Not as helpfull as I'd liked,
Christian