October 27, 2016 Misha Davidov

The Science of Duo in Space, Pt. 4: Back to the Lab

Previously: Duo in Space's objective of performing the first Push authentication from the boundary of space is becoming more real, following a test launch conducted on the outskirts of the Mojave Desert. The flight data and on-camera footage yielded valuable insights for Duo Labs' Mikhail Davidov, the technical brains behind Duo in Space, to make refinements ahead of the official launch during DEF CON 24.

While we weren’t able to see the balloon burst, we did have some footage up to 50,000 feet. As we watched the videos, a few things became apparent. The phone screen looked completely black. We assumed that in the rush to close the payload, I forgot to turn on the screen. Also, there was a fair bit of payload spin, making the footage somewhat dizzying.

Everyone packed up in Vegas and headed back to HQ to begin reviewing the telemetry and footage.

We synchronized all the camera footage we had into one giant video: (Watch the four-camera view video on the Duo in Space website)

From there, we were able to create a timeline of events:
T-18:00 (07:45) - Camera Started
T-00:00 (08:03) - Launch
T+13:00 (08:16) - Camera Failure: Rear facing (15,000ft)
T+25:00 (08:28) - Last SPOT GPS Ping (30,000ft)
T+44:00 (08:47) - Camera Failure: Foward facing (48,000ft)
T+48:00 (08:51) - Camera Failure: Down facing (50,000ft)
T+90:00 (09:53) - Burst (100,826ft)
T+130:00 (10:13) - SPOT GPS Back Online (30,000ft)
T+147:00 (10:30) - Landing

Overall, the test flight was a huge success. We knew that we could successfully track the payload. All of our flight modeling calculations were correct, and the simulations accurately predicted where we landed. The servo didn’t quite freeze, but it began to struggle so that needed to be addressed. The smartphone survived and didn’t get too hot or too cold, but it did break off on landing.

Climate Control

Why the cameras failed, and at such wildly different times, needed further investigation, but I had a theory. As air pressure decreases, it’s increasingly difficult for electronics to shed heat. When the pressure drop of the ascent combined with the latent heat of running on the ground surrounded by inch-thick foam, the cameras might not have been able to shed excess heat and failed.

I set about testing this theory and looked at the parts in my shop from which I could build a low-pressure test chamber. While sticking a GoPro under a bell jar and sucking it down would have been relatively easy, I wanted to monitor the camera’s temperature. I needed electrical vacuum passthroughs. I also needed a way to monitor the pressure. Luckily, I had just the thing at my disposal: a 1980s-era sputter coater. These devices are typically used to coat electron microscopy samples in an extremely thin layer of gold by applying high voltage under a vacuum to a gold disc. It had everything I needed. I quickly put together a test chamber.

I borrowed the vacuum pump and vacuum gauge attached to Milly and connected them to the sputter coater. Then, I attached a K-type thermocouple to a carefully placed GoPro and the high voltage vacuum passthrough.

This allowed me to monitor the temperature and adjust the pressure from sea level to 130,000 feet almost instantaneously. With permission to potentially destroy a GoPro in this test, I set the camera recording and sucked the chamber down to 3.8 torr, around 120,000 feet. I watched the temperature slowly creep up. After just 20 minutes at simulated altitude, the front face of the camera, where most of the electronics resided, reached 125 degrees Fahrenheit and the camera went into thermal shutdown. I had found the culprit.

With the problem identified, I started thinking about potential solutions. I knew I couldn’t actively remove all the waste heat, but I could buffer it with some mass. I raced down to the local electronics recycler and purchased several old CPU heatsinks, strapping one to a GoPro with some Teflon tape and thermal paste.

The idea was that while there was airflow waste, heat would be dumped and there would be enough breathing room from the cold temperatures of the ascent to create a buffer at the higher altitudes until the balloon burst and started descending. Sucking the chamber back down to altitude, the camera lasted well over an hour with the added heatsink — safely within our margin.

A New Look

With the critical failure identified, I switched focus to the second payload. While a pink insulation foam box with a noodly finger sticking out of it was fine for the test launch, for the big show we wanted something that conjured the look and feel of a real spacecraft.

That’s when we asked Duo Security’s 2016 Women in Security Industry Award winner Scotland Symons for help. After some discussion about technical requirements and challenges, a design took shape.

We would fly a boxy, and slightly 50 percent larger, internal frame with an added panel sticking out of the side to attempt to quell the spinning we saw in the first launch. In addition to the front and bottom camera views, we moved the back camera to an offboard boom to give a more over-the-shoulder perspective. In place of the fluorescent orange finger, we called for an astronaut glove.

Because we planned to use a large outboard boom, attaching directly to the insulating foam as we had for the test launch was out of the question. The payload needed an internal skeleton, something lightweight yet strong. I chose MicroRAX, a miniature aluminum extrusion with decent attachment options. Once the parts arrived, I began assembling the main cube.

At this point, the true size difference really became noticeable. This payload would be much larger. Because the boom itself would stretch to more than three feet long and support a significant amount of weight, I turned to the space flight simulation game Kerbal Space Program for inspiration and designed some support struts to 3-D print in super-strong nylon. While I waited for the nylon to arrive, I prototyped them out of much weaker ABS plastic and handed over the entire frame to Scotland for the foam work.

Boop!

With the body out for initial detail work, I began making the new actuator assembly that would attach to the frame. Because the servo from the first launch started struggling with the colder temperatures, I chose a much stronger servo and then printed and mounted a linear actuator to a spare MicroRAX beam to stand in as the payload frame.

I took another beam and made a stationary platform for mounting the smartphone, as well as a sliding rail on top where I could attach the astronaut glove.

Scotland came by with the source material for our astronaut gloves — a pair of child’s snowboarding gloves shaped with wire into a natural position and filled with spray foam. To make the fingers touchscreen-sensitive, she coated them with Nanotips.

I quickly strapped one of the gloves to the sliding rail and fired up the Raspberry Pi scripts.

One of the last structural elements remaining were the new external camera mounts. I designed some nylon GoPro clips and tapped the heatsinks so that the cameras could just screw on and plug in.

The structural and mechanical aspects of the build were complete! I handed things back over to Scotland to mount the components and fit the foam form surrounding the frame.

Making the Connection

Finally, I had a chance to investigate what prevented the satellite phone from maintaining a connection. I could dial out to the internet via the satphone, but the connection became non-responsive after only a few packets and needed to be redialed. After a lot of troubleshooting, I identified the issue: the USB to RS232 serial adapter that I was using incorrectly implemented hardware flow control. In other words, after a few hundred bytes the serial driver inside the satphone thought that the Raspberry Pi hadn’t ingested all the data it had sent, and it would block until the serial port was reset.

I ordered a few different brands of USB serial adapters and found one that worked perfectly, and then I could finally start testing the logic with the satellite link. Doing development using an Iridium satphone is tricky. You have to test things as quickly as possible, because every minute of connectivity costs almost $2. Establishing a data call itself took a couple minutes, so even before any application-specific logic was executed I had to burn $4. This also meant that I’d have to limit the push attempts; keeping the connection active for the duration of the flight would be cost-prohibitive.

Iridium’s current satellite constellation is extremely old and finicky, but it’s one of the few providers covering the continental U.S. Its underlying transport operates at a maximum of 2400 baud. Even with full signal strength, I was looking at only ~0.3kb/s of throughput. It would halve for every “bar” lost, and latency would skyrocket to upwards of 10 seconds round trip.

Previously, I was detecting the incoming Duo Push by parsing out the certificate common names as they came through the Raspberry Pi from whatever uplink was available. With such a slow and latent connection, this simply didn’t work. The TLS handshake was not even completing by the time the gloved finger struck the phone. Even worse, it was sporadic, sometimes taking 10 seconds, and other times a minute. I had to devise a completely new approach.

What I came up with was more of a Python-powered Rube Goldberg machine than anything else. After the Raspberry Pi booted and initialized itself, it would sit on the serial connection and wait for an incoming call. When I wanted to send a push to the payload, I would literally call the satphone on the payload. The Raspberry Pi would answer the call and send me a DTMF beep code, telling me that it acknowledged my request, and hang up. This would tell the Raspberry Pi to establish a 15-minute long connectivity window, where it would continually try to connect to the internet.

Once the payload detected that it established some semblance of an internet connection, it would send a packet back down to a waiting ground station script with the connection latency and notify it that it was clear to send a Duo Push to the payload. The Raspberry Pi would then restart the Duo Mobile app running on the phone over ADB and, by doing so, initiate an outstanding push check.

While this was happening, a separate thread on the Raspberry Pi periodically checked the color of a pixel in the lower left corner of the screen. A green pixel meant that a push request had come in, and it caused the servo controller to actuate and approve the request. This had the hilarious side effect of the glove continuously poking at the screen, trying to approve the request, whenever a shot of the Duo Push screen displayed. Once the 15-minute connectivity window closed, the Raspberry Pi would disconnect from the internet and return to its waiting-for-incoming-call state.

With Duo’s extremely easy-to-use integration API, I wrote a ground control script that I could log in to from my laptop to trigger this chain of events. I placed all the communication components into the frame, powered them on and set the frame outside. I ran the ground control script, entered my fake credentials and called the satphone. After three rings, the line went quiet for a brief moment before a familiar DTMF tone typically made by pressing the number 5 played for a full second, and then the line went and the call ended.

We were off to the races! The payload should’ve have initiated its connection window. As I watched the Duo Auth API poll requests come back, I nearly burned a hole through my screen with the sheer force of my gaze. Then it happened:

The Duo Auth API returned success! My script dumped me in a fake terminal, and I jumped for joy. I managed to squeeze a push through an improbably small pipe. I spent the next few days doing more testing and adding coverage to the various edge cases in all the state machines. By the time I was done and happy with the software stack, only 50 minutes remained of the 300-minute SIM card. We only had three connectivity windows left before Iridium cut us off. The next one was planned to be right as we began the balloon chase.

A Spacecraft Is Born

While I was getting the software to where it needed to be for the launch, Scotland was busy turning pink foam into a spaceship.

Once we had the payload built and dressed, we took a look at its launch weight to recalculate fill volumes. Our previous payload weighed about seven pounds, quite heavy for a balloon payload, and the new one was 10.6lbs.

We absolutely had to hit our target ascent rate of five meters per second or risk losing the payload, so we were faced with some choices:

• Put an extra 100 cubic feet of helium into the balloon to bring launch volume to around 300cF. This would cost us 10,000 feet of maximum altitude and put extra stress on the one balloon.

• Have our latex balloon supplier ship us a larger balloon (3kg) and pray it arrived in time for the launch.

• Use our existing backup balloon in addition to the main balloon to lift the payload with complicated rigging, smashing our altitude at the risk of potentially having the balloons tear each other apart.

• Switch to using hydrogen for an extra 15% of lift at the same gas volume, at the risk of blowing ourselves to smithereens Hindenburg style.

• Start stripping parts off the payload.

Needless to say, we chose to eat the 10,000ft altitude cost and overfill our existing flight balloon. The largest helium tank available to us had a capacity of only 220cF, but we needed slightly more than 300cF. This decision added some complexity to the launch plan; during the filling operation, we somehow had to swap the heavy helium tanks.

As launch day approached and weather forecasts began to stabilize, we ran numerous flight simulations. This time, the payload would head northwest of the launch site, as opposed to southeast, and it was most likely to land in the desert plains surrounding Jean, Nevada. Launch for the Saturday morning of DEF CON was a go!

We notified the FAA and of our intent to launch and filed a Notice to Airmen for the upcoming weekend.

Up next: Time for the official launch