This article was meant to be a... tweet. A thread of few tweets to be precise. But when I started to dig more and more into statistics and real satellite cases, I found so many interesting details that it would be a shame to shorten it to several dozen sketchy social media flashcards. So here we are with a catchy topic: how not to kill our satellite when it’s already in space.
The background story
As always, the idea of the following article’s subject didn’t come out of nowhere.
Starting from the very beginning: here, in Poland, the participation in the satellite industry is still very small. Before the end of June, we, as a country, had only launched 6 satellites into LEO (two scientific, three academic, and one commercial proof-of-concept). I already wrote a hopelessly long text about all polish satellites, so I won’t repeat myself more than needed - the only important information here is that every incoming launch becomes The Event. And so it was with two new CubeSats - STORK-4 and STORK-5 - that were launched on the 30th of June this year. They are said to be commercial, built for Earth observation purposes by a small company called SatRevolution.
How is this case different from any other satellite launch, you could ask? Yet another bricky project - and taking into consideration that there were more than 11,000 objects put by the human race in the dark and scary space with almost 1,560 CubeSats included (according to the Nanosats Database), the next one really shouldn’t be a big deal. But it is kind of a loud case, at least here in Poland. Why? Mostly due to the fact that there was absolutely no information released about the mission status - neither after one day nor after a week. The polish space sector started to speculate if satellites are even working properly. Two weeks after the deployment, on the 15th of July, the company published a short notice that the communication was established and both birds are chirping as it was expected. So what is the issue? It’s the fact that… they admitted that they made the first contact on that day. Two weeks after the start of the mission. And it was announced that it was not because of some unexpected problems with the ground station or other sudden issues - no. The excuse was clearly indicated: the vacation.
Fig. 1 - rough translation: Time just before the launch was extremely intensive for our team. That’s why, after the event, most of the people engaged in the project went on a well-deserved vacation. It delayed the first contact a little bit. We are glad that we managed to start the communication on the first attempt.
Source: Space24
I’m not here to criticize anyone and I won’t do that. I just realized that maybe, maybe… there actually are people who just don’t know about the possible danger of delaying the first communication with a new shiny CubeSat? My goal is always to popularize any knowledge that could be useful for someone (and I always like to think that if something was valuable even for one person, it was worth working on), so that’s why we are here. To debunk myths and find evidence that communication from the very first day of the mission is crucial for its success. Let’s start with science!
Fig. 2 - Please, don't do this. CubeSats want to live...
A bunch of statistics
First analyze our favourite thing: data. According to the already mentioned Nanosats Database, the number of CubeSats in orbit is growing rapidly every year. Small orbiters started to gain some popularity in the late 90s, achieving the peak last decade - with predictions that this craziness it’s not going to stop for at least the next few years. It’s easy to understand: testing new technologies, learning, and fulfilling scientific goals in space is quite inexpensive and accessible with CubeSats; no matter if you represent a group of students or a small company - a CubeSat is within your reach.
However, there are also disadvantages of small satellites utilisation. Space is hard in many ways and it doesn’t spare anyone. When your device is meant to weigh only a few kilograms maximum with strictly limited size, there are not many options available to protect its guts. You have to resign from additional shielding or redundancy to be able to fit into the standard - and these decisions come with some risks.
Fig. 4 - Reliability of CubeSats compared to bigger satellites
Source: Methods for dependability analysis of small satellite missions, Oibols G., 2015
Fig. 5 - Reliability of CubeSats in different sizes
Source: Improving Mission Success of CubeSats, Venturini C., Braun B., Hinkley D.
There are two main concepts referring to the satellite’s deaths - Death on Arrival (DOA) and Infant Mortality. Both are quite self-explanatory; while the first one describes the situation that the CubeSat never gives any sign of life from the lauch - the second focuses on missions with deadly failure during the commissioning phase. Excluding launch failures that are not the case here, the ratio of infant mortality was dropping for the first few years when these kinds of missions gained popularity, but now it’s stuck somewhere between 15-25%.
Fig. 6 - Infant mortality for CubeSats from 2002 up to May 31, 2018 (launch failures excluded)
Source: Towards the Thousandth CubeSat: A Statistical Overview, T. Villela, C.A. Costa, A.M. Brandao, F.T. Bueno, R. Leonardi, 2019
We could, with some probability, assume that now, three years after preparing these statistics, the rate could be slightly better. There are quite a few papers about this topic (that are linked at the bottom of this article) and it’s clearly visible that the information and charts included in them is nonidentical. It is understandable because of different groups of satellites taken into consideration in specific research and the time of publication. Depending on the database used, the numbers of DOA and early losses vary, but it’s always a non-zero value. Also, some of the satellites’ fates are marked as unknown, which can also suggest some problems.
Fig. 7 - Infant mortality for CubeSats from 2002 up to May 31, 2018 (launch failures excluded)
Source: Towards the Thousandth CubeSat: A Statistical Overview, T. Villela, C.A. Costa, A.M. Brandao, F.T. Bueno, R. Leonardi, 2019
Although there are many statistics about the likelihood of satellites’ failures, the engineers' beliefs are a completely different matter. One really interesting research from 2016 about CubeSats’ failures points out that even knowing what is the probability of the mission to fail within the first 6 months, developers use to think that their own university-built mission failure’s likelihood is a lot, lot lower. Estimation for somebody else’s mission is slightly below 50% while, for their own creation, it varies between 15 and 20%. Intriguingly optimistic, isn’t it?
Fig. 8 - What do engineers think about probability of death of someone else's CubeSat
Source: Reliability of CubeSats – Statistical Data, Developers' Beliefs and the Way Forward, M. Langer, 2016
Fig. 9 - What do engineers think about probability of death of their CubeSat
Source: Reliability of CubeSats – Statistical Data, Developers' Beliefs and the Way Forward, M. Langer, 2016
So - why shouldn’t the communication with a satellite be delayed?
To be honest, for me it’s even hard to imagine such a delay in communication. Mostly because there is… no good reason to wait? If we take this problem and look at it from the other point of view, trying to find one good cause to wait some time before the first contact, we should, with quite a big probability, fail. It doesn’t make sense from a technical, business or even PR point of view. Starting the commissioning phase as early as possible should be in great interest of any party connected to the satellite production and utilisation - from engineers to sales. We can get results of our experiments from the start of the mission, avoid issues connected to long exposure to radiation, or even sell photographs taken by an Earth Observation CubeSat and start making money from it at no time.
Moreover, there are many good reasons to start the communication from the moment of the launch. I made a bet with myself that I can find at least as many reasons as there are Harry Potter novels in the series. So here there are: the adventures of Harry Cubber.
Fig. 10 - Harry Cubber
1. Harry Cubber and the Philosopher’s Position
In one of the previous articles about the satellites' orbits I mentioned the so-called TLE. In brief, it’s the short notice that includes the information about the object’s orbit (inclination, mean motions, eccentricity etc.) that allows us to compute the exact position of the CubeSat at any time. The 18th Space Control Squadron from the USA, responsible for tracking Earth’s orbiters, uses observations from sources like optical, radar, or radio, to predict their current location and path.
What does it have in common with the communication, you could ask? Remember that you need to know the precise location of the satellite to be able to “talk” with it - especially when using S-band or X-band frequencies or very directional antennas for UHF/VHF. Launch for many objects is often connected - a dozen or even several dozen devices can be placed in orbit in very short time intervals. In this case, there is only one TLE released at first and after a few days, it’s being divided into a number of notices marked as `OBJECT-A`, `OBJECT-B`, etc. People who are preparing TLEs don’t know which satellite is which - they only know that there is a new device in space with these specific orbital parameters. So, if we don’t follow our satellite from the very beginning, it will be harder to find it later - there will be more and more new objects being randomly extracted from the first mother-TLE and the time difference between them will be growing. It means that we will be forced to try to communicate with all possible satellites as long as we find ours. That is also why the beacon is so important to satellite localisation - but we’ll come back to this topic later.
In summary: it’s not a critical thing, but following your CubeSat from the start makes your life easier. You don’t have to experiment with a lot of random TLEs (especially if there were dozens of objects released at one time). You can safely focus on the important stuff - the mission itself.
2. Harry Cubber and the Chamber of Probability
Probability is the branch of mathematics concerning numerical descriptions of how likely an event is to occur, or how likely it is that a proposition is true - says Wikipedia. So how does the probability work in terms of the space missions’ failures? In the previous chapter, we discussed some interesting statistics that clearly suggested that the probability of a satellite’s failure changes with time in orbit. There are also a few sources that show one more intriguing fact about practically any hardware system - a bathtub-shaped line that describes a failure rate in function of time.
Fig. 11 - General evolution of failure rate over the life-time of a hardware system
Source: Methods for dependability analysis of small satellite missions, Oibols Gerard (2015)
In the early life region, a probability to fail is quite high, reaching some constant values in the middle of the mission and increasing again when the components wear out in the end. If we think about the spacecraft, this chart seems to be quite valid too. Taking this into consideration, we should start the mission as early as possible - delaying the start of communication for a few weeks can lead us to find the satellite... already dead.
Here, as an example, let’s talk about the PicSat case. Satellite’s operators made the first contact almost instantly, deployed solar panels a week after a launch, and, one more week later, activated the science payload. Sounds like a perfect mission, doesn’t it? But that’s when the problems began - they wanted to point the cube, using off-the-shelf ADCS, towards one chosen star. It turned out to be impossible, because of the sudden subsystem failure. Operators tried to fix the issue by telecommands, software updates etc., when, on one day, the satellite just… fell silent. Ten weeks after the start of the mission, it just stopped responding and never came back to life. So, let’s go ahead and make a thought experiment. If they waited for the communication for two or three weeks, they would lose a precious time of their CubeSat’s lifetime. Maybe they wouldn’t have time to test the software update mechanism, maybe they wouldn’t define the root cause of the ADCS failure? And going further - if they waited for 10 weeks (god, no), they would probably think that their cube was dead from the beginning of the mission!
Fig. 12 - About PicSat death
Source: Short life and abrupt death of PicSat, a small 3U CubeSat dreaming of exoplanet detection, M. Nowak, S. Lacoura, A. Crouziera, L. Davida, V. Lapeyrerea, and G. Schworera, 2019
3. Harry Cubber and the Prisoner of Energy Budget
The energy budget is one of the most important topics for me personally - especially after KRAKsat failed and became stuck in a boot loop, mostly because the calculations were wrong. If the CubeSat needs more energy to work than it can get from the solar panels, the race with the time begins. The voltage on the battery is decreasing - first slowly and then so rapidly that there’s no chance to save the spacey child. If you recognize the issue early enough, you can either try to turn off some unnecessary subsystems or just perform your most important experiment very quickly. It is then essential to have everything planned in advance to be able to act as soon as the potential energy deficit becomes visible.
Fig. 13 - Battery voltage- KRAKsat
Source: KRAKsat Lessons Learned, A. Musiał, D. Markowski, J. Życzkowski, 2019
How is this connected to the communication timings? If we wait with the first contact, we will never know about the battery status at the beginning of the mission. All the important data that could help us recognize some power issues very quickly, is just… gone. And we can wake up with our satellite already discharged or in a boot loop, which seems even worse.
As a curiosity - we have one more example of this case, even in our small polish space industry. It’s PW-Sat, the first student CubeSat placed on LEO in 2012. Although it started to transmit a beacon just after the deployment, the operators’ team had some problems with the mission. They came back from the launch spot after a few days of celebration, so the communication was delayed a little bit - and the device itself seemed not to react to the sent telecommands. The main experiment, the impressive deorbit tail, was never opened and the energy level started to decrease gradually. It turned out that the energy budget was calculated mistakenly and after a short period in the boot-loop, PW-Sat went silent forever.
4. Harry Cubber and the Mission is on Fire, oh no, no, noooo
Energy budget and decreasing battery problems are not the only issues that we could overlook if we start the communication late. Many CubeSats have some mechanisms that need to perform an action while in orbit - e.g. unfold antennas, open solar panels or extend some scientific payload. If we don’t check the subsystem status early enough, the mission could fail on many levels and in different styles. One of the examples of that kind of issue could be still folded antennas that could cause unexpected problems with communication and, while trying to “talk” to our satellite, we will accidentally burn the radio. Solar panels that didn’t open are also a problem (connected to point 3 of this list) - without them, the energy consumption could be higher than its production, and our poor cube could discharge while we’re enjoying our holiday trip. Some software bugs can also be visible only after some time - and not having enough data from the start of the mission, you could never know if it’s a new error or something that was present from the deployment. To summarize: without communication in the first days of the mission, it’s easy to overlook important issues and accidentally damage the orbiter.
Example time: let’s think about the MOVE-II mission. MOVE-II is the German CubeSat released in December 2018, whose mission was to test the satellite bus built by students at the university.
Fig. 14 - Timeline of the MOVE-II mission
Source: https://www.move2space.de/MOVE-II/satellite/launch/
If we take a look at the above timeline, there is one thing that stands out - the time gap between the first contacts and the real commissioning. It’s been more than a month! There are two possible causes for that: one is more official - ground station problems - and the one less official - student winter break. As we can see, when the next semester started, operators came back to the university, just to find their satellite… rotating like crazy. According to the papers, it spun up to more than 500 deg/s! After a lot of calculations and trials, the team managed to take control over the object, but it still requires constant observation and corrections to maintain stability. They had an extreme amount of luck that the satellite was even responsive and the battery was still charging while spinning. If you are interested in this case, I strongly recommend the MOVE-II team’s publications on ResearchGate. But yeah, another example that letting the satellite live its own life can be… risky.
Fig. 15 - you spin me round round, baby, round round 🎵
5. Harry Cubber and the Order of the Single Event Effects
What is the most dangerous thing in space that destroys not only humans but also electronics? The answer is quite intuitive: radiation. Every day, every hour and every minute our poor satellite in orbit is being bombarded with high-energy particles. Here, on Earth, the atmosphere works like an umbrella, protecting us from their destructive effects - but spacecraft is exposed to danger all the time. Electronics can be quite sensitive for this kind of impact and there are specific areas around our planet where the radiation influence can be ever more evident (e.g. South Atlantic Anomaly, where Van Allen radiation belt comes closest to Earth's surface, even to an altitude of 200 kilometers).
Radiation can cause a very dangerous thing called Single Event Effect (SEE). One bad particle hits the electronics - and, ta-da, we have a bit flipped. Somewhere in the satellite’s memory a zero changes to one or one changes to zero. Not a big deal, you could assume? But imagine, it was one bit inside very important software. Maybe somewhere in the on-board computer’s function that was responsible for communication… Or opening the solar panels. Or in a sensor’s telemetry data. Although there are a lot of mitigation techniques, we can’t assume that every subsystem is absolutely safe. Not everything in our satellite can be redundant (remember, it has to fit the CubeSat standard!), and not every possible bit flip is easy to predict.
There is also another possible issue, even more dangerous - also caused by a single crazy particle. Single Event Latch-up. It can be defined as a short circuit inside an integrated circuit, which triggers a parasitic structure that can lead to its improper function due to overcurrent. What makes it so vicious, is the fact that you can’t see any sign of this event happening - except sudden and random higher power consumption. You can stop it only by a power reset.
Not-so-fun fact: if the operators of the PW-Sat2 mission spend the first two weeks on the beach, the satellite would be dead before their return. Three days after the deployment a single event latch-up occurred.
Fig. 16 - Latch-up on the beginning of the PW-Sat2 mission
Source: data from PW-Sat2 team
Battery voltage started to decrease rapidly and only a quick reaction of the team saved the satellite (by a reboot). After the manual restart, the situation came back to normal and the positive energy budget allowed the battery to recharge to the default value. You can read more about radiation in the PW-Sat2 mission on the blog, unfortunately only in polish.
6. Harry Cubber and the Half-Eyed Lens
Radiation is not only about rapid single events, but there is also another important phenomenon called Total Ionizing Dose (TID). As the name suggests, it can be defined as the impact of the particles hitting the spacecraft - but in the longer perspective. This effect gradually degrades transistors, shifting their characteristics, and - what’s important - the changes are cumulative and practically irreversible. It can lead to improper functioning of the electronics or, for example, destroy parts susceptible to damage.
As we already mentioned, there is not enough space in the little CubeSat to put all needed radiation resistant shielding or the redundant subsystems inside. We have to live with the sad perspective that our mission won’t last very long. Optics is one of the most sensitive parts, so it won’t resist radiation very long. It’s no coincidence that the majority of observation satellites are really, really big. In small objects, it’s harder to protect lenses from the radiation influence and we can quite quickly be left with blurry or punctured with dead pixels photos. The degradation is clearly visible for example on photos taken by Lem or Heweliusz from the BRITE constellation. Just to be precise: they were not CubeSats, but also small satellites (~20x20x20cm), so the limitations connected to the payload were quite similiar.
Fig. 17 - Full-frame from one of the BRITE's CCD sensor. Visible degradation.
Source: Radiation defects of BRITE CCD sensors, A. Popowicz, 2015
7. Harry Cubber and the Deathly Ignorance
Hi, so you are back from the vacation?! How was it? Was the weather nice? I know you probably forgot about this CubeSat we sent to space two weeks ago, but I’m afraid we should go back to that topic. Okay, the antenna, the radio… Everything is ready, send the telecommand!
…what? No answer???
For me, this kind of scenario is so unreasonable that it’s even hard to imagine. But assuming that SatRevolution’s CEO was honest in the press, a similar situation really took place - of course with the satellite working somehow in the end. But what would have happened if STORKs didn’t answer the commands?
One of the first European Space Agency student satellite projects, called SSETI Express, became known as a satellite “quickly built and quickly gone”. As a victim of very quick discharge, caused by the power system failure, it was working properly only for 8 hours. If the operating team waited with the communication even for two days, they would never know what caused the satellite’s sudden death.
If we know what issues were the main cause of the mission failure (there are often more than one), we can draw conclusions and learn from our mistakes. Without this knowledge, we are left in the void - something never works and we have absolutely no idea why. And I think, personally, that this point is the most important one. There is no place for ignorance and procrastination in the space mission. We have to be prepared and gain as much data as possible to be able to react quickly in any unfavorable circumstances.
The mission doesn’t end when the satellite is built. It’s the moment it starts.
To beacon or not to beacon?
Concluding this article, let’s touch on one more small topic. Maybe some of you noted one more weird thing connected to STORK-4 and STORK-5 missions - lack of beacon. In terms of space engineering, a beacon is usually defined as short notice that provides important information about a satellite's status and is sent automatically with a fixed interval.
Fig. 18 - First KRAKsat beacon.
Source: SatNOGS
Using the small turnstile antenna I recently built, I tried to track STORKs from the deployment day - and I wasn’t the only one listening. More radioamateurs from SatNOGS found these birds interesting and aimed their antennas at the sky, with the hope that they could catch some signals. And it seems we were all… wrong. As you can see on the waterfalls below, STORKs never “talk” without action from the satellite’s operators. No beacon then - why?
Fig. 19 - Empty waterfalls with STORK-4 observations (2.07 - 13.07); recordings from my own station
Source: SatNOGS
To be honest, I have absolutely no idea. I have never heard about any satellite without a beacon (if you did, please let me know, it would be extremely interesting!). One day I came across a project called Enoch, which was literally an... urn for the ashes of a cremated person. It was deployed together with PW-Sat2 in December 2018. If there was meant to be even one satellite project without a beacon, we could possibly speculate, it should be this one. But, I checked it today and... surprise. There was a separate small device attached, called CUBIT, which responsibility was only to transmit in short intervals. So, even the space urn had a beacon.
Fig. 20 - A good plan for eternity?
Source: https://www.nytimes.com/2018/11/13/arts/design/spacex-enoch-tavares-strachan.html
What kind of threat for the mission the lack of the beacon poses then?
Without a beacon, it’s harder to find a “lost” satellite. As we already noticed, TLEs update frequency can sometimes be laggy, mostly if many orbiters are deployed at the same time. If there are many TLEs from one launch, it’s sometimes difficult to find our spacey child - a beacon helps spectacularly. We can listen on a specific frequency and compare the signal we got with the possible TLEs, quickly finding the one that is valid for our satellite. There is great software for these purposes available here.
Fig. 21 - Matching a satellite transmission with a TLE using STRF software
Without a beacon, we don’t have information about the CubeSat status from all over the world. There are many amazing radioamateurs with incredible radio equipment that are often willing to help with the mission (e.g. on SatNOGS). The presence of a beacon allows them to get important status data about the satellite from any place on Earth. They don’t have to be near the ground station, where they could listen only to the commands’ answers - their activity can be completely independent. Even if the operating team has some problems with their main ground station, any antenna can always act as a passive backup - providing at least some information about the orbiter’s vital functions. And if the satellite goes silent, we can at least try to follow its way through the orbit, speculating if it stopped working above the Anomaly or somewhere completely unrelated. Maybe it transmitted some valuable data just before turning itself off and somebody recorded it? Frequent beacon = more possibilities. Not to mention the situation with KRAKsat - our CubeSat was silent for the first two weeks and we almost started to lose hope that it's going to start transmitting again. It did! But only because of the automatic beacon we knew it was alive.
Oops!...I Did It Again
Maybe it’s not a coincidence that Britney Spear’s music video for this old hit starts with the landing on Mars… When it comes to space (maybe not deep space, but LEO and small science, student, or even commercial CubeSats) mistakes are being repeated quite passionately. Why is this happening? My theory is connected to the lack of “lessons learned” published after missions. Someone builds a satellite, pushes it into orbit - then something goes wrong and… silence. There are no papers published, no blog posts, no presentations about the failure. Problems and errors are cheerfully swept under the rug. The mission was a success? Yay, let’s announce it everywhere we can! The mission was a failure? Welp, what mission? I never heard of any mission. Of course, there are glorious exceptions to this rule and there are some really good documents about failures you can find on the internet - but it’s still not something common to publish them. It’s a pity, but I don’t really see the way to change it easily - I can only emphasise the importance of learning from somebody’s else mistakes instead of doing the same shitty errors by ourselves over and over again.
So, if you ever built a CubeSat - share your experience! Even if you failed. And, on the other hand, if you are planning to start your own satellite mission - read about all previous missions as much as you can! It will pay off, I promise :).
Thank you :)
Thank you all for your atention! If you have any comments or other failed CubeSats’ use cases, don’t hesitate to share them in the comment section.
PS How could I even think it could possibly be fitted into a few tweets?... 😬
Links
How the commissioning can be performed? - interesting articles
- https://www.capellaspace.com/8-steps-for-commissioning-a-sar-satellite/
- https://www.esa.int/Enabling_Support/Operations/Sentinel-2_operations
- https://www.isispace.nl/news/brik-ii-first-signals-received/
- https://www.inmarsat.com/en/insights/corporate/2015/meet-the-satellite-gatekeepers.html
- Nanosats Database, [access: 19.07.2021] - https://www.nanosats.eu/
- Methods for dependability analysis of small satellite missions, Oibols G., 2015
- Improving Mission Success of CubeSats, Venturini C., Braun B., Hinkley D.
- First Flight Results of the MOVE-II Satellite, S. Ruckerl and others, 2019
- Short life and abrupt death of PicSat, a small 3U CubeSat dreaming of exoplanet detection, M. Nowak, S. Lacoura, A. Crouziera, L. Davida, V. Lapeyrerea, and G. Schworera, 2019
- KRAKsat Lessons Learned, A. Musiał, D. Markowski, J. Życzkowski, 2019
- Radiation defects of BRITE CCD sensors, A. Popowicz, 2015
- Towards the Thousandth CubeSat: A Statistical Overview, T. Villela, C.A. Costa, A.M. Brandao, F.T. Bueno, R. Leonardi, 2019
- Reliability of CubeSats – Statistical Data, Developers' Beliefs and the Way Forward, M. Langer, 2016
- MOVE-II website, [access: 19.07.2021] - https://www.move2space.de/MOVE-II/satellite/launch/
- Czy satelity STORK-4 i STORK-5 działają?, Kosmonauta.net, [access: 19.07.2021] - https://kosmonauta.net/2021/07/satelity-stork-4-i-stork-5-prawdopodobnie-nie-dzialaja/
- PW-Sat2 Blog, [access: 19.07.2021], https://pw-sat.pl/en/project-blog/
- Article about Enoch satellite, [access: 19.07.2021] - https://www.nytimes.com/2018/11/13/arts/design/spacex-enoch-tavares-strachan.html