Hi Job, On Fri, 31 Jul 2026 at 15:44, Job Snijders via routing-wg <routing-wg@ripe.net> wrote:
On Thu, Jul 30, 2026 at 10:52:28PM +0200, Lukas Tribus wrote:
There are many reasons rpki-client could stop validating. Connectivity issues, upstream FW changes, read-only FS, no enough memory, not enough disk space, bugs, permission problems, "it's a container so nobody understands what is going on, but the lights are all green".
This seems weird to pick on rpki-client this. What software is resistent against hardware failures or execution in a unfitting environment?
FWIW I don't feel like I'm picking on rpki-client at all :)
exactly these reasons, a sane program will error out with a non-zero exit code when problems are detected. Whether the program is one-shot or long running is irrelevant in this context: the operator must monitor the process execution. Rpki-client even warns the operator on STDERR when it suspects there won't be enough inodes or disk space.
I agree completely. I'm just disagreeing with the notion that it is "easy to setup", as suggested in this thread. A single software stack is easier to monitor than 2 software stacks that have to talk to each other. It's is only natural that this architecture has more moving parts, and with more moving parts comes more external monitoring requirements.
All in one software stacks will crash and tear down RTR sessions with them in most of those situations.
This doesn't match my experience: I've discovered catastrophic silent bugs in basically every RPKI validator projects. And not even all of those bugs have been solved when I last took stock!
I do agree that programs which as a general rule 'crash hard and fast' are easier to manage than programs that just 'limp on'. In my experience it is easier to construct reliable setups when you can monitor each individual component in the pipeline (fwiw, both rpki-client and StayRTR support OpenMetrics/Grafana, which helps me with faster fault identification). This is also why I love programs that properly set exit codes.
That's exactly my point, it needs monitoring. It depends only on the administrator/operator that is setting up those services, which is what this thread is about (how easy or hard it is to properly run a RPKI RP and RTR Server in a reliable way). This thread and my points are not about software quality.
To pivot to a more constructive line: in the realm of dead man's switches, I've grown fond of https://healthchecks.io/ (up to 20 monitors is free) linked with Pushover ($5 one-time purchase). Very cheap way to keep an eye on whether backups or rpki-client invocations are humming along nicely.
Agreed, I'm using the very same 2 services as well.
Alternatively, OpenBSD's crontab implementation has built-in 'cronic' functionality. In systemd do this: https://wiki.archlinux.org/title/Systemd#Notifying_with_e-mail
I'm using chronic in regular cronjobs on Linux. Same here: the person setting those services up needs to know this and set it up accordingly. I'm not worried about my clients or your clients. I'm worried about the guy that has to setup all of this, without the time to properly analyze all of those possible operational problems. Like the original poster in this thread. I have written rtrcheck as a simple nagios compatible plugin and written a RIPE labs post about it in 2020 and after 6 years I am not aware of a single user of rtrcheck besides my own clients. We all like to think that at least the big SPs are tracking and monitoring those metrics, and then we find out that the reality is not not great. I also feel like there is an amount of hindsight bias involved in these discussions. I always hear that everything is so obvious. And yet people don't do and don't really know how to do proper monitoring. Perhaps a proper BCP document about these operational aspects would be good idea. But then I will probably just get the usual "Why? Everything is so obvious!" ...
With StayRTR v0.6.3+ your RTR count will go to 0, the RTR connection will keep running. Also something you'd wanna know, and it will probably not generate syslogs on your router.
Wouldn't any validator's count go to zero if there are connectivity issues to the rest of the Internet (but not with the BGP routers it is serving)?
Yes, I'm just advocating for monitoring for RTR serial monitoring here.
Every RTR server should be monitored against a never changing RTR serial,
Yes.
but the all in one validation + RTR packages are a lot less painful to deal with in these situations.
On the other hand, separation of the 'validation' and 'RTR distribution' function allows for seamless upgrades of the security-sensitive component WITHOUT flapping the BGP router-facing RTR sessions. Different deployment models bring different benefits to the table.
That is absolutely a big advantage. *I* would use rpki-client/stayrtr every day of the week. I'm just not recommending it to someone that I know will think of monitoring "at some point in the future", I'd rather this operator uses a All-in-One stack if he can't invest the time for proper operation *before* going into production, because that is likely a less severe problem than the alternative. Nothing about this is about software quality. Best regards, Lukas