RPKI Validator 3 Disk Space Requirement
Hi, the wiki page [1] states:
The repository objects are stored in a file based database, rather than in memory - we recommend at least 10GB of available disk space (under /var/lib/rpki-validator-3/db for the RPM).
the rpki-validator.h2.mv.db file is already >22GB in size, I assume there is no way to reduce the disk usage requirements via a configuration change? How big will the db file become? thanks, nusenu [1] https://github.com/RIPE-NCC/rpki-validator-3/wiki#system-requirements -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Hi nusenu,
Op 8 okt. 2018, om 10:54 heeft nusenu <nusenu-lists@riseup.net> het volgende geschreven:
Hi,
the wiki page [1] states:
The repository objects are stored in a file based database, rather than in memory - we recommend at least 10GB of available disk space (under /var/lib/rpki-validator-3/db for the RPM).
the rpki-validator.h2.mv.db file is already >22GB in size, I assume there is no way to reduce the disk usage requirements via a configuration change?
How big will the db file become?
There is a config setting regulating how often the validator will remove the "garbage", i.e. unused objects that it still stores but they are not really used but kept just in case (no connectivity to download updated records, broken manifests, CRLs, etc.). The setting is called "rpki.validator.rpki.object.cleanup.grace.duration" and by default it is set to "P7D", i.e. a period of 7 days, which is pretty conservative. You can set it to much smaller periods, 1-2 day or 12 hours and still have reasonable behaviour. There is another parameter “rpki.validator.validation.run.cleanup.grace.duration" with the similar meaning, but it's less crucial when it comes to storage space. Also Validator version 3 is lighter than 2 in a sense that it consumes less memory, but it can only do it by storing more data on the disk. I hope this helps. Kind regards, Nathalie Trenaman RIPE NCC
the rpki-validator.h2.mv.db file is already >22GB in size, I assume there is no way to reduce the disk usage requirements via a configuration change?
How big will the db file become?
There is a config setting regulating how often the validator will remove the "garbage", i.e. unused objects that it still stores but they are not really used but kept just in case (no connectivity to download updated records, broken manifests, CRLs, etc.). The setting is called "rpki.validator.rpki.object.cleanup.grace.duration" and by default it is set to "P7D", i.e. a period of 7 days, which is pretty conservative. You can set it to much smaller periods, 1-2 day or 12 hours and still have reasonable behaviour. There is another parameter “rpki.validator.validation.run.cleanup.grace.duration" with the similar meaning, but it's less crucial when it comes to storage space. Also Validator version 3 is lighter than 2 in a sense that it consumes less memory, but it can only do it by storing more data on the disk.
I hope this helps.
thanks for your reply. I changed it to: rpki.validator.rpki.object.cleanup.grace.duration=P1D in conf/application.properties, restarted the validator and expected some reduction in disk usage after a day or so, but the db folder uses almost 50GB now.. What disk usage should I expect / prepare for? There are also some errors shown on stdout (details see bellow). Since there doesn't appear much activity on the RPKI validator 3 issue tracker a general question: What is the general maintenance status of RPKI Validator 3? thanks, nusenu ERROR pool-1-thread-1 o.h.i.ExceptionMapperStandardImpl HHH000346: Error during managed flush [org.hibernate.exception.ConstraintViolationException: could not execute statement] ERROR pool-1-thread-1 o.s.s.s.TaskUtils$LoggingErrorHandler Unexpected error occurred in scheduled task. org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint ["RPKI_OBJECT__SHA256_IDX ON PUBLIC.RPKI_OBJECT(SHA256) VALUES (X'e17d74b2082b15701d300e24d97a3d7b0eb081d59034c6d026b9555b0509bc0e', 3443950)"; SQL statement: /* insert net.ripe.rpki.validator3.domain.RpkiObject */ insert into rpki_object (created_at, updated_at, version, authority_key_identifier, last_marked_reachable_at, serial_number, sha256, signing_time, type, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [23505-197]]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:259) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:225) at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:540) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:746) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:714) at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:532) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:304) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at net.ripe.rpki.validator3.domain.validation.RpkiRepositoryValidationService$$EnhancerBySpringCGLIB$$3380f99b.validateRsyncRepositories(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:178) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3032) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3547) at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:600) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:474) at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337) at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39) at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1437) at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:494) at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3245) at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2451) at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:473) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:156) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$100(JdbcResourceLocalTransactionCoordinatorImpl.java:38) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:231) at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:68) at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:536) ... 21 common frames omitted -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Hi Nusenu,
Since there doesn't appear much activity on the RPKI validator 3 issue tracker a general question: What is the general maintenance status of RPKI Validator 3?
Don’t worry, we’re keeping track of all the issues you’ve posted. Soon we’ll turn our focus back to validator 3. For the time being, we’re going to investigate why such big file. The rpki-validator-3 is open source and the community is more than welcome to send pull requests and/or create new issues. Cheers, Thiago
Thiago da Cruz:
Hi Nusenu,
Since there doesn't appear much activity on the RPKI validator 3 issue tracker a general question: What is the general maintenance status of RPKI Validator 3?
Don’t worry, we’re keeping track of all the issues you’ve posted. Soon we’ll turn our focus back to validator 3.
that sounds great.
For the time being, we’re going to investigate why such big file.
The rpki-validator-3 is open source and the community is more than welcome to send pull requests and/or create new issues.
Does the RIPE NCC (or any other entity) offer (paid) support for (non-)members? thanks, nusenu -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Hi Nusenu,
Does the RIPE NCC (or any other entity) offer (paid) support for (non-)members?
Good question… But I don’t know. Cheers, Thiago
participants (3)
-
Nathalie Trenaman -
nusenu -
Thiago da Cruz