Goobi Viewer Solr indexer error

Hello again,

Another probably-easy-to-solve installation problem. I’m getting a “record index offline” error on the Goobi Viewer admin dashboard. When I dig into the solr service, I find that it’s not initializing, but the error log is not particularly clarifying on the subject.

Here’s the status of my solr service:

root@intranda-viewer-4:~# systemctl status solrindexer
× solrindexer.service - Goobi viewer - Indexer
     Loaded: loaded (/etc/systemd/system/solrindexer.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2024-03-08 23:43:36 UTC; 5 days ago
   Main PID: 32913 (code=exited, status=255/EXCEPTION)
        CPU: 4.904s

And here’s the contents of its log:

root@intranda-viewer-4:~# cat /opt/digiverso/logs/indexer.log 
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
solr url: http://localhost:8983/solr/collection1/admin/file/?contentType=text/xml;charset=utf-8&file=schema.xml

Any tips on how I can go about debugging or fixing this?

All the best,

John

Hi @JohnConnor

please look in the folder /opt/digiverso/logs/ and check the rights for the indexer.log file. They might be owned by root. Change this to tomcat:tomcat and restart the daemon. This could be a problem for the missing log file content…

Best wishes from

Jan :slight_smile:

Hey, Jan! Thank you, that’s exactly what was happening.

The logs pointed out 2 errors to me, which I think should be flagged in GitHub. If you agree with me, I can open those up as issues. Also, let me know if I should break these sorts of follow-ups off to their own threads for the sake of discoverability on here.

  1. This one I should have caught myself: the Solr core had not been created in a prior step. That’s down to this line in the setup instructions, as far as I can tell:
systemctl start solr
sudo -u solr bin/solr create -c collection1 -n goobiviewer

I believe the second line in the docs should read solr/bin/solr instead of bin/solr based on what the solr user’s home directory seems to be set to as a default.

  1. I’m getting a misleading free space message on my system:
ERROR 2024-03-17 19:20:34.594 [main] io.goobi.viewer.indexer.helper.Hotfolder.checkFreeSpace(Hotfolder.java:548)
        Insufficient free space: -302 / 2048 MB available. Indexer will now shut down.
ERROR 2024-03-17 19:20:34.594 [main] io.goobi.viewer.indexer.SolrIndexerDaemon.main(SolrIndexerDaemon.java:180)
        Insufficient free space, exiting...

Whereas I have an effectively unlimited amount of space:

root@intranda-viewer-4:/opt/digiverso# ll viewer/hotfolder
lrwxrwxrwx 1 root root 32 Mar 17 19:00 viewer/hotfolder -> /mnt/FileSystem-20240308-0311-02/
root@intranda-viewer-4:/opt/digiverso# df -h viewer/hotfolder
Filesystem                               Size  Used Avail Use% Mounted on
10.0.1.102:/FileSystem-20240308-0311-02  8.0E  302M  8.0E   1% /mnt/FileSystem-20240308-0311-02

I think that’s down to the toFile().getFreeSpace() method incorrectly reading the size of a soft link: goobi-viewer-indexer/goobi-viewer-indexer/src/main/java/io/goobi/viewer/indexer/helper/Hotfolder.java at master · intranda/goobi-viewer-indexer · GitHub

If you think I’m right that this is a bug, I could flag it on GitHub. Otherwise, is there some way I can get the system to either read that correctly or to pass over that free space check?

John

1 Like

Hmmm… I am not sure… To work with relative paths in installation instructions is always not good, BUT … it should work. You should be in the correct directory:

Source: 3.1 Classical | Goobi viewer (English)

I will forward this to @andrey.kozhushkov and let him investigate further. As a quickfix, you could change the path within the <hotfolder /> setting in the config_indexer.xml to /mnt/FileSystem-20240308-0311-02 and see if that works?

I think you’re correct regarding the relative path – it was just a mistake I made in the process of correcting the prior one I’d made :wink:

To revise the file system problem, I think it’s something to do with the way it’s mounted – pointing directly at the mount and mounting directly on hotfolder yield the same results. I’m using Oracle’s FileSystem, which is mounted, beginning-to-end, as follows:

sudo apt-get install nfs-common
sudo mkdir -p /mnt/FileSystem-20240308-0311-02
sudo mount 10.0.1.102:/FileSystem-20240308-0311-02 /mnt/FileSystem-20240308-0311-02

It seems to me that this shouldn’t act differently than your NFS export instructions?

Ownership seems correct, too:

root@intranda-viewer-4:/opt/digiverso/indexer# ll /mnt
total 9
drwxr-xr-x  3 root   root   4096 Mar  8 03:57 ./
drwxr-xr-x 19 root   root   4096 Mar  8 02:20 ../
drwxr-xr-x  3 tomcat tomcat    2 Mar 19 20:38 FileSystem-20240308-0311-02/
root@intranda-viewer-4:/opt/digiverso/indexer# ll /mnt/FileSystem-20240308-0311-02/
total 133
drwxr-xr-x 3 tomcat tomcat      2 Mar 19 20:38 ./
drwxr-xr-x 3 root   root     4096 Mar  8 03:57 ../
drwxr-xr-x 2 tomcat tomcat      0 Mar 19 21:24 .snapshot/
-rwxr-xr-x 1 tomcat tomcat 101309 Mar 19 20:38 Memorandum_Book.xml*
drwxr-xr-x 2 tomcat tomcat    108 Mar 19 20:38 Memorandum_Book_media/
root@intranda-viewer-4:/opt/digiverso/indexer# df -h /mnt/FileSystem-20240308-0311-02/
Filesystem                               Size  Used Avail Use% Mounted on
10.0.1.102:/FileSystem-20240308-0311-02  8.0E   13G  8.0E   1% /mnt/FileSystem-20240308-0311-02
root@intranda-viewer-4:/opt/digiverso/indexer# df -h /mnt/FileSystem-20240308-0311-02/

Could it possibly be this: https://bugs.openjdk.org/browse/JDK-8233426

Yes, maybe! Can you give it a try and start the solrIndexer with Java 17? That should give us an answer to this problem probably?!

Hi Jan,

I couldn’t resolve it that way – what I did was run a dockerized version from your latest on github, which should be in 17, and still got the same error. Darn!

However, having gone through that trouble, I was able to patch the files on my local repo and recompile, so it all came off nicely.

Here are the two lines I had to change, for your reference:

  1. I dropped the check free space call (not a long-term fix, obviously): goobi-viewer-indexer/goobi-viewer-indexer/src/main/java/io/goobi/viewer/indexer/helper/Hotfolder.java at master · intranda/goobi-viewer-indexer · GitHub

  2. I also had to point the solr indexer helper at a different endpoint, for some reason – maybe an error in my configuration…: goobi-viewer-indexer/goobi-viewer-indexer/src/main/java/io/goobi/viewer/indexer/helper/SolrSearchIndex.java at master · intranda/goobi-viewer-indexer · GitHub – the url now reads, instead, solrUrl + "/schema?wt=schema.xml"

Thank you so much for answering my questions! Hopefully the exchanges will be useful to others.

I am interested about your ultimate conclusions on the getfreespace question.

Best,

John

For some reason, the allocated free space was cast to integer. Not sure why, but I’ve removed it. Please try running it on your system after checking out the current develop branch - maybe that Java 14 vs 17 won’t be an issue after all?

Best,
Andrey