Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13285

ByteArrayUtf8CharSequence cannot be cast to java.lang.String exception during replication

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Since upgrading to 7.7 (also tried 7.7.1, and 8.1.0) from 6.6.4, we're seeing the following errors in the SolrCloud elected master for a given collection when updates are written. This was after a full reindex of data (fresh build).

      request: http://solr-1.search-solr.preprod.k8.atcloud.io:80/solr/at-uk_shard1_replica_n2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsolr-2.search-solr.preprod.k8.atcloud.io%3A80%2Fsolr%2Fat-uk_shard1_replica_n1%2F&wt=javabin&version=2
      Remote error message: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
          at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:385) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:183) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
          at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
      

      Following this through to the replica, you'll see:

      08:35:22.060 [qtp1540374340-20] ERROR org.apache.solr.servlet.HttpSolrCall - null:java.lang.ClassCastException: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
          at org.apache.solr.common.util.JavaBinCodec.readEnumFieldValue(JavaBinCodec.java:813)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:339)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:640)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:337)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.readMapEntry(JavaBinCodec.java:819)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:341)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:295)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:333)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:235)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:298)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:191)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
          at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:123)
          at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
          at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
          at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
          at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.Server.handle(Server.java:502)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
          at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
          at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
          at java.lang.Thread.run(Thread.java:748)
      

      I noticed there some similar issues raised in 7x (https://issues.apache.org/jira/browse/SOLR-13255), so I don't know if this is the same sort of thing?

      I've tried solr-users mailing list, and stackoverflow but to no avail. Please let me know if there is any other information I can provide.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            noble.paul Noble Paul
            kstoney Karl Stoney
            Votes:
            3 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment