Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13285

ByteArrayUtf8CharSequence cannot be cast to java.lang.String exception during replication

    XMLWordPrintableJSON

Details

    Description

      Since upgrading to 7.7 (also tried 7.7.1, and 8.1.0) from 6.6.4, we're seeing the following errors in the SolrCloud elected master for a given collection when updates are written. This was after a full reindex of data (fresh build).

      request: http://solr-1.search-solr.preprod.k8.atcloud.io:80/solr/at-uk_shard1_replica_n2/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fsolr-2.search-solr.preprod.k8.atcloud.io%3A80%2Fsolr%2Fat-uk_shard1_replica_n1%2F&wt=javabin&version=2
      Remote error message: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
          at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:385) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:183) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
          at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
      

      Following this through to the replica, you'll see:

      08:35:22.060 [qtp1540374340-20] ERROR org.apache.solr.servlet.HttpSolrCall - null:java.lang.ClassCastException: org.apache.solr.common.util.ByteArrayUtf8CharSequence cannot be cast to java.lang.String
          at org.apache.solr.common.util.JavaBinCodec.readEnumFieldValue(JavaBinCodec.java:813)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:339)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:640)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:337)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.readMapEntry(JavaBinCodec.java:819)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:341)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:295)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:333)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:235)
          at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:298)
          at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278)
          at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:191)
          at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
          at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:123)
          at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
          at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
          at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
          at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.Server.handle(Server.java:502)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
          at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
          at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
          at java.lang.Thread.run(Thread.java:748)
      

      I noticed there some similar issues raised in 7x (https://issues.apache.org/jira/browse/SOLR-13255), so I don't know if this is the same sort of thing?

      I've tried solr-users mailing list, and stackoverflow but to no avail. Please let me know if there is any other information I can provide.

      Attachments

        1. SOLR-13285.patch
          3 kB
          Karl Stoney
        2. SOLR-13285.patch
          3 kB
          Noble Paul

        Issue Links

          Activity

            People

              noble.paul Noble Paul
              kstoney Karl Stoney
              Votes:
              3 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: