Postings listsWhile it may surprise you if you are new to search engine internals, one of the most important building blocks of a search engine is the ability to efficiently compress and quickly decode sorted lists of integers. Why is this useful? As you may know, Elasticsearch shards, which are Lucene indices under the hood, split the data that they store into segments which are regularly merged
圧縮されたソート済の整数列ってのは汎用的なデータ構造で、たとえば検索エンジンの転置インデックスとか、いろんなところで使うわけです。で、検索エンジンの場合は速度重要なので、PForDeltaとか様々なデータ構造が研究されてる。 一方、H2O には、ブラウザキャッシュに載ってない js や css をサーバプッシュする仕組み「cache-aware server push」があって、何がキャッシュされているか判定するためにブルームフィルタを全ての HTTP リクエストに含める必要がある。 で、ブルームフィルタを圧縮しようと思うと、ブルームフィルタってのはソート済の整数列として表現できるので、これを圧縮しようって話になる。 検索エンジン等で使う場合は速度重要だけど、HTTPリクエストに載せる場合は空間効率のほうが重要になる。ってことで、空間効率が理論限界に近いゴロム符号（の特殊系であるライス符号
A better compressed bitset Roaring bitmaps are compressed bitmaps. They can be hundreds of times faster. Grab one of our research papers Roaring Bitmaps on GitHub Widely used Roaring is found in Google Procella: YouTube’s SQL Engine, Apache Lucene and derivative systems such as Solr and Elasticsearch, Apache Druid, Apache Spark, Apache Hive, Apache Tez, Apache Zeppelin, Apache Doris, Apache Carbon
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014 Apache Parquet is an open-source columnar storage format for efficient data storage and analytics. It provides efficient compression and encoding techniques that enable fast scans and queries of large datasets. Parquet 2.0 improves on these efficiencies through techniques like delta encoding, dictionary encoding, run-length
This article is about codes that are a variable number of bytes. For a more general category of variable-bit-length codes, see variable-length code. A variable-length quantity (VLQ) is a universal code that uses an arbitrary number of binary octets (eight-bit bytes) to represent an arbitrarily large integer. A VLQ is essentially a base-128 representation of an unsigned integer with the addition of
Ville Tuulos Principal Engineer @ AdRoll ville.tuulos@adroll.com We faced the key technical challenge of modern Business Intelligence: How to query tens of billions of events interactively? Our solution, DeliRoll, is implemented in Python. Everyone knows that Python is SLOW. You can't handle big data with low latency in Python! Small Benchmark Data: 1.5 billion rows, 400 columns - 660GB. Smaller e
For many companies, understanding what is going on in your business involves lots of data. But, how do you query 10s of billions of data points? How can a company begin to make sense of so much information? Ville Tuulos, Principle Engineer at AdRoll, a company producing tons of big data, demonstrates how AdRoll uses Python to squeeze every bit of performance out of a single high-end server. They m
ORC File Format File Structure Stripe Structure HiveQLSyntax Serialization and Compression Integer Column Serialization String Column Serialization Compression ORC File Format The Optimized Row Columnar (ORC) file format provides a highly efficient way to store Hive data. It was designed to overcome limitations of the other Hive file formats. Using ORC files improves performance when Hive is readi
Mostly encoding - Amazon Redshift
Mostly encodings are useful when the data type for a column is larger than most of the stored values require. By specifying a mostly encoding for this type of column, you can compress the majority of the values in the column to a smaller standard storage size. The remaining values that cannot be compressed are stored in their raw form. For example, you can compress a 16-bit column, such as an INT2
Delta encoding - Amazon Redshift
Delta encodings are very useful for date time columns. Delta encoding compresses data by recording the difference between values that follow each other in the column. This difference is recorded in a separate dictionary for each block of column values on disk. (An Amazon Redshift disk block occupies 1 MB.) For example, suppose that the column contains 10 integers in sequence from 1 to 10. The firs
JavaでByte型変数をInt型にキャストする際には、気を付けなければならないことがあります。 前回のエントリーで、Byte型配列を16進数文字列に変換するメソッドを作りましたが、その時にByte型をInt型へ変換するには、ただ単にキャスト変換するのではなく、「0xFF」でANDビット演算しなければ値がおかしくなるという事を書きました。 どういう事かというと… [java] byte a = (byte)0xC8 //10進数では200のはず int b = (int)a; Sytem.out.println(b); [/java] この実行結果は、「200」ではなく、「-56」が表示されます。 それを、 [java] byte a = (byte)0xC8 //10進数では200のはず int b = a & 0xff; Sytem.out.println(b); [/
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
