feat: add Connection interface#1374
Merged
stephaniewang526 merged 311 commits intogoogleapis:mainfrom May 6, 2022
Merged
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
shollyman
reviewed
Aug 6, 2021
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryDryRunResult.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryResultSet.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuerySQLException.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Connection.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Connection.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionSettings.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/QueryResultRowFormat.java
Outdated
Show resolved
Hide resolved
eb824b4 to
8935c10
Compare
54f0826 to
3595799
Compare
e2dc6b9 to
4584ef2
Compare
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java
Show resolved
Hide resolved
…m Long to Integer
…ConnectionSettings connectionSettings)
…ncy to fix dependency check failure
This was referenced May 6, 2022
Merged
Contributor
|
Hi Read of 100_000 rows takes 23930 ms. Mono.fromCallable { bigQueryOptionsBuilder.build().service }
.flatMap { context ->
val connectionSettings = ConnectionSettings.newBuilder()
.setRequestTimeout(10L)
.setUseReadAPI(true)
.setMaxResults(1000)
.setNumBufferedRows(1000)
.setUseQueryCache(true)
.build();
val connection = context.createConnection(connectionSettings)
val bqResult = connection.executeSelect(sql)
val result = Flux.usingWhen(
Mono.just(bqResult.resultSet),
{ resultSet -> resultSet.toFlux(bqResult.schema) },
{ _ -> Mono.fromRunnable<Unit> { connection.close() } }
)
Mono.just(Data(result, bqResult.schema.toSchema()))
}
...
fun ResultSet.toFlux(schema:Schema): Flux<DataRecord> {
return Flux.generate<DataRecord> { sink ->
if (next()) {
sink.next(toDataRecord(schema))
} else {
sink.complete()
}
}
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background: go/bq-sql-client-java
This PR provides a completely new Connection interface which defines separate APIs for different types of queries. This allows us to provide an industry standard way for database applications to build against the Java client library. We provide 3 new JDBC-esque API methods:
[In-scope for this PR] executeSelect - Only supports read only SELECT queries.
[In-scope for this PR] dryRun - returns som query processing statistics including schema and query parameters.
[Not in this PR] executeUpdate - Only supports DML and DDL.
[Not in this PR] execute - Any SQL - scripts, DML, DDL, SELECT etc statements.
We also integrate with the BigQueryStorage client library and use the high throughput Read API when applicable to parse query results using Arrow format. Arrow has shown better performance over Avro as the row serialization format in this experiment.