Added collection of Managing Dataset samples by lbristol88 · Pull Request #6 · tswast/google-cloud-python

lbristol88 · 2019-04-05T20:58:06Z

tswast · 2019-04-05T22:48:42Z

bigquery/samples/delete_dataset.py

+    # TODO(developer): Set model_id to the ID of the model to fetch.
+    # dataset_id = 'your-project.your_dataset'
+
+    client.delete_dataset(dataset_id)


The original sample also had an example showing the delete_contents=True option. I think you could actually do the same, but also show off not_found_ok=True.

client.delete_dataset(dataset_id) # Comment about deleting a table that contains tables. # Comment about tables that might not exist. client.delete_dataset(dataset_id, delete_contents=True, not_found_ok=True)

Ok! Added the parameters and notes about what they do.

tswast · 2019-04-05T22:51:18Z

bigquery/samples/get_dataset.py

+    )
+
+    # View dataset properties
+    print("Dataset ID: {}".format(dataset_id))


We can delete this line since you're printing the ID in the same sample in the print statement above.

tswast · 2019-04-05T22:53:38Z

bigquery/samples/tests/test_dataset_samples.py

    assert "Created dataset {}".format(random_dataset_id) in out
+
+    # get dataset
+    get_dataset.get_dataset(client, random_dataset_id)


These should all be separate tests. That way we can more easily tell when an individual sample is broken. Models API is an exception because it's so slow to create one.

All separated out into their own tests. Had to include the create_dataset function for some of the tests or else they wouldn't pass (needs a dataset to make changes to!).

bigquery/samples/update_dataset_access.py

tswast · 2019-04-05T22:56:48Z

bigquery/samples/update_dataset_default_table_expiration.py

+
+def update_dataset_default_table_expiration(client, dataset_id):
+
+    # [START bigquery_update_dataset_default_table_expiration]


Needs to be removed from docs/snippets.py.

Also, please keep the "region tag" the same. That way we can track changes over time better. It was called bigquery_update_dataset_expiration in snippets.

Removed, but forgot to push /facepalm. Updated now!

Updated the region tag as directed.

bigquery/samples/update_dataset_description.py

tswast · 2019-04-05T23:00:30Z

bigquery/samples/update_dataset_default_table_expiration.py

+    )  # API request
+
+    full_dataset_id = "{}.{}".format(dataset.project, dataset.dataset_id)
+    print("Updated dataset {}".format(full_dataset_id))


For testing purposes, it would be good to print the new dataset.default_table_expiration_ms also.

Added the expiration to the print statement.

tswast · 2019-04-05T23:02:12Z

bigquery/samples/tests/test_dataset_samples.py

+        client, random_dataset_id
+    )
+    out, err = capsys.readouterr()
+    assert "Updated dataset {}".format(random_dataset_id) in out


Please look for the expected expiration str(24 * 60 * 60 * 1000) in the output too. That way we can be more certain that the expiration was actually updated as expected.

Moved the time into it's own fixture so it's easier to be used in the test.

tswast · 2019-04-08T21:22:15Z

bigquery/samples/tests/conftest.py



+@pytest.fixture
+def one_day_ms(client):


This doesn't need to be a fixture. Just a normal variable/constant is fine. Fixtures are more for setup & clean-up code.

Actually, now that I see where this is used in the sample code, I'd prefer this code stay in the sample. That way it's clear to someone reading the sample what expected values for expiration look like.

Took this out of the fixture and put it back into the sample and test.

tswast · 2019-04-08T21:23:09Z

bigquery/samples/tests/test_delete_dataset.py

+from .. import delete_dataset
+
+
+def test_delete_dataset(capsys, client, random_dataset_id):


I recommend you use the dataset_id fixture instead of random_dataset_id, since the dataset_id fixture will create a dataset for you (to then delete in the sample).

This is now updated.

tswast · 2019-04-08T21:24:32Z

bigquery/samples/tests/test_get_dataset.py

+from .. import get_dataset
+
+
+def test_get_dataset(capsys, client, random_dataset_id):


Ditto. I recommend you use the dataset_id fixture instead of random_dataset_id, since the dataset_id fixture will create a dataset for you (to then get in the sample).

This is now updated.

tswast · 2019-04-08T21:25:09Z

bigquery/samples/tests/test_list_datasets.py

+from .. import list_datasets
+
+
+def test_list_datasets(capsys, client, random_dataset_id):


The random_dataset_id fixture isn't necessary for this test.

tswast · 2019-04-08T21:25:25Z

bigquery/samples/tests/test_update_dataset_access.py

+from .. import update_dataset_access
+
+
+def test_update_dataset_access(capsys, client, random_dataset_id):


Ditto. I recommend you use the dataset_id fixture instead of random_dataset_id, since the dataset_id fixture will create a dataset for you (to then update in the sample).

Replaced with dataset_id as directed.

tswast · 2019-04-08T21:26:04Z

bigquery/samples/tests/test_update_dataset_default_table_expiration.py

+
+
+def test_update_dataset_default_table_expiration(
+    capsys, client, random_dataset_id, one_day_ms


Ditto. I recommend you use the dataset_id fixture instead of random_dataset_id, since the dataset_id fixture will create a dataset for you (to then update in the sample).

one_day_ms should just be a constant in this file.

Updated dataset_id and added the constant for one_day_ms from the fixture.

tswast · 2019-04-08T21:30:51Z

bigquery/samples/update_dataset_default_table_expiration.py

+    # dataset_id = 'your-project.your_dataset'
+
+    dataset = client.get_dataset(dataset_id)
+    dataset.default_table_expiration_ms = one_day_ms


I know it's against DRY principles, but In sample code it's better to actually show the actual value rather than create a constant. That way people can see typical values and understand that an integer is expected.

Took out the extra variable as directed.

…_day_ms fixture

tswast

Looks good, thanks!

* Added delete dataset function * Added get dataset function * Added list dataset function * Added update dataset description sample * Added update dataset access sample * Added update dataset table expiration sample * Added tests for dataset samples and updated docs * Removing original update dataset access from snippets file. * Moved all dataset tests into own file. Made changes based on feedback. * Made changes based on feedback * Removed unnecessary use of random_dataset_id in tests and removed one_day_ms fixture * Removed unnecessary constant * Stored the math as a constant to make it look cleaner.

lbristol88 added 8 commits April 5, 2019 13:47

Added delete dataset function

6d70183

Added get dataset function

27f5828

Added list dataset function

ea12d6f

Added update dataset description sample

9cc5169

Added update dataset access sample

55e6a61

Added update dataset table expiration sample

1a5270f

Added tests for dataset samples and updated docs

a589575

Removing original update dataset access from snippets file.

ba7c6fd

tswast requested changes Apr 5, 2019

View reviewed changes

lbristol88 added 2 commits April 8, 2019 11:24

Moved all dataset tests into own file. Made changes based on feedback.

34db4d4

Made changes based on feedback

502facb

tswast reviewed Apr 8, 2019

View reviewed changes

lbristol88 added 3 commits April 8, 2019 15:33

Removed unnecessary use of random_dataset_id in tests and removed one…

8f72584

…_day_ms fixture

Removed unnecessary constant

2b30305

Stored the math as a constant to make it look cleaner.

d8e579a

tswast approved these changes Apr 8, 2019

View reviewed changes

tswast merged commit e1bb09c into tswast:bq-snippets Apr 8, 2019


		def update_dataset_default_table_expiration(client, dataset_id):

		# [START bigquery_update_dataset_default_table_expiration]

		from .. import delete_dataset


		def test_delete_dataset(capsys, client, random_dataset_id):

		from .. import get_dataset


		def test_get_dataset(capsys, client, random_dataset_id):

		from .. import list_datasets


		def test_list_datasets(capsys, client, random_dataset_id):

		from .. import update_dataset_access


		def test_update_dataset_access(capsys, client, random_dataset_id):



		def test_update_dataset_default_table_expiration(
		capsys, client, random_dataset_id, one_day_ms

Conversation

lbristol88 commented Apr 5, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tswast left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants