Automate Your Data, Free Your Mind by Aaron Swerlein
•Download as PPTX, PDF•
0 likes•365 views
This document provides an overview of database automation using the Ruby Gem Sequel. It discusses using Sequel to connect to databases, define models and tables, and perform CRUD operations. Examples are provided for setting up connections, querying with SQL and Ruby methods, and running tests with Cucumber. Benefits covered include increased speed, coverage, and maintaining living documentation.
1 of 44
More Related Content
Automate Your Data, Free Your Mind by Aaron Swerlein
2. • About Me
• CRUD
• ETL
• Testing
• Benefits of Automation
• Tools
• Examples
• Questions
Contents
2
3. • I have a 7 year old daughter.
• I enjoy playing card games (Euchre, Rummy, Uno, MTG)
• Love watching movies -> /(H|B)ollywood/
• I am an identical twin.
About Me
3
5. Introduction | ETL Flow
5
Arrange
• Source file/table
• Front end
application
Action
• Stored Procedure,
SQL script
• Submitting form
on front end
Assert
• Data in target
8. Database Testing | CRUD
8
Create
• New groups
or users
Read
• Connect to
database
• Validate
tables/views
Update
• Change
existing
groups or
accesses
Delete
• Inability to
connect
12. Data Testing | CRUD
12
Create
• Inserted record
is present
Read
• Can access data
Update
• Record is
present, but
original record
no longer there
Delete
• Record no
longer exists
21. Benefits of Automation | Persistence
21
Scenario: validating speaker at QA or the Highway
When the user navigates to the qa or highway speakers webpage
Then the user validates the speaker Kyle Swerlein is present
23. Ruby/Cucumber
23
• Ruby
– Expressive language
– Good flexibility
• Cucumber
– Allows for plain English scenarios to express requirements.
– Helps bridge the developer – analyst knowledge gap
24. Cucumber Example
24
@regression
Scenario: validating test table loaded after jobs run
Given data is present in the source
When job 1 is executed
And job 2 is executed
And job 3 is executed
Then test table is loaded
25. Ruby Gems
25
• Active Record:
– Default for Ruby on Rails
– Large bulky library, includes all of functionality
• Sequel:
– Lighter gem
– Uses adapters and plugins to add functionality to core gem
• Other:
– IBM_DB, OCI8, ADO
26. • Sequel is a RubyGem which allows a user to connect to a
database.
• It includes a comprehensive ORM layer for mapping records to
Ruby objects and handling associated records. (Active-Record
Design Pattern)
• Sequel has adapters for ADO (access), IBM_DB, JDBC,
MySQL, ODBC, Oracle, Hadoop, and many others
Ruby Gem - Sequel
26
29. Creating Table classes
29
class TableName <
Sequel::Model($oracle_connection[“database_name__table_name".to_sym])
set_primary_key :primary_key
end
class TableName < Sequel::Model(
$oracle_connection[Sequel.qualify(“database_name”, “table_name")])
set_primary_key :primary_key
end
Sequel.split_symbols = true
30. Querying with Sequel
30
• Once connected to a Database you can query a couple of
different ways.
– Straight SQL statement
– Sequel ruby methods
31. Sequel Using SQL
31
• The Sequel library has different methods which can be called
passing in a SQL query. To do this, you need to call the
instantiated connection object you created i.e. $connection
– $connection.run(‘sql’)
– $connection.fetch(‘sql’)
35. SQL VS. SEQUEL | Update
35
TableName.where(presenter_name: 'Aaron Swerlein')
.where(conference_name: 'QA or the Highway')
.update(talk_time: '2018-02-27 11:05:00.000000')
36. SQL VS. SEQUEL | Delete
36
TableName.where(presenter_name: 'Aaron Swerlein')
.where(conference_name: 'QA or the Highway').delete
37. Helpful Sequel library
37
• A built in Sequel library which is very useful is the inflector
library. This allows for basic string manipulations which can
make testing easier. Here are some examples and usages.
method input output
camelize/camelcase ‘test_class’ ‘TestClass’
constantize ‘test_class’ TestClass
underscore ‘TestClass’ ‘test_class’
CRUD operations are the basis of all ETL testing. (Extract, Transform, Load).
Not only limited to Database testing. Andy presented on Services which hit on Post (create), Read, Update, Delete
A traditional flow in the ETL process can be broken down with Behavior Driven Actions (BDD)
Arranging data (creating source data, inserting seed record, filling out a web page)
Running the ETL process (Stored Procedure, BTEQ, SQL script, Submitting web page)
Validating results in target as expected (database, file)
The first triangle speaks to the number of tests which will be written for the various aspects of ‘Database’ testing.
Similar to the ‘testing triangle’.
A good strategy is to dive in from the top and work our way down to the Data testing, which is the most time consuming.
The database is traditionally the highest level to look at when looking into testing related to Data. Things which can be tested from this level, would be:
Table/views/procedure all exist as expected
Different access groups exist
Those groups accesses to the specific database are as expected
Table testing lends to many more test cases that are possible. Those include:
Table structure is correct
Columns exist (names are correct)
Columns defined correctly (defaults, size, data type, language, format)
Primary/Foreign Keys exist
Relationships between tables are present
Constraints are built correctly
Data testing has a literal infinite number of tests which could be considered. Those include:
Column values
Column relationships
Foreign key relationship with other tables
Date chaining (transaction management)
Creation:
Data refreshes
PII (personal identifiable information) masking
Actual migration of data between environments
Creating data to meet scenario coverage
Maintain:
Reset updates to data back to default state (rollback)
Removal:
Leaving the database in same state before we started testing (ninja testing)
There are an infinite number of data scenarios which can be tested, all of which are not valid scenarios.
PROD data might not cover all scenarios, and being able to manipulate existing data or create our own for testing will ensure those scenarios are covered
Automating that data creation/destruction process will save time for anyone working with data setup.
Anytime we can reduce the burden of test execution from testers, we give them more time to focus on other ‘household’ items. Refactoring, doing more exploratory or vulnerability tests.
As soon as code is promoted to a higher environment, the high level checks can be run automatically to ensure everything promoted correctly, without the need of manual intervention.
Creation:
Data refreshes
PII (personal identifiable information) masking
Actual migration of data between environments
Creating data to meet scenario coverage
Maintain:
Reset updates to data back to default state (rollback)
Removal:
Leaving the database in same state before we started testing (ninja testing)
Creation:
Data refreshes
PII (personal identifiable information) masking
Actual migration of data between environments
Creating data to meet scenario coverage
Maintain:
Reset updates to data back to default state (rollback)
Removal:
Leaving the database in same state before we started testing (ninja testing)
Creation:
Data refreshes
PII (personal identifiable information) masking
Actual migration of data between environments
Creating data to meet scenario coverage
Maintain:
Reset updates to data back to default state (rollback)
Removal:
Leaving the database in same state before we started testing (ninja testing)
Any test tracking tool (QC, TFS, MTM) is not needed to be relied on so heavily to track test cases, because the automation suite does that for you.
The automation suite is the documentation and the results can be persisted in a DB.
Any test tracking tool (QC, TFS, MTM) is not needed to be relied on so heavily to track test cases, because the automation suite does that for you.
The automation suite is the documentation and the results can be persisted in a DB.
Personally I have worked with Hadoop, Teradata, Oracle, DB2, MSSQL, and Microsoft Access. All of them except Access use the connect method where Access uses the ado method.
Assign each statement to it’s own object. To allow multiple connections.
Prior to Sequel 5.0 the syntax for defining tables was DB__Tablename.
Primary key is optional.
Database reference ($connection) is needed to find table.
Split symbols allows legacy code to keep the ‘__’.
When trying to actually mock an ETL process run is going to be a better option. This is because we can prepare data, run a stored procedure of some sort of ETL flow, and validate the result without caring about the result of the ETL process.