Data Comparison Use Cases
GemPypOverview of important tags:
1. source_db and target_db tags
- Mandatory for Database Comparison.
- Accepts a single string or dictionary.
- Values for single string can be “MySQL, PostgreSQL, Snowflake, custom”
- Dictionary is used for mongoDB where keys are “type, database and collection” where value of type is mongoDB.
<source_db>Define type of database or custom</source_db>
<target_db>Define type of database or custom</target_db>
Example:
<source_db>MySQL</source_db>
<target_db>Custom</target_db>
2. source_conn and target_conn tags
- Mandatory for Database Comparison.
- Accepts a single string or dictionary.
- We provide database credentials or database connection string.
Connection String corresponding to databases
For Snowflake -
snowflake.connector.connect(user="xxxxxxxxxx",password="xxxxxxxxxx",account="xxxx.ap-south-1",warehouse='COMPUTE_WH',database="SNOWFLAKE_SAMPLE_DATA",schema="TPCH_SF100")
For MongoDB -
pymongo.MongoClient(host=['mongodb+srv://xxxxxxxx:[email protected]/test'], connect=True)
For MySQL -
mysql.connector.connect(host='xxxxxxxxx',user='xxxxxxx', password='xxxxxxxxx',database='xxxxxxxxxxx',port=3306)
For Postgresql
pg8000.connect(host='xxxxxxxxx',user='xxxxxxx', password='xxxxxxxxx',database='xxxxxxxxxxx',port=3306)
Scenario 1: Comparison between two tables in similar database
<data>
<suite>
<!--Here brigeToken and User are the jewel credentials-->
<project-name>TEST_PY</project-name>
<report-name>Data Validation Scenarios</report-name>
<environment>prod</environment>
<mode>optimize</mode>
<enter-point>EnterPointUrl</enter-point>
<threads>4</threads>
<jewel-user>jewelUserName</jewel-user>
<jewel-bridge-token>JewelBridgeToken</jewel-bridge-token>
</suite>
<testcases>
<testcase>
<!-- We use key "type", "database", "Collection" in <source_db> tag
In type key we will provide the type of Database .i.e., Oracle, MySQL,
MongoDB etc -->
<name>Compare data from two tables from MongoDB database</name>
<category>MongoDB</category>
<type>DV</type>
<run_flag>Y</run_flag>
<source_db>{'type':'mongoDB','database':'sourceDataBaseName','collection':'sourceCollectionName'}</source_db>
<source_conn>{'host':['mongodb+srv://xxxxxxxxxx:[email protected]/test'], 'connect':True}</source_conn>
<source_sql>source query</source_sql>
<target_db>{'type':'mongoDB','database':'targetDataBaseName','collection':'targetCollectionName'}</target_db>
<target_conn>{'host':['mongodb+srv://xxxxxxx:[email protected]/test'], 'connect':True}</target_conn>
<target_sql>target query</target_sql>
<keys>_id</keys>
</testcase>
<testcase>
<name>Compare data from two tables from MongoDB database with custom connection</name>
<category>MongoDB</category>
<type>dv</type>
<run_flag>Y</run_flag>
<source_db>{'type':'custom','database':'sample_airbnb','collection':'listingsAndReviews'}</source_db>
<source_conn>pymongo.MongoClient(host=['mongodb+srv://xxxxxxxxx:[email protected]/test'],connect=True)</source_conn>
<source_sql>source query</source_sql>
<target_db>{'type':'custom','database':'sample_airbnb','collection':'listingsAndReviews'}</target_db>
<target_conn>pymongo.MongoClient(host=['mongodb+srv://xxxxxxxx:[email protected]/test'], connect=True)</target_conn>
<target_sql>target query</target_sql>
<keys>_id</keys>
</testcase>
</testcases>
</data>